demos/fashion-mnist-adversarial/ART_with_FfDL.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Generate Adversarial Samples for Deep Learning Models with the Adversarial Robustness Toolbox (ART)\n",
    "\n",
    "This notebook shows how to use adversarial attack techniques from the [Adversarial Robustness Toolbox (ART)](https://developer.ibm.com/code/open/projects/adversarial-robustness-toolbox/) on Deep Learning models trained with [Fabric for Deep Learning (FfDL)](https://github.com/IBM/ffdl/). The ART library supports crafting and analyzing various attack and defense methods for deep learning models. \n",
    "\n",
    "In this notebook, you will learn how to incorporate one of the attack methods supported by ART, the *Fast Gradient Method (FGM)*, into your training pipeline to generate adversarial samples to evaluate the robustness of the trained model. The model is a *Convolutional Neural Network (CNN)* trained on the [Fashion-MNIST](https://github.com/zalandoresearch/fashion-mnist) dataset using the [Keras](https://keras.io/) deep learning framework with a [TensorFlow](https://www.tensorflow.org/) backend.\n",
    "\n",
    "The Github repository for the ART library can be found here: https://github.com/IBM/adversarial-robustness-toolbox\n",
    "\n",
    "This notebook uses Python 3.\n",
    "\n",
    "\n",
    "## Contents\n",
    "\n",
    "1.\t[Set up the environment](#setup)\n",
    "2.\t[Create a Keras model](#model)\n",
    "3.  [Train the model](#train)\n",
    "4.\t[Generate adversarial samples for a robustness check](#art)\n",
    "5.\t[Summary and next steps](#summary)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"setup\"></a>\n",
    "## 1. Setup\n",
    "\n",
    "It is recommended that you run this notebook inside a Python 3 virtual environment. Make sure you have all required libraries installed.\n",
    "\n",
    "To store model and training data, this notebook requires access to a Cloud Object Storage (COS) instance. [BlueMix Cloud Object Storage](https://console.bluemix.net/catalog/services/cloud-object-storage) offers a free *lite plan*. Follow [these instructions](https://dataplatform.ibm.com/docs/content/analyze-data/ml_dlaas_object_store.html) to create your COS instance and generate [service credentials](https://console.bluemix.net/docs/services/cloud-object-storage/iam/service-credentials.html#service-credentials) with [HMAC keys](https://console.bluemix.net/docs/services/cloud-object-storage/hmac/credentials.html#using-hmac-credentials).\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Enter your cluster and object storage information:**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "env = dict(os.environ)\n",
    "\n",
    "user_data = {\n",
    "    \"public_ip\"                  : env.get(\"PUBLIC_IP\"),             # Public IP of your Kubernetes cluster with FfDL deployed on it \n",
    "    \"kubeconfig_file\"            : env.get(\"KUBECONFIG\"),            # Path pointing to your Kubernetes cluster configuration file \n",
    "    \"cos_hmac_access_key_id\"     : env.get(\"AWS_ACCESS_KEY_ID\"),     # Cloud Object Storage (AWS) Access Key ID \n",
    "    \"cos_hmac_secret_access_key\" : env.get(\"AWS_SECRET_ACCESS_KEY\"), # Cloud Object Storage (AWS) Secret Access Key \n",
    "    \"cos_service_endpoint\"       : env.get(\"AWS_ENDPOINT_URL\"),      # Cloud Object Storage endpoint, i.e. 'https://s3-api.us-geo.objectstorage.softlayer.net' \n",
    "    \"cos_region_name\"            : env.get(\"AWS_DEFAULT_REGION\"),    # Cloud Object Storage default region, i.e. 'us-east-1' \n",
    "    \"ffdl_dir\"                   : env.get(\"FFDL_DIR\", os.getcwd().replace(\"/demos/fashion-mnist-adversarial\", \"\"))\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "unset_vars = [key for (key, value) in user_data.items() if not value]\n",
    "\n",
    "for var in unset_vars:\n",
    "    print(\"Dictionary 'user_data' is missing '%s'\" % var)\n",
    "    \n",
    "assert not unset_vars, \"Enter 'user_data' to run this notebook!\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.1. Verify or Install Required Python Libraries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Using TensorFlow backend.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "All required libraries are installed.\n",
      "keras>=2.1.6\r\n",
      "tensorflow>=1.8\r\n",
      "ipython>=5.0.0\r\n",
      "jupyter>=1.0.0\r\n",
      "requests>=2.12.0,<=2.18.4\r\n",
      "wget\r\n",
      "boto3\r\n",
      "git+git://github.com/IBM/adversarial-robustness-toolbox@master\r\n"
     ]
    }
   ],
   "source": [
    "import sys\n",
    "\n",
    "def is_venv():\n",
    "    return (hasattr(sys, 'real_prefix') or (hasattr(sys, 'base_prefix') and sys.base_prefix != sys.prefix))\n",
    "\n",
    "try:\n",
    "    import keras, tensorflow, requests, wget, boto3, art\n",
    "    print(\"All required libraries are installed.\")\n",
    "    !cat requirements.txt\n",
    "except ModuleNotFoundError:\n",
    "    if is_venv:\n",
    "        print(\"Installing required libraries into virtual environment.\")\n",
    "        !python -m pip install -r requirements.txt\n",
    "    else:\n",
    "        print(\"Please install the required libraries.\")\n",
    "        !cat requirements.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.2. Connect to Cloud Object Storage  (COS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a `boto3.resource` to interact with the COS instance. The `boto3` library allows Python developers to manage Cloud Object Storage (COS)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "cos = boto3.resource(\"s3\", \n",
    "                     aws_access_key_id     = user_data[\"cos_hmac_access_key_id\"],\n",
    "                     aws_secret_access_key = user_data[\"cos_hmac_secret_access_key\"],\n",
    "                     endpoint_url          = user_data[\"cos_service_endpoint\"],\n",
    "                     region_name           = user_data[\"cos_region_name\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# for bucket in cos.buckets.all():\n",
    "#     print(bucket.name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create two buckets, which you will use to store training data and training results.\n",
    "\n",
    "**Note:** The bucket names must be unique."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Creating bucket \"training-data-b467af13-ab67-4b5d-b0ca-c522fa0f4802\" ...\n",
      "Creating bucket \"training-results-b467af13-ab67-4b5d-b0ca-c522fa0f4802\" ...\n"
     ]
    }
   ],
   "source": [
    "from uuid import uuid4\n",
    "\n",
    "bucket_uid             = str(uuid4())\n",
    "training_data_bucket   = 'training-data-' + bucket_uid\n",
    "training_result_bucket = 'training-results-' + bucket_uid\n",
    "\n",
    "def create_buckets(bucket_names):\n",
    "    for bucket in bucket_names:\n",
    "        print('Creating bucket \"{}\" ...'.format(bucket))\n",
    "        try:\n",
    "            cos.create_bucket(Bucket=bucket)\n",
    "        except boto3.exceptions.botocore.client.ClientError as e:\n",
    "            print('Error: {}.'.format(e.response['Error']['Message']))\n",
    "\n",
    "buckets = [training_data_bucket, training_result_bucket]\n",
    "\n",
    "create_buckets(buckets)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now you should have 2 buckets."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1.3. Download MNIST Training Data and Upload it to the COS Buckets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select a data set (https://keras.io/datasets/):\n",
    "- `mnist.npz`\n",
    "- `fashion_mnist.npz`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets = [\"mnist.npz\", \"fashion_mnist.npz\"]\n",
    "\n",
    "dataset_filename = datasets[1] # 'fashion_mnist.npz'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download the training data and upload it to the `training-data` bucket."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Uploading files to training-data-b467af13-ab67-4b5d-b0ca-c522fa0f4802:\n",
      "- fashion_mnist.npz was uploaded\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import mnist, fashion_mnist\n",
    "import numpy as np\n",
    "\n",
    "if \"fashion\" in dataset_filename:\n",
    "    (x_train, y_train), (x_test, y_test) = fashion_mnist.load_data() \n",
    "else:\n",
    "    (x_train, y_train), (x_test, y_test) = mnist.load_data()\n",
    "    \n",
    "np.savez_compressed(dataset_filename, x_train=x_train , y_train=y_train, x_test=x_test, y_test=y_test)\n",
    "\n",
    "bucket_obj = cos.Bucket(training_data_bucket)\n",
    "print(\"Uploading files to {}:\".format(training_data_bucket))\n",
    "\n",
    "bucket_obj.upload_file(dataset_filename, dataset_filename)\n",
    "print('- {} was uploaded'.format(dataset_filename)) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Have a look at the list of the created buckets and their contents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training-data-b467af13-ab67-4b5d-b0ca-c522fa0f4802\n",
      "  File: fashion_mnist.npz, 30146.33kB\n",
      "training-results-b467af13-ab67-4b5d-b0ca-c522fa0f4802\n"
     ]
    }
   ],
   "source": [
    "def print_bucket_contents(buckets):\n",
    "    for bucket_name in buckets:\n",
    "        print(bucket_name)\n",
    "        bucket_obj = cos.Bucket(bucket_name)\n",
    "        for obj in bucket_obj.objects.all():\n",
    "            print(\"  File: {}, {:4.2f}kB\".format(obj.key, obj.size/1024))\n",
    "\n",
    "print_bucket_contents(buckets)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You are done with COS, and you are ready to train your model!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<a id=\"model\"></a>\n",
    "## 2. Create the Keras model\n",
    "\n",
    "In this section we:\n",
    "\n",
    "- [2.1 Package the model definition](#zip)\n",
    "- [2.2 Prepare the training definition metadata](#manifest)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.1. Create the Model Zip File <a id=\"zip\"></a>\n",
    "\n",
    "Let's create the model [`convolutional_keras.py`](../edit/convolutional_keras.py) and add it to a zip file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "script_filename  = \"convolutional_keras.py\"\n",
    "archive_filename = 'model.zip'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Writing convolutional_keras.py\n"
     ]
    }
   ],
   "source": [
    "%%writefile $script_filename\n",
    "\n",
    "from __future__ import print_function\n",
    "from keras.models import Sequential\n",
    "from keras.layers import Dense, Dropout, Flatten\n",
    "from keras.layers import Conv2D, MaxPooling2D\n",
    "from keras import backend as K\n",
    "\n",
    "import keras\n",
    "import numpy as np\n",
    "import sys\n",
    "import os\n",
    "\n",
    "batch_size = 128\n",
    "num_classes = 10\n",
    "epochs = 1\n",
    "\n",
    "img_rows, img_cols = 28, 28\n",
    "\n",
    "\n",
    "def main(argv):\n",
    "    if len(argv) < 2:\n",
    "        sys.exit(\"Not enough arguments provided.\")\n",
    "    global image_path\n",
    "    i = 1\n",
    "    while i <= 2:\n",
    "        arg = str(argv[i])\n",
    "        if arg == \"--data\":\n",
    "            image_path = os.path.join(os.environ[\"DATA_DIR\"], str(argv[i+1]))\n",
    "        i += 2\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    main(sys.argv)\n",
    "\n",
    "\n",
    "# load training and test data from npz file\n",
    "f = np.load(image_path)\n",
    "x_train = f['x_train']\n",
    "y_train = f['y_train']\n",
    "x_test  = f['x_test']\n",
    "y_test  = f['y_test']\n",
    "f.close()\n",
    "\n",
    "if K.image_data_format() == 'channels_first':\n",
    "    x_train = x_train.reshape(x_train.shape[0], 1, img_rows, img_cols)\n",
    "    x_test = x_test.reshape(x_test.shape[0], 1, img_rows, img_cols)\n",
    "    input_shape = (1, img_rows, img_cols)\n",
    "else:\n",
    "    x_train = x_train.reshape(x_train.shape[0], img_rows, img_cols, 1)\n",
    "    x_test = x_test.reshape(x_test.shape[0], img_rows, img_cols, 1)\n",
    "    input_shape = (img_rows, img_cols, 1)\n",
    "\n",
    "x_train = x_train.astype('float32')\n",
    "x_test = x_test.astype('float32')\n",
    "x_train /= 255\n",
    "x_test /= 255\n",
    "\n",
    "# convert class vectors to binary class matrices\n",
    "y_train = keras.utils.to_categorical(y_train, num_classes)\n",
    "y_test = keras.utils.to_categorical(y_test, num_classes)\n",
    "\n",
    "# model\n",
    "model = Sequential()\n",
    "model.add(Conv2D(32, kernel_size=(3, 3),\n",
    "                 activation='relu',\n",
    "                 input_shape=input_shape))\n",
    "model.add(Conv2D(64, (3, 3), activation='relu'))\n",
    "model.add(MaxPooling2D(pool_size=(2, 2)))\n",
    "model.add(Dropout(0.25))\n",
    "model.add(Flatten())\n",
    "model.add(Dense(128, activation='relu'))\n",
    "model.add(Dropout(0.5))\n",
    "model.add(Dense(num_classes, activation='softmax'))\n",
    "\n",
    "model.compile(loss=keras.losses.categorical_crossentropy,\n",
    "              optimizer=keras.optimizers.Adadelta(),\n",
    "              metrics=['accuracy'])\n",
    "model.fit(x_train, y_train, batch_size=batch_size, epochs=epochs, verbose=1, validation_split=0.1)\n",
    "\n",
    "score = model.evaluate(x_test, y_test, verbose=0)\n",
    "\n",
    "print('Test loss:', score[0])\n",
    "print('Test accuracy:', score[1])\n",
    "\n",
    "model_wt_path = os.environ[\"RESULT_DIR\"] + \"/keras_original_model.hdf5\"\n",
    "model.save(model_wt_path)\n",
    "print(\"Model saved to file: %s\" % model_wt_path)\n",
    "\n",
    "model_def_path = os.environ[\"RESULT_DIR\"] + \"/keras_original_model.json\"\n",
    "model_json = model.to_json()\n",
    "with open(model_def_path, \"w\") as json_file:\n",
    "    json_file.write(model_json)\n",
    "print(\"Model definition saved to file: %s\" % model_def_path)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "import zipfile\n",
    "\n",
    "zipfile.ZipFile(archive_filename, mode='w').write(script_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2.2. Prepare the Training Definition Metadata <a id=\"manifest\"></a>\n",
    "- *FfDL* does not have a *Keras* community image so we need to `pip`-install *Keras* prior to running the `training_command` \n",
    "- Your COS credentials are referenced in the `data_stores` > `connection` data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "import yaml\n",
    "\n",
    "training_command = \"pip3 install keras; python3 %s --data ${DATA_DIR}/%s\" % (script_filename, dataset_filename)\n",
    "\n",
    "manifest = {\n",
    "  \"name\":        \"keras_digit_recognition\",\n",
    "  \"description\": \"Hand-written Digit Recognition Training\",\n",
    "  \"version\":     \"1.0\",\n",
    "  \"memory\":      \"2Gb\",\n",
    "  \"gpus\": 0,\n",
    "  \"cpus\": 2,\n",
    "  \"data_stores\": [\n",
    "    {\n",
    "      \"id\":   \"s3-art-notebook-files\",\n",
    "      \"type\": \"mount_cos\",\n",
    "      \"training_data\": {\n",
    "        \"container\": training_data_bucket\n",
    "      },\n",
    "      \"training_results\": {\n",
    "        \"container\": training_result_bucket\n",
    "      },\n",
    "      \"connection\": {\n",
    "        \"auth_url\":  user_data[\"cos_service_endpoint\"],\n",
    "        \"user_name\": user_data[\"cos_hmac_access_key_id\"],\n",
    "        \"password\":  user_data[\"cos_hmac_secret_access_key\"]\n",
    "      }\n",
    "    }\n",
    "  ],\n",
    "  \"framework\": {\n",
    "    \"name\":    \"tensorflow\",\n",
    "    \"version\": \"1.5.0-py3\",\n",
    "    \"command\": training_command\n",
    "  },\n",
    "  \"evaluation_metrics\": {\n",
    "    \"type\": \"tensorboard\",\n",
    "    \"in\":   \"$JOB_STATE_DIR/logs/tb\"\n",
    "  }\n",
    "}\n",
    "\n",
    "yaml.dump(manifest, open(\"manifest.yml\", \"w\"), default_flow_style=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Train the Model<a id=\"train\"></a>\n",
    "\n",
    "In this section, learn how to:\n",
    "- [3.1. Setup the command line environment](#cmd_setup)\n",
    "- [3.2. Train the model with FfDL](#train_ffdl)\n",
    "- [3.3. Monitor the training log](#log)\n",
    "- [3.4. Cancel the training](#cancel)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.1. Setup the Command Line Environment <a id=\"cmd_setup\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: PUBLIC_IP=169.60.36.238\n",
      "env: KUBECONFIG=~/.bluemix/plugins/container-service/clusters/my-ffdl-cluster/kube-config-dal12-my-ffdl-cluster.yml\n"
     ]
    }
   ],
   "source": [
    "%env PUBLIC_IP  {user_data[\"public_ip\"]}\n",
    "%env KUBECONFIG {user_data[\"kubeconfig_file\"]}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Setup the DLaaS URL, username and password"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "env: DLAAS_URL=http://169.60.36.238:30154\n",
      "env: DLAAS_USERNAME=test-user\n",
      "env: DLAAS_PASSWORD=test\n"
     ]
    }
   ],
   "source": [
    "restapi_port = !kubectl get service ffdl-restapi -o jsonpath='{.spec.ports[0].nodePort}'\n",
    "dlaas_url    = \"http://%s:%s\" % (user_data[\"public_ip\"], restapi_port[0])\n",
    "\n",
    "%env DLAAS_URL        $dlaas_url\n",
    "%env DLAAS_USERNAME = test-user\n",
    "%env DLAAS_PASSWORD = test"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Obtain the correct FfDL CLI for your machine"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import platform\n",
    "\n",
    "ffdl = \"%s/cli/bin/ffdl-%s\" % (user_data[\"ffdl_dir\"], \"osx\" if platform.system() == \"Darwin\" else \"linux\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.2. Start the Training Job <a id=\"train_ffdl\"></a>\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[\"Deploying model with manifest 'manifest.yml' and model file 'model.zip'...\",\n",
       " 'Model ID: training-lh276tdmR',\n",
       " 'OK']"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "out = !{ffdl} train \"manifest.yml\" \"model.zip\"\n",
    "out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.3.  Monitor the Training Logs<a id=\"log\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Getting model training logs for '\u001b[1;36mtraining-lh276tdmR\u001b[0m'...\n",
      "Training with training/test data at:\n",
      "  DATA_DIR: /mnt/data/training-data-b467af13-ab67-4b5d-b0ca-c522fa0f4802\n",
      "  MODEL_DIR: /job/model-code\n",
      "  TRAINING_JOB: \n",
      "  TRAINING_COMMAND: pip3 install keras; python3 convolutional_keras.py --data ${DATA_DIR}/fashion_mnist.npz\n",
      "Storing trained model at:\n",
      "  RESULT_DIR: /mnt/results/training-results-b467af13-ab67-4b5d-b0ca-c522fa0f4802/training-lh276tdmR\n",
      "Contents of $MODEL_DIR\n",
      "total 12\n",
      "drwxrwxrwx 2 6342627 4294967294 4096 Jul 14 01:37 .\n",
      "drwxrwxrwx 4 nobody  4294967294 4096 Jul 14 01:37 ..\n",
      "-rwxrwxrwx 1 6342627 4294967294 2673 Jul 13 18:23 convolutional_keras.py\n",
      "Contents of $DATA_DIR\n",
      "total 30151\n",
      "drwxrwxr-x 1 root root        0 Jan  1  1970 .\n",
      "drwxr-xr-x 3 root root     4096 Jul 14 01:36 ..\n",
      "---------- 1 root root 30869845 Jul 14 01:23 fashion_mnist.npz\n",
      "CHECKPOINT_DIR=/mnt/results/training-results-b467af13-ab67-4b5d-b0ca-c522fa0f4802/_wml_checkpoints\n",
      "DATA_DIR=/mnt/data/training-data-b467af13-ab67-4b5d-b0ca-c522fa0f4802\n",
      "DOWNWARD_API_POD_NAME=learner-a2b0c40c-9975-415a-46d9-4f1c89ce7dcc-0\n",
      "DOWNWARD_API_POD_NAMESPACE=default\n",
      "ELASTICSEARCH_PORT=tcp://172.21.204.56:9200\n",
      "ELASTICSEARCH_PORT_9200_TCP=tcp://172.21.204.56:9200\n",
      "ELASTICSEARCH_PORT_9200_TCP_ADDR=172.21.204.56\n",
      "ELASTICSEARCH_PORT_9200_TCP_PORT=9200\n",
      "ELASTICSEARCH_PORT_9200_TCP_PROTO=tcp\n",
      "ELASTICSEARCH_SERVICE_HOST=172.21.204.56\n",
      "ELASTICSEARCH_SERVICE_PORT=9200\n",
      "ELASTICSEARCH_SERVICE_PORT_HTTP=9200\n",
      "FFDL_LCM_PORT=tcp://172.21.37.214:80\n",
      "FFDL_LCM_PORT_80_TCP=tcp://172.21.37.214:80\n",
      "FFDL_LCM_PORT_80_TCP_ADDR=172.21.37.214\n",
      "FFDL_LCM_PORT_80_TCP_PORT=80\n",
      "FFDL_LCM_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_LCM_SERVICE_HOST=172.21.37.214\n",
      "FFDL_LCM_SERVICE_PORT=80\n",
      "FFDL_LCM_SERVICE_PORT_GRPC=80\n",
      "FFDL_RESTAPI_PORT=tcp://172.21.137.113:80\n",
      "FFDL_RESTAPI_PORT_80_TCP=tcp://172.21.137.113:80\n",
      "FFDL_RESTAPI_PORT_80_TCP_ADDR=172.21.137.113\n",
      "FFDL_RESTAPI_PORT_80_TCP_PORT=80\n",
      "FFDL_RESTAPI_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_RESTAPI_SERVICE_HOST=172.21.137.113\n",
      "FFDL_RESTAPI_SERVICE_PORT=80\n",
      "FFDL_RESTAPI_SERVICE_PORT_FFDL=80\n",
      "FFDL_TRAINER_PORT=tcp://172.21.146.32:80\n",
      "FFDL_TRAINER_PORT_80_TCP=tcp://172.21.146.32:80\n",
      "FFDL_TRAINER_PORT_80_TCP_ADDR=172.21.146.32\n",
      "FFDL_TRAINER_PORT_80_TCP_PORT=80\n",
      "FFDL_TRAINER_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_TRAINER_SERVICE_HOST=172.21.146.32\n",
      "FFDL_TRAINER_SERVICE_PORT=80\n",
      "FFDL_TRAINER_SERVICE_PORT_GRPC=80\n",
      "FFDL_TRAININGDATA_PORT=tcp://172.21.180.242:80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP=tcp://172.21.180.242:80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_ADDR=172.21.180.242\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_PORT=80\n",
      "FFDL_TRAININGDATA_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_TRAININGDATA_SERVICE_HOST=172.21.180.242\n",
      "FFDL_TRAININGDATA_SERVICE_PORT=80\n",
      "FFDL_TRAININGDATA_SERVICE_PORT_GRPC=80\n",
      "FFDL_UI_PORT=tcp://172.21.53.228:80\n",
      "FFDL_UI_PORT_80_TCP=tcp://172.21.53.228:80\n",
      "FFDL_UI_PORT_80_TCP_ADDR=172.21.53.228\n",
      "FFDL_UI_PORT_80_TCP_PORT=80\n",
      "FFDL_UI_PORT_80_TCP_PROTO=tcp\n",
      "FFDL_UI_SERVICE_HOST=172.21.53.228\n",
      "FFDL_UI_SERVICE_PORT=80\n",
      "FFDL_UI_SERVICE_PORT_HTTP=80\n",
      "GPU_COUNT=0.000000\n",
      "HOME=/root\n",
      "JOB_STATE_DIR=/job\n",
      "LEARNER_ID=1\n",
      "LEARNER_NAME_PREFIX=learner-a2b0c40c-9975-415a-46d9-4f1c89ce7dcc\n",
      "LOG_DIR=/job/logs\n",
      "MODEL_DIR=/job/model-code\n",
      "NUM_LEARNERS=1\n",
      "OLDPWD=/notebooks\n",
      "PATH=/usr/local/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin\n",
      "PROMETHEUS_PORT=tcp://172.21.74.212:9090\n",
      "PROMETHEUS_PORT_9090_TCP=tcp://172.21.74.212:9090\n",
      "PROMETHEUS_PORT_9090_TCP_ADDR=172.21.74.212\n",
      "PROMETHEUS_PORT_9090_TCP_PORT=9090\n",
      "PROMETHEUS_PORT_9090_TCP_PROTO=tcp\n",
      "PROMETHEUS_SERVICE_HOST=172.21.74.212\n",
      "PROMETHEUS_SERVICE_PORT=9090\n",
      "PROMETHEUS_SERVICE_PORT_PROMETHEUS=9090\n",
      "PWD=/job/model-code\n",
      "PYTHONPATH=:/job/model-code\n",
      "RESULT_DIR=/mnt/results/training-results-b467af13-ab67-4b5d-b0ca-c522fa0f4802/training-lh276tdmR\n",
      "S3_PORT=tcp://172.21.46.129:80\n",
      "S3_PORT_80_TCP=tcp://172.21.46.129:80\n",
      "S3_PORT_80_TCP_ADDR=172.21.46.129\n",
      "S3_PORT_80_TCP_PORT=80\n",
      "S3_PORT_80_TCP_PROTO=tcp\n",
      "S3_SERVICE_HOST=172.21.46.129\n",
      "S3_SERVICE_PORT=80\n",
      "SHLVL=3\n",
      "TRAINING_COMMAND=pip3 install keras; python3 convolutional_keras.py --data ${DATA_DIR}/fashion_mnist.npz\n",
      "TRAINING_ID=training-lh276tdmR\n",
      "_=/usr/bin/env\n",
      "Sat Jul 14 01:37:35 UTC 2018: Running training job\n",
      "Collecting keras\n",
      "  Downloading https://files.pythonhosted.org/packages/68/12/4cabc5c01451eb3b413d19ea151f36e33026fc0efb932bf51bcaf54acbf5/Keras-2.2.0-py2.py3-none-any.whl (300kB)\n",
      "Requirement already satisfied: numpy>=1.9.1 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Collecting pyyaml (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/9e/a3/1d13970c3f36777c583f136c136f804d70f500168edc1edea6daa7200769/PyYAML-3.13.tar.gz (270kB)\n",
      "Requirement already satisfied: scipy>=0.14 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Requirement already satisfied: h5py in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Requirement already satisfied: six>=1.9.0 in /usr/local/lib/python3.5/dist-packages (from keras)\n",
      "Collecting keras-preprocessing==1.0.1 (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/f8/33/275506afe1d96b221f66f95adba94d1b73f6b6087cfb6132a5655b6fe338/Keras_Preprocessing-1.0.1-py2.py3-none-any.whl\n",
      "Collecting keras-applications==1.0.2 (from keras)\n",
      "  Downloading https://files.pythonhosted.org/packages/e2/60/c557075e586e968d7a9c314aa38c236b37cb3ee6b37e8d57152b1a5e0b47/Keras_Applications-1.0.2-py2.py3-none-any.whl (43kB)\n",
      "Building wheels for collected packages: pyyaml\n",
      "  Running setup.py bdist_wheel for pyyaml: started\n",
      "  Running setup.py bdist_wheel for pyyaml: finished with status 'done'\n",
      "  Stored in directory: /root/.cache/pip/wheels/ad/da/0c/74eb680767247273e2cf2723482cb9c924fe70af57c334513f\n",
      "Successfully built pyyaml\n",
      "Installing collected packages: pyyaml, keras-preprocessing, keras-applications, keras\n",
      "Successfully installed keras-2.2.0 keras-applications-1.0.2 keras-preprocessing-1.0.1 pyyaml-3.13\n",
      "You are using pip version 9.0.1, however version 10.0.1 is available.\n",
      "You should consider upgrading via the 'pip install --upgrade pip' command.\n",
      "2018-07-14 01:37:43.291928: I tensorflow/core/platform/cpu_feature_guard.cc:137] Your CPU supports instructions that this TensorFlow binary was not compiled to use: SSE4.1 SSE4.2 AVX AVX2 FMA\n",
      "Train on 54000 samples, validate on 6000 samples\n",
      "Epoch 1/1\n",
      "\n",
      "  128/54000 [..............................] - ETA: 5:28 - loss: 2.3091 - acc: 0.1250\n",
      "  256/54000 [..............................] - ETA: 4:38 - loss: 2.2554 - acc: 0.1211\n",
      "  384/54000 [..............................] - ETA: 4:25 - loss: 2.1936 - acc: 0.1589\n",
      "  512/54000 [..............................] - ETA: 4:20 - loss: 2.1520 - acc: 0.1895\n",
      "  640/54000 [..............................] - ETA: 4:17 - loss: 2.1030 - acc: 0.2188\n",
      "  768/54000 [..............................] - ETA: 4:15 - loss: 2.0351 - acc: 0.2565\n",
      "  896/54000 [..............................] - ETA: 4:13 - loss: 2.0135 - acc: 0.2667\n",
      " 1024/54000 [..............................] - ETA: 4:11 - loss: 1.9821 - acc: 0.2852\n",
      " 1152/54000 [..............................] - ETA: 4:08 - loss: 1.9363 - acc: 0.3099\n",
      " 1280/54000 [..............................] - ETA: 4:07 - loss: 1.8913 - acc: 0.3305\n",
      " 1408/54000 [..............................] - ETA: 4:06 - loss: 1.8613 - acc: 0.3438\n",
      " 1536/54000 [..............................] - ETA: 4:05 - loss: 1.8418 - acc: 0.3503\n",
      " 1664/54000 [..............................] - ETA: 4:04 - loss: 1.8098 - acc: 0.3666\n",
      " 1792/54000 [..............................] - ETA: 4:03 - loss: 1.7659 - acc: 0.3823\n",
      " 1920/54000 [>.............................] - ETA: 4:01 - loss: 1.7355 - acc: 0.3885\n",
      " 2048/54000 [>.............................] - ETA: 4:01 - loss: 1.7017 - acc: 0.4009\n",
      " 2176/54000 [>.............................] - ETA: 4:00 - loss: 1.6676 - acc: 0.4127\n",
      " 2304/54000 [>.............................] - ETA: 4:00 - loss: 1.6366 - acc: 0.4249\n",
      " 2432/54000 [>.............................] - ETA: 3:59 - loss: 1.6042 - acc: 0.4375\n",
      " 2560/54000 [>.............................] - ETA: 3:59 - loss: 1.5827 - acc: 0.4449\n",
      " 2688/54000 [>.............................] - ETA: 3:58 - loss: 1.5640 - acc: 0.4513\n",
      " 2816/54000 [>.............................] - ETA: 3:57 - loss: 1.5430 - acc: 0.4585\n",
      " 2944/54000 [>.............................] - ETA: 3:57 - loss: 1.5205 - acc: 0.4671\n",
      " 3072/54000 [>.............................] - ETA: 3:55 - loss: 1.4985 - acc: 0.4749\n",
      " 3200/54000 [>.............................] - ETA: 3:55 - loss: 1.4744 - acc: 0.4828\n",
      " 3328/54000 [>.............................] - ETA: 3:54 - loss: 1.4552 - acc: 0.4904\n",
      " 3456/54000 [>.............................] - ETA: 3:53 - loss: 1.4363 - acc: 0.4977\n",
      " 3584/54000 [>.............................] - ETA: 3:53 - loss: 1.4161 - acc: 0.5067\n",
      " 3712/54000 [=>............................] - ETA: 3:52 - loss: 1.3991 - acc: 0.5132\n",
      " 3840/54000 [=>............................] - ETA: 3:52 - loss: 1.3832 - acc: 0.5167\n",
      " 3968/54000 [=>............................] - ETA: 3:51 - loss: 1.3695 - acc: 0.5217\n",
      " 4096/54000 [=>............................] - ETA: 3:51 - loss: 1.3520 - acc: 0.5286\n",
      " 4224/54000 [=>... \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                          ... ===========>.] - ETA: 4s - loss: 0.5864 - acc: 0.7932\n",
      "53120/54000 [============================>.] - ETA: 4s - loss: 0.5858 - acc: 0.7935\n",
      "53248/54000 [============================>.] - ETA: 3s - loss: 0.5855 - acc: 0.7935\n",
      "53376/54000 [============================>.] - ETA: 2s - loss: 0.5853 - acc: 0.7935\n",
      "53504/54000 [============================>.] - ETA: 2s - loss: 0.5850 - acc: 0.7937\n",
      "53632/54000 [============================>.] - ETA: 1s - loss: 0.5844 - acc: 0.7939\n",
      "53760/54000 [============================>.] - ETA: 1s - loss: 0.5844 - acc: 0.7939\n",
      "53888/54000 [============================>.] - ETA: 0s - loss: 0.5842 - acc: 0.7939\n",
      "54000/54000 [==============================] - 254s 5ms/step - loss: 0.5841 - acc: 0.7939 - val_loss: 0.3561 - val_acc: 0.8700\n",
      "/usr/local/lib/python3.5/dist-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n",
      "Using TensorFlow backend.\n",
      "Test loss: 0.3804996592760086\n",
      "Test accuracy: 0.8649\n",
      "Model saved to file: /mnt/results/training-results-b467af13-ab67-4b5d-b0ca-c522fa0f4802/training-lh276tdmR/keras_original_model.hdf5\n",
      "Model definition saved to file: /mnt/results/training-results-b467af13-ab67-4b5d-b0ca-c522fa0f4802/training-lh276tdmR/keras_original_model.json\n",
      "Training process finished. Exit code: 0\n",
      "\n"
     ]
    }
   ],
   "source": [
    "if \"Model ID\" in out[1]:\n",
    "    model_id = out.fields()[1][-1]\n",
    "    !{ffdl} logs --follow {model_id}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3.4.  Cancel the Training Job (Optional)<a id=\"cancel\"></a>\n",
    "Interrupt the execution of the cell above which should be following the training logs, then uncomment the code in the cell below and run it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "# !{ffdl} halt {model_id}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Generate Adversarial Samples for a Model Robustness Check<a id=\"art\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After successfully training our model, we can now use the Fast Gradient Method (FGM) to generate adversarial samples for the trained model. The adversarial samples are then used to check the robustness of the trained model. Both the trained model as well as the dataset used to train the model are required inputs to the robustness check."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- [4.1. Download and evaluate the trained model](#eval_model)\n",
    "- [4.2. Craft adversarial samples](#craft_adv_samples)\n",
    "- [4.3. Generate robustness metrics](#robustness_check)\n",
    "- [4.4. Show model predictions on adversarial samples](#show_adv_samples)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.1. Download and Evaluate the Trained Model <a id=\"eval_model\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "weights_filename            = \"keras_original_model.hdf5\"\n",
    "network_definition_filename = \"keras_original_model.json\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print contents of the `training_result_bucket`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "training-results-b467af13-ab67-4b5d-b0ca-c522fa0f4802\n",
      "  File: _wml_checkpoints/, 0.00kB\n",
      "  File: training-lh276tdmR/_submitted_code/model.zip, 2.75kB\n",
      "  File: training-lh276tdmR/keras_original_model.hdf5, 14092.55kB\n",
      "  File: training-lh276tdmR/keras_original_model.json, 2.75kB\n",
      "  File: training-lh276tdmR/learner-1/, 0.00kB\n",
      "  File: training-lh276tdmR/learner-1/.log-copy-complete, 0.00kB\n",
      "  File: training-lh276tdmR/learner-1/training-log.txt, 42.02kB\n"
     ]
    }
   ],
   "source": [
    "print_bucket_contents([training_result_bucket])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Download network definition and weights to current working directory"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloaded keras_original_model.hdf5\n",
      "Downloaded keras_original_model.json\n"
     ]
    }
   ],
   "source": [
    "weights_file_in_cos_bucket = os.path.join(model_id, weights_filename)\n",
    "network_definition_file_in_cos_bucket = os.path.join(model_id, network_definition_filename)\n",
    "\n",
    "bucket_obj = cos.Bucket(training_result_bucket)\n",
    "\n",
    "bucket_obj.download_file(weights_file_in_cos_bucket, weights_filename)\n",
    "print('Downloaded', weights_filename)\n",
    "\n",
    "bucket_obj.download_file(network_definition_file_in_cos_bucket, network_definition_filename)\n",
    "print('Downloaded', network_definition_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load & compile the model that we created using `convolutional_keras.py`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Network Definition: keras_original_model.json\n",
      "Weights:            keras_original_model.hdf5\n"
     ]
    }
   ],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from keras import backend as K\n",
    "from keras.models import model_from_json\n",
    "\n",
    "print('Network Definition:', network_definition_filename)\n",
    "print('Weights:           ', weights_filename)\n",
    "\n",
    "# load model\n",
    "json_file = open(network_definition_filename, 'r')\n",
    "model_json = json_file.read()\n",
    "json_file.close()\n",
    "\n",
    "model = model_from_json(model_json)\n",
    "model.load_weights(weights_filename)\n",
    "comp_params = {'loss': 'categorical_crossentropy',\n",
    "                       'optimizer': 'adam',\n",
    "                       'metrics': ['accuracy']}\n",
    "model.compile(**comp_params)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the test data and labels from `.npz` file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "from keras.utils import np_utils\n",
    "\n",
    "f = np.load(dataset_filename)\n",
    "x_original = f['x_test']\n",
    "y = f['y_test']\n",
    "f.close()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Visualize an original (non-adversarial) sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAJIAAACPCAYAAAARM4LLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAC15JREFUeJztnVtoVtkVx//LeL/H2xgv6ETjJQgqDFKxSMUqVhBftEweah8G8mCLFfrQGfsgiII+2LcKBqujUKYWWhjRUbFiLIOXRGW0jpKLxXEyqDEq3q/j7kM+073+yfed77K/y9H1A/H8z/m+c3bCyt7rrL322uKcg2HkSq9iN8B4NzBDMoJghmQEwQzJCIIZkhEEMyQjCGZIRhDMkIwg5GRIIrJMRJpEpFVEPg3VKCN+SLaRbREpA9AMYAmANgCNAGqcc1dSfKdkw+i9eum/qUGDBin96NGjrO89cOBApX/88UelX7x4kfW9C0CHc2501Id65/CAeQBanXP/BQAR+RuAlQCSGlIpw4Yzb948pY8fP571vWfMmKH048ePlW5ubs763gXgu3Q+lMvQNh7A955uS5xTiEitiJwTkXM5PMsocXLpkdLCOVcHoA4o7aHNyI1cDOkHABM9PSFxriTo37+/0uvXr1e6pqZG6fLycqVHj9ZuwdOnT5UeMWJE2m15/vy50s+ePVOafaaTJ08qvWvXLqWPHDmS9rMLRS5DWyOAKhH5UET6AvgYwIEwzTLiRtY9knPutYj8FsBRAGUAdjvnvg3WMiNW5OQjOee+AvBVoLYYMSbrOFJWD8ujs71t2zala2trlR4yZIjS7KewfvXqldIDBgxQuk+fPl3HZWVl6trLly+VZv+KY1b9+vVL+Sy+/+nTp5VeuHAh8sh559xHUR+yKRIjCGZIRhDMkIwgxNZHYh9o586dSt+6dUvp169fZ3T/vn37Ks2xHh/+Hb5580Zp359K5/vcVn72hAkTlD58+LDSK1asSPm8DDEfySgcZkhGEMyQjCDE1ke6ffu20jy3xqkaHLsZO3Zsyvvfv39fac4Z8v0YTkHhtty9e1dpjguxD8RxJRFRmuNUgwcPVnrKlCldxx0dHcgR85GMwmGGZAQh7/lI+WLYsGFK89ATNZTt2LFD6bq6OqXPnz+v9M2bN5X2X8E5DffGjRtKjxkzRmkemioqKpRua2tTmn+2oUOHKs1TKpWVlV3HAYa2tLAeyQiCGZIRBDMkIwix9ZH4FZnTWfmVmdmwYYPSDx48UJpf0XlJUX19fdfxokWLUj7ryhW9sGbmzJlKs8+zbt06pTdv3qz0nTt3lGZ/cMGCBV3HDQ0NKdsWCuuRjCCYIRlBMEMyghCrKRI/tYNjKzylwT7S8OHDlT5wQC94WblypdJRvxf//ps2bVLXHj58qPSxY8eU5qVM7e3tSvPP1tLSojRPuXAa8f79+7uO16xZ063tGWJTJEbhMEMygmCGZAQhVnGkcePGJb3G6a08/8SMH9+t3kVKVq9enfTavn37lOaYFsekLl68qDTPtXEKTKZUVVXl9P1ssB7JCIIZkhEEMyQjCLHykUaNGpX2Z3kJEC/BZh+J56sYLjXjc/ToUaX9fCCge9xn+fLlSp84cUJp9qGi0oZ5+VJUGnE+sB7JCEKkIYnIbhFpF5HL3rkRInJMRFoS/5enuofx7pNOj/Q5gGV07lMAx51zVQCOJ7TxHhPpIznn/i0ik+n0SgA/SxzvBVAP4A8B29UjvFTZJyr/iEvLsB/BcSi+3/Tp05XeunVr17G//Kcnrl69qjRXuZ00aZLSa9euVXr+/PlK37t3T2nOAc80RhaCbH2kD5xzb7PhbwH4IFB7jJiS81ubc86lmtUXkVoAtcmuG+8G2fZIt0WkAgAS/7cn+6Bzrs4591E6qQhGfMm2RzoA4NcAtib+/zJYi1LAJYt92Mfh+S3WHJvZsmWL0hyHWrp0qdKzZ8/uOp41a5a6xvlB7BP5/hWg84cAYM6cOUgF/yyZltHJB+m8/n8B4DSA6SLSJiKfoNOAlohIC4CfJ7TxHpPOW1tNkkuLA7fFiDEW2TaCEKu5Ns7b8WE/geej2G/gdWy8zo3hz/tldaqrq1N+l8sQsq/H+UsM549H+UipPpuqhGEuWI9kBMEMyQiCGZIRhFj5SKniSAzPP/EOkLztAtckYl+CyyX37v3/X13UNqXsn7HPxKUC+X7sn3GcifOdfCZPnqz0tWvXUrY1W6xHMoJghmQEIVZDGy+79uHKrjxU7d27V2lOd+U0E4bDCX6aiT/M9QS/vvNQxyV6OHV2z549SkdNofhwerINbUZJY4ZkBMEMyQhCrHwkvxwM+x1cmo/L43HZG4bDBezH5FL+J2qKg69zqOHs2bMZ3d/fDTMqBTkU1iMZQTBDMoJghmQEIVY+kh9H4vJ4UbsjcUliJmpKhMnEZ2I/hb/LmuNlmZQhBHTMK5NppVywHskIghmSEQQzJCMIsfKR/PhLlN/Q1NSkdNSyar4fz63x9UziM1FxJPb3eAsxLp/M8P38tmVSCigXrEcygmCGZATBDMkIQqx8JD/vJ2pZTXNzs9KcWpvq3j3BPpGvc4nzAN3zjxjOrWI9cuTIpN/l5eP5wnokIwhmSEYQzJCMIMTKR/LzbKJ8JF7GzKVluFxyVHnkTIiKQXHbon6WqVOnKs3LmbiMoZ9bxXla+cJ6JCMI6dRHmigiJ0Tkioh8KyK/S5y3EslGF+n0SK8B/N45Vw3gJwB+IyLVsBLJhkc6hbZuAriZOH4kIlcBjEcRSiT7vgTPLzEcF+JYC69ji7pfKjLN544qU8jwNqnXr19Xeu7cuUnvX15emIEiIx8pUW97LoCzsBLJhkfab20iMhjAPwCsd849pMhu0hLJVh75/SCtHklE+qDTiP7qnPtn4nRaJZKtPPL7QWSPJJ1dz18AXHXO/cm7VPASyb6PxDnaDOdocw425wCxTxW1pUSqa1E52kyUj8SlaS5duqT0qlWrkn63UKWS0xnaFgD4FYD/iMg3iXMb0GlAf0+US/4OwC/z00QjDqTz1vY1gGR/jlYi2QBgkW0jELGaa/PnkKJypjl+wtu381r/VCWGo65HrVOLyvdmzaX+eJstzrVK1Z6obetDYT2SEQQzJCMIZkhGEGLlI/k5RH5uEtC9huT27duVXrxYv2Cy75Dp1gq+H5LpmreobR2GDh2qdH19vdIHDx5UeuPGjUnvF1XDIBTWIxlBMEMyghCroc1PG+XhgFNnuUvv6OhQuqqqSmkuG5xJ6m3UUMbXOZTAy5H8EodA9yXb/LMw/u+Gd/DOF9YjGUEwQzKCYIZkBCFWPtKpU6e6jnnagHdh5GmEadOm5a9hBaayslJp3k3J35KisbGxIG2yHskIghmSEQQzJCMIsfKRGhoauo55KXKmaSFxJmqbLj+GxmWi84X1SEYQzJCMIJghGUGIlY/kl7y7cOGCusZxpCdPnqS8Fy8/4rm7Qm1P1RP8bG5ba2ur0ocOHVLaL6985syZwK3rGeuRjCCYIRlBMEMygiC5bLGZ8cNE7qBzVe4oAKmTaopHqbatWO2a5JyL3KuroIbU9VCRc6VaVKJU21aq7XqLDW1GEMyQjCAUy5DqivTcdCjVtpVquwAUyUcy3j1saDOCUFBDEpFlItIkIq0iUtRyyiKyW0TaReSyd64kaofHsbZ5wQxJRMoA/BnALwBUA6hJ1OsuFp8DWEbnSqV2ePxqmzvnCvIPwHwARz39GYDPCvX8JG2aDOCyp5sAVCSOKwA0FbN9Xru+BLCkVNvnnCvo0DYewPeebkucKyVKrnZ4XGqbm7OdBNf5Z1/UV1qube5fK4X2+RTSkH4AMNHTExLnSom0aocXglxqmxeDQhpSI4AqEflQRPoC+BidtbpLibe1w4EC1Q7viTRqmwNFbF+PFNhpXA6gGcA1AH8ssgP7BTo363mFTn/tEwAj0fk21ALgXwBGFKltP0XnsHUJwDeJf8tLpX09/bPIthEEc7aNIJghGUEwQzKCYIZkBMEMyQiCGZIRBDMkIwhmSEYQ/ge/lt3UHfc3WgAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 144x144 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "%matplotlib inline\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "\n",
    "plt.figure(figsize=(2, 2))\n",
    "plt.imshow(x_original[1], cmap='gray')\n",
    "print(y[1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Standardize the Numpy array"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "x_original = np.expand_dims(x_original, axis=3)\n",
    "x_original = x_original.astype('float32') / 255\n",
    "y = np_utils.to_categorical(y, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Evaluate the model and calculated test accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "model test loss:     38.04996589422226\n",
      "model test accuracy: 86.49\n"
     ]
    }
   ],
   "source": [
    "scores = model.evaluate(x_original, y, verbose=0)\n",
    "print('model test loss:    ', scores[0]*100)\n",
    "print('model test accuracy:', scores[1]*100)\n",
    "model_accuracy = scores[1]*100"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.2. Craft Adversarial Samples <a id=\"craft_adv_samples\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After loading & compiling the model, we can create a `KerasClassifier` from the ART library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "from art.classifiers.keras import KerasClassifier\n",
    "\n",
    "classifier = KerasClassifier((0, 1), model)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ART exposes many attacks like FGM, NewtonFool, DeepFool, Carlini etc. Let's use the Fast Gradient Method (FGM) to craft adversarial samples based on the test data (`x_test`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of adversarial samples crafted: 10000\n",
      "adversarial samples saved to: adv_samples.npz\n"
     ]
    }
   ],
   "source": [
    "from art.attacks.fast_gradient import FastGradientMethod\n",
    "\n",
    "# configuration\n",
    "epsilon = 0.2\n",
    "\n",
    "# create crafter object\n",
    "crafter = FastGradientMethod(classifier, eps=epsilon)\n",
    "\n",
    "# craft samples on x_test (stored in variable x_original)\n",
    "x_adv_samples = crafter.generate(x_original)\n",
    "\n",
    "# save the adversarial samples\n",
    "adv_samples_filename = \"adv_samples.npz\"\n",
    "np.savez(adv_samples_filename, x_original=x_original, x_adversarial=x_adv_samples, y=y)\n",
    "\n",
    "print(\"Number of adversarial samples crafted:\", len(x_adv_samples))\n",
    "print(\"adversarial samples saved to:\", adv_samples_filename)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.3. Generate Model Robustness Metrics <a id=\"robustness_check\"></a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The following functions can be used for gathering metrics like model robustness, confidence metric, perturbation metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy.linalg as la\n",
    "import json\n",
    "\n",
    "\n",
    "def get_metrics(model, x_original, x_adv_samples, y):\n",
    "    scores = model.evaluate(x_original, y, verbose=0)\n",
    "    model_accuracy_on_non_adversarial_samples = scores[1] * 100\n",
    "\n",
    "    y_pred = model.predict(x_original, verbose=0)\n",
    "    y_pred_adv = model.predict(x_adv_samples, verbose=0)\n",
    "\n",
    "    scores = model.evaluate(x_adv_samples, y, verbose=0)\n",
    "    model_accuracy_on_adversarial_samples = scores[1] * 100\n",
    "\n",
    "    pert_metric = get_perturbation_metric(x_original, x_adv_samples, y_pred, y_pred_adv, ord=2)\n",
    "    conf_metric = get_confidence_metric(y_pred, y_pred_adv)\n",
    "\n",
    "    data = {\n",
    "        \"model accuracy on test data:\": model_accuracy_on_non_adversarial_samples,\n",
    "        \"model accuracy on adversarial samples\": model_accuracy_on_adversarial_samples,\n",
    "        \"reduction in confidence\": conf_metric * 100,\n",
    "        \"average perturbation\": pert_metric * 100\n",
    "    }\n",
    "    return data\n",
    "\n",
    "\n",
    "def get_perturbation_metric(x_original, x_adv, y_pred, y_pred_adv, ord=2):\n",
    "\n",
    "    idxs = (np.argmax(y_pred_adv, axis=1) != np.argmax(y_pred, axis=1))\n",
    "\n",
    "    if np.sum(idxs) == 0.0:\n",
    "        return 0\n",
    "\n",
    "    perts_norm = la.norm((x_adv - x_original).reshape(x_original.shape[0], -1), ord, axis=1)\n",
    "    perts_norm = perts_norm[idxs]\n",
    "\n",
    "    return np.mean(perts_norm / la.norm(x_original[idxs].reshape(np.sum(idxs), -1), ord, axis=1))\n",
    "\n",
    "\n",
    "# This computes the change in confidence for all images in the test set\n",
    "def get_confidence_metric(y_pred, y_pred_adv):\n",
    "\n",
    "    y_classidx = np.argmax(y_pred, axis=1)\n",
    "    y_classconf = y_pred[np.arange(y_pred.shape[0]), y_classidx]\n",
    "\n",
    "    y_adv_classidx = np.argmax(y_pred_adv, axis=1)\n",
    "    y_adv_classconf = y_pred_adv[np.arange(y_pred_adv.shape[0]), y_adv_classidx]\n",
    "\n",
    "    idxs = (y_classidx == y_adv_classidx)\n",
    "\n",
    "    if np.sum(idxs) == 0.0:\n",
    "        return 0\n",
    "\n",
    "    idxnonzero = y_classconf != 0\n",
    "    idxs = idxs & idxnonzero\n",
    "\n",
    "    return np.mean((y_classconf[idxs] - y_adv_classconf[idxs]) / y_classconf[idxs])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Display the robustness check metrics\n",
    "\n",
    "1. Model accuracy on test data\n",
    "2. Model robustness on adversarial samples\n",
    "3. Reduction in confidence\n",
    "4. Perturbation metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "    \"model accuracy on test data:\": 86.49,\n",
      "    \"model accuracy on adversarial samples\": 13.71,\n",
      "    \"reduction in confidence\": 49.82583820819855,\n",
      "    \"average perturbation\": 43.59939992427826\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "result = get_metrics(model, x_original, x_adv_samples, y)\n",
    "\n",
    "print(json.dumps(result, indent=4, sort_keys=False))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4.4. Show Model Predictions on Adversarial Samples <a id=\"show_adv_samples\"></a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "# https://keras.io/datasets/#fashion-mnist-database-of-fashion-articles\n",
    "\n",
    "fashion_labels = {\n",
    " 0: \"T-shirt/top\",\n",
    " 1: \"Trouser\",\n",
    " 2: \"Pullover\",\n",
    " 3: \"Dress\",\n",
    " 4: \"Coat\",\n",
    " 5: \"Sandal\",\n",
    " 6: \"Shirt\",\n",
    " 7: \"Sneaker\",\n",
    " 8: \"Bag\",\n",
    " 9: \"Ankle boot\"\n",
    "}\n",
    "\n",
    "def get_label(y):\n",
    "    if \"fashion\" in dataset_filename:\n",
    "        return fashion_labels[y]\n",
    "    else:\n",
    "        return \"Predict: %i\" % y"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compare original images in the top row with the adversarial samples in the second row and test model predictions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAA1oAAAC/CAYAAADnw60yAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMi4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvhp/UCwAAIABJREFUeJzsnXeYJFXV/z9tQtGXDAZyRsISlxwkg0jOKBkTKuKrIiLaDEGEnyKCCVSigUUEiQKSJEcJu8CScwbBV8EI9fuj53v7W9t3untmume6d8/neXi2ONNdXXXq3ltV93zvOZWiKAiCIAiCIAiCIAg6x9vG+wCCIAiCIAiCIAimN+JFKwiCIAiCIAiCoMPEi1YQBEEQBEEQBEGHiRetIAiCIAiCIAiCDhMvWkEQBEEQBEEQBB0mXrSCIAiCIAiCIAg6TLxoBUEQBEEQBEEQdJh40QqCIAiCIAiCIOgw8aIVBEEQBEEQBEHQYd4xnA9XKpWiWwcyHfByURRzD/dLY+HTt72t/j793ve+F4C//e1vbX9/5plnBuDNN99Mtn/9618dOrqm9KxP/+d//idtzzPPPAD84x//SLZ3vKPWtdxPb3/72xu2i6J+qDPNNBMAjzzySBeOONFzPpWv5pprrmRTW3P/TPu3af9eqVRK/wL897//BcrtPbfPUdJzPs2hvu/jQc5njuzyI8Df//73bh2i07M+XWKJJRpsb731VtqWf73vu8/f+c53AuV2nPP/ww8/PPqDLdOzPnU0Nrp/NDZqrIByP5b///nPf47FIToj8imM3/PUnHPOmbY1JuTGTKj785VXXhmjo6tRFEV+QGpBPKM2pS/6f5/Rlk+H9aIVNOWJ8T6AodBgCrDqqqsCcOWVV7b9/aWWWgooP2A9+OCDHTq6pvSET/0mpJu7/AhwwAEHAHDXXXcl2wc+8AGg/LD0vve9L23PPvvsAPznP/9JtkUWWQSAbbfdtmPHnqEnfOrIF/vuu2+y/fWvfwXKL6/T/g3KD1t6QHvXu96VbC+++CIA11xzTbL9+9//7sBRl+iKT/Vw7g/xuQfydl8cl1tuOaA8HshXPgng6AH3pZdeSrZrr722rd8bJT3XTsXJJ5+ctvVQ6i9V7373uwF4/PHHG2wA73//+4HyeCr/+wvZFlts0cGjBnrEp7nx1JllllkAePXVV5NtvvnmA/KTMVB/IZgyZUpnD7Y1PeFTyE+guI+Et6s11lgDKL/Aut/vv/9+AE499dSG/bS6jsP9XNB1eqatTke05dOQDgZBEARBEARBEHSYynBmGCKE2JQ7iqJYZbhf6pRPNWN64IEHJtuuu+4K1CMGAHPPXYtyvvHGG8k2xxxzNN23Zgs9uqCZsj/96U/J9vOf/xyASy+9dPgnkGdcfSp8plDRheuuuy7Z1l577SG/+3//939pWxJMqM8g+nXQ37fccstku+iii0Z62EPREz51PvvZzwLw/e9/P9n+8pe/APDcc88lmyJ+Tz/9dLI99NBDafvDH/4wUJYPXXHFFQDcc889yXbmmWd27NgH6YpPh5LzQX5m2OWsG2ywAQArrbRSsm2++eYAPPDAAw378WirS4tefvllAN7znvckm6IvF154YbJdcMEFADz55JNDHvMw6bl2qkiLS3sVMXXUj33c8DapsdP7viKHvr8NN9ywE4ftjKtPc5JAtXGPCEpa6f5R+3vttdcaPgf1yOLPfvazZDvooIM6cditGJFPYeyfpyZMmADA3XffnWw33ngjUI6au3RQ9zaPyOaiZLqOnYpYhXSwK/TcmDod0JZPI6IVBEEQBEEQBEHQYWKNVh9zzDHHpO1PfepTQHlWWxEoj0QpUuAz1For4Os0fB2LZhZ9hlYzsB/72MeSbeuttwbgpptuSrZ11113eCfVg/hsn1hhhRXStnyq2X+oz2q79t0XFGvW0KMWiy22GFBfEwddiWj1HEom4mtacrOmim55O/XoiyIOHkX80Ic+BMDUqVM7d8BjhGaHW61xUN/3JA3ykZ/3pEmTgHLbVSTB26lHvORLjy4oKr7gggsm23HHHdfwuYMPPhiAZ599dshz7Cc0q+/XQH7z8VLbvtbF26zaqe9HbT+3JnF6Idend955ZwAOP/zwZFPkZYcddki27373uwCsuOKKybbRRhulbUWuf/zjHyebro1HaDodeelVdA/RekCAF154AYDVVlst2QYGBoB6m4RyH95vv/2A8n1cUS5//ujCutcgmG6IiFYQBEEQBEEQBEGHiRetIAiCIAiCIAiCDhPSwT5EUiFf7Pv8888DrWvcKJ2zL87Wdq4uCZQXHU/7Hf89SUPWXHPNZNOCeU/wMD3gyQMkGXT5Ra6WjsuHJL3M1SObf/75O3uwPY7kf55CXIkvJMuEuizW29xss82WtiULcvms2vHkyZM7fdhdp5nMSQlEoO4/l16qbIDLfZVowRPYqJSAxg8ot0n9tvtPSTW8xINS7ruc8MgjjwRgn332GeoU+4rtt98eKCcPeuqpp4Cy9LJVHS1JEP07s846KwAf/OAHk23llVcG4I477ujMCfQgkvU988wzyaZ2c8kllyTbZpttBsDCCy+c3Y/6g/eBHNOjZFDtZJtttkk2taMbbrgh2TRWuoRdMmHJt6EsHVTiDC+ZITmxP3+ofIZLlV1KHwQzMhHRCoIgCIIgCIIg6DAR0epDjjjiCKC86F8z9z5LqqK5jhZo59K5eiFTT+eqGTCPyCh6pcgM1GfgtegW6otovdBkP890+eJioeiBz5ZqBtt95ouy5X//jq6nzy7OCDzxRK3m3/LLL59s8o+3U820+sJrjxQoKuMRB/29H5Nh5CJainYusMACyfboo48C5SireP3119O22q6nJ9d3F1988WTzGe9bb70VKC+GV/TBxwgl1/FkDhp/dt9992RTav1+LGKqgtpeckBRWO+z6ucqsgvlKIHatKsK9B0fX1QUvV8iWrlr6pEQlRrwKLTuH0oEBLDMMssA8NGPfjTZlNbdfe/JX8SSSy7ZsG9PxiJ1ht+jcsmOeh1PRHHllVcC5fuqxrt777032RZaaCEA9thjj2RT2/IEON6vt9pqKwAuu+yyZFMR49VXXz3ZNt54Y6BeABngvPPOA+Dhhx9u/8SCYDokIlpBEARBEARBEAQdJl60giAIgiAIgiAIOkxIB/sQLZzOLbZ2uaBqipx88snJJqmASzAkcfnb3/6WbE8++WTalizGJVtabPv0008nm47Hk0JIUqTkBtDf0sFll122wSbpoNcmk7TSa8e4zE24tFD+c5nljICkO/fcc0+ySfLmcqRFF10UgNlnnz3Z/O8PPfRQw74ljXPZZr+QkzRJYuXnI7mwJwmRbMrbl/7u0i0lHPj2t7+dbC7/075dkizZlUuN1eddKqb27LWPJB3sF7mgI1maS/nU5z1hkPq5yzbdL0IJRHzbr7lqwPULuWu69NJLp+2JEycCZZma+qySLkD9fuRJbZTo4c4770w2Hyd1HdznShLjskSN1foX+ut+pPuPJH0AX/va14ByIhCNDxr//O8+fp566qlA+f7s9zHV3LvllluSTTUiXZIpObHX6Pvyl78MlBP3BMGMSES0giAIgiAIgiAIOkxEtPoQzVb7Ymqf2ReHHHIIUJ451Qy3ZqWgnpp1/fXXz/7efffdB8CHP/zhZNMM9gEHHJBsSsvrabo1u7vWWmslmxbY9yMTJkwAytE9XQf3qa6RR/c8Vbnw66bv+KzsjIBmwj06qjbn7LDDDkB9phrqC+cBrr32WqAccdBMq0cUPDFBv6Hz9b7vCWmE2lAugY23SUW2L7/88mTzaJm+4wva1WY9eq6Ily+kF4pk9COebl3nqDT5UI/2ezRHY4OXafDrpciiR8G0b/9crvRDv+HRE7Uhj4TKl94mlYzFI02rrLIKUE8QAjBlypS0PffccwPlKJgSP3lyF0UMPWrTT8gPSncPsPfeewPl9O7ynScBUkTWo2HyuxJlQDmxixKOeJuXTQoDqPvYx+2LL764/RMLgumYiGgFQRAEQRAEQRB0mHjRCoIgCIIgCIIg6DAzlHTQZTS5OkbCpTiSb/iC2vGoC5FbTO0Lp3PyoTPOOAOArbfeuuFvXmtIksHDDz882bxG16677trwHdXxmTRpUrJJOuhJHyQ98gXx/YykK+57SQZdcqWEJX/+85+TzRcKS9bi8iDt56mnnur0Yfc0qsuy4YYbNtjcP5KluPT0pJNOStvym0sQ5WdP8NDPKFGAy4FzfV9SH5ezSp7msldJET0RifdzLXj3xAxKpuE1nyRBdCnnY489BpQlsxrH/Bh6GT/HnKRXMkpvX5K23n777cnmSXQknfPkQxozfQxxGWG/oZpuLuVTW/L70eTJk4G85NSTu0hm6ZI/T2gh//n9XBJhlwqrP3i/6Cc22GADoN63oJ5IxO/Z8p3LKxdccEGgnAhLNbj8+cYlrcsttxxQXg6gPuG1yDxZjtBYNb3U0AyCkRIRrSAIgiAIgiAIgg7TdxEtzSB6EgGPLsw777xAuUL5H/7wB6D9JAO5Rcjbb7992vaq7GNFLtWvn3duca98kWPHHXdssCkCBuXZVEUCPQWvFon7rGMzFl988bY+1+soIYjPpuo6aBYX6rOGq6++erL5bKtmYD36p1nBXNKM6RnNLnv/VKIFRaQcnz31aI586W1XEQKfMe+3JAMeVRHe1pRwwKNSap8exRc+bsgXnrTAo+caZ93n6vvuR33HU8cLb+NKJuPRnl5GCQSg3pZy9xHv2/KPRwk8LbmSCXgJDV0vLwfRb+3UUTvw/qkIiLdnJV5wn8rP3o8V/fNx158BNGZ6pCc3xipakxtD+sHfSl7hiVbUlzxSpXN67bXXkk3XxKOmUudIgQHl6Kzaqicr0Zjs1+dPf/oTUH5OUvv35EUR0QpyeF/Wdq68SSvWXXfdtK3kWJ1CSoSRJCuLiFYQBEEQBEEQBEGHiRetIAiCIAiCIAiCDtN30kExVFhxnXXWAWC11VZLNsnuTjjhhLb27XUkNt10U6C80HQ88AWlOSSJcGmFpIMunRAK9TuXXXZZ2vZK8aqR8dGPfjTZrr76aqAsJ5SM0H9PMgWvudPPSGLh8oucdPDcc89tuh9JulwqJHKJT6ZnFIr3BeryqUtmJfdxGZZLtiSf9cXc8rP3i35j4YUXTtvqYy7JkqTBfaGEFu6LXMIB9VVvhz62qj6Ro992+ZWunSd40Oe8r+hc+kU6uNRSS6VttVOvA6VzczlcTh518803p+3ll18eyCczytXj6kc0Tvo5SMbmcmCdt8ul1SbdP2q7Lmtz6Zr+7hJ6JcHw+7narCeTkSzOEz70KvKhS30333xzoNzu5AdPWKG+5zWztO01Mr3umJ4DfvGLXySbxmS1Y4D11lsPgDXXXDPZ5P9csp4gcHzcyyWoE/4Mr4RsANdddx1QTqglGXGr5GIaE/w+Jb761a+mbS23UUIaaH/pTES0giAIgiAIgiAIOkzfRbQ0Q+1vn6qWDvWZGZ/JUSKG8847L9k0g+YzYE888QRQXryp2S5PGT0eKFWq4wsIhaeyVRTJZwb1HV/k/Z3vfAcoV3p3lGrbZ3eVKnb//fdPNiUg8dlJzWg2S8zRT2h21P2cm4H5zW9+02DzxdaKOPjsoejX1MMjRb70dpqbKZLtrrvuyu5HfdlnuuXzfo5o+cydzi0XpfbPaSzziILGTk+QIZ/6eOr7yY238mkuQYb3C/ncfa/F9f2CJ7RQFMQjzvKLR15PO+20hv14ROAzn/kMkE9U4n7ORbv7BfVFb386H7/nSqmhcgRQH09z46r7x/2n/uBjbC6pRs6n/ZRG/4477gDg9NNPTzZFkTzyrWcY9UuoR8FceaEEGZ6G3/2q6+PPH3qe8siuIt8eqVb0bUZL7hTkyZVgaBa9clWVSrr4c5WXzlG/9uepE088EYBtttmm6XHlIlm77747ADvvvHOyqY/4c3C7yoyIaAVBEARBEARBEHSYeNEKgiAIgiAIgiDoMH0hHcwlV/CwtdeEknTAF34r5OdSO+3TbcssswxQXjynhbu5yudjSW5RukutmsmCjjrqqGTT4vhNNtkk2bSoddlll002lxIoVCqJIcCkSZMAWGGFFRqOy49Bx+iL8vsZyfpc2pZrG0oW4tx0001pWzLLnHwoJyecnlEbcYlZTj6UkxP64nhJunI1efpZhuWyNJ2HJ+fRYnOvdSOfetvUd73Nyb++YN3/ruQWvvheUiuXgOl4PGmPpEM+fufGi17Gfaq25m1S/vXx7fjjj2/Yj0tMdG1y97Wc1K4f0f3Xz0HtxhOHqF15cgrJ3nISTd+f+yrnU7VJJWqAeiIdv4Y5CX4v4fflXXbZBShLqHT83gblTx8zc9JXbQ91f9a9yOtxNbsWl156abJp6cL666+fbGeeeWb2d3oNtUGXY+qZ06XVkydPBuDTn/50sukcn3322WTz9p2rDZlLAJND17qZ5G6syR1TzpY7N7U/T5YmSeAPfvCDZDv22GOBcq1IT+qia3Pfffcl28YbbwyUpatHH300UF5GpPa81lprJZuWxLisUInfnnnmmYbzaEVEtIIgCIIgCIIgCDrMmIVpfNZIb7k++5SbwW6WAlsLigGef/75tK1ZM3/b1eyaJ8jQvv0tWzPhPlOmGc1cOuWRVIgeKb6oVfixy5e5Wa1DDjmk4bs+wyK/LL300tnfln89qpZbQKxrl4toOc2uaz8in+cSBjiPP/542l577bWB/GyqX5sZAaUl9r6v9uyzr7k25zO28qV/TrNPI6ky3yv44nWNTT4rqhnW888/v+E77lNFDH0s07aPGx5ZVMTGFQLypft56tSpAGy11VbJpt/28TSXYr6XcV9ovHefKsLt96BHH3206T4VJfC+r+vpEcF+85WjduXJUXS+HiWU35SUAer+9T6r65B7ZvC/59QFO+ywQ9p+8MEHgXK0odf97P1fM/977bVXsqnsysDAQLLpPP2ZR33cE1NJZeH3Yk9zr2jAww8/3PB3j3IrQuBp4qWUUQIPGJuI1rT31Fz0x59RdO4eeTvggAOAcoIw9XUfzx555BGgHJFR6ZzPf/7zybbRRhulbY2RXvKhWbTHf6+XIlkid0w5m555HLVZ74/77bcfUL6O888/PwCrrrpq9hikrvDvXHzxxUD5eUrvDXvvvXey6RnCk+BJ1eYqJP22j1/PPfdc9nimJSJaQRAEQRAEQRAEHSZetIIgCIIgCIIgCDpMV6SDOZlgLpTYTFYGeWnZrrvuCuQXz0E9PO5SBEk1fFGcJBpD1Y8Qkip4bSPVkRiqnk83yCXDcBRevvLKK5Nt3XXXBco1wORTl2RJbqGF79Min7o8RnIL/45CtL7gPZfYQbJOhd37EW/P8k+r8/HrkKspMaOi8Lu3SeH9Lrdg26VCknZ5oohcn+43XOqnhAwuU9V46wuB11lnHSCfQMTHVY2TLkX0NqnfcQldTu4qqZJfL33OZbQ+LvcDfs/ItT/JujwJQCs0jvr9LyfH6ue2q77s7SpXw1HyU5ehqg3lzn8oWbrG01x733bbbdP29773PaAsx3JpXi/i/VrLAC6//PJkU9vZfvvtk033Yr/nyF+77bZbsknm6jWLPPmOxhG/jpJx+bOTxoxLLrkk2ZQQyo9/LGmWYMLHwJVWWgmAL33pS8n2wAMPAPWkX1BPaONSNMk2ldwK6tI3b4su4Tz33HMBeOyxx5LtmGOOAeCCCy5INm+j/YbqD/p4r2d3r0N15JFHAuXkdnq2d5vu8z5O5OrouQxYY9DZZ5+dbPKvj0GShz755JPJpudoTwKz0047AfllIa2IiFYQBEEQBEEQBEGH6UpEKzdL74tYte2zCvpOLorlC9f0Juop2H0BsWbNPPWwFsT7DIxmOXyxrt6GcxE5Z9NNNwXGNqKVmwn2mTjNXHnFeM22+DkKvx4636FS2MsHPqOrWXafWT/11FOB1imcdb36OaLlM/yaeZkyZUrT72hxJsBBBx0ElK/DjIrap7dTRafcP3PMMUfDdz0hjdqkz3r1c6p89UeP9OVm+dUWfUFxLuqkMdHHWLVd95OPedrORbT8WB566CGgHNHStfNxRb/nY1cuCtEreMRe0SY/H82GfvnLX274rrddn1XXTLYnJVBCGN/3fPPNN6pj7wVyZQg8Zbb+7jPR2vY2J/+5H3PPCrnkG67EkM89TXSvj8FS0AAsscQSQPnc55lnHqDcH7Xtz0H6jiJSUE+A5UksPIKeSx2v5Ds+Ht97771AOXKj454wYUKyud+7RbNnyRxK1uHJEDyS3Qw9b/lzl/CEbIceemja1vORP49+/etfB8p9Q0oP97Ouh7dZXWsf83W9rrrqqmRz5ddwmGmmmVKbUVTqxRdfTH9Xf/Xz0e+7H6+55hqgXOpCCSa8X2tM8GdLna8ru1zVpvuJt3e1Y7fpeUERS4Drr78eKEdt5XOPhutcvNyCJzhrRm+PMEEQBEEQBEEQBH1IvGgFQRAEQRAEQRB0mFFLB3Nh91yFaA8NNqtp4wsxt9tuO6Ac+pNExaUnHupW+DdXe8BlLcLDy1rk5jaFGv2YvYL0WOHh49z5aEFsruq4+0Ih3eEkYcjVx5LNZU233HLLkN/VIn7Iy5r6jZyEyxe35nDZhPyWW2A/lvXZegH1N5eQ5WRnXt9FaDyA+jjhbbLXa+Q0QxLbnJTZ/aL+7TZtu/xC46TLOSS18nbo460kIj4m6njcJplLTq7jfV/H77IPr9HTa+RqgPm9R+00t+B/qMQNklnlJHQujcmN5b2M9zX5xReOe/0ZISmpS/5yCVjUp/1z3t5zNdskE/QalDk5Zj9JByWL9v6qRfoHH3xwsqmN+WJ+naf78Ne//jUAK664YsNvQL2N/uEPf0g21RbyZ5Lvf//7DfvR84lfRy2B8OPqJO985zt5//vfD9T95uOPtn2cOv7444Hyc+Saa64JwKyzzppsat9+f5FPV1tttWTT2OayWZeqXXHFFUD53qWlH9tss02yKRGJ70d9w9us+oHb5OfbbruN0TL33HOz//77A3UZaC4hhPtUCUN8PJMvXXaoe76PhZLmubRa5+P3Jh9vcstedIx+/SUTnThxYrKp3pkfv8Zzf07W30dyv+rtESYIgiAIgiAIgqAPiRetIAiCIAiCIAiCDjNs6aDkEAqjNZMBQl6ipnDiggsumGzKre9hfskAPHSqEKLLEHLZ8Py49Dv+OYWuc5mNPASrsKPLQJSJaplllkk2heq7hWcdVEjUQ6cKwXr2IOEh0VytolYyQkmFctnI/Lhy+8llyWlVE6yXUYjfZZs6b8/6lsNlXCInQZzRpIPC+2cuw1tOSuWSLcmCfGzIZdzsF9S33AeS9Xi/UwZWz5AnSZZnXMtJTDQ2+FiSy1TmbVf7cQmdtl0WovHUj1+/rUxp0NvSQZf7KkOWS4wk/3E/i6Hujco++oUvfCHZdL0ke4L+y5iZG8v83u0SOKH7q0uR1Ea8fakdDpXJMSc1VrbhXCY8JyeNbzdj3Viw8sorp23Jfj1DnjIxex9df/31gXp9O6j7c7311ku2O++8E6hnM4Ty2KLfufbaa5NNNaNcpqkaRC4dlP89K7S2uyUdfOutt9KzkMY0f87UPcKvr/r4vvvu27A/7+s6X5e4a7zzWk1aQiA59XA46aST0rbuY7mx19EzVm5JRif8/Oqrr3LOOecA9fbnmSt1r/asg3qO91pYysLodcgkGfTP6XxzUkSXAfrYrIyG3i+09GiTTTZpen467tzSIn8W0/UfSd29iGgFQRAEQRAEQRB0mGFHtKad6fEZOM0c+Nuptn2WVG+x/gapyFJuQbwvSNR+/C3f96MZbJ8hUxTHZxi0T/+uZsz9jVVv6/5mq8WO/vbcbXKJKBwttlRdF8c/L5/mEpYMRW7GT/71a+Oz2dMet/+Gz3D1G5oddT/rHH1WMEeu0ntu5jQ3szIj4P1JkQLVgoPybJ/w2iCKOCjqCPlZ9n5B/c7HRPU7b2tTp05t+Fyz6KlHDvUbvgDeZw01K5xLGOCL4TU+Tp48Odk0U+iRSEUhRjIrOB74TPU+++wDlPusZp032GCDZLv88suBocdVjdXeTnNqilzyiH4hV6dSkRm/9+jv/nyg77raJDdOehvXd3I+93u3oj9OLhrWSxGtG2+8MW0r4ZTX8snVAdLfva/rPN1HuQQZrjjJ+Ub79PuZxh5vs4o4uC2X0KiTvPnmmymK4wk8+hGPBo8n//znP5Ni64knngDy0Tq/1+q+scgiiySbVAybb755sp122mlAOTqlSH7ueWk4XHjhhQBsttlmyXb33XcD+Xp8fv9UH/F3GT33u63demsR0QqCIAiCIAiCIOgw8aIVBEEQBEEQBEHQYUZcR2ujjTYCynWvFOr3hc4KPXuoTp/zxduSknh9FYXvfEGiwuMusXAZisKXLhfQ7/giPD/GaclJXVzaICliTp7TLVrJGrTodd111236XeHygVyyi9xn3ee5c5cUxiUxOXmlL5rsN1SXwpOOSM61/PLLD3t/3ran3d+Mhi/SljTTZQa77757w3emTJmStiVlU10MqEsSVD+jn1Df8b6mccgXrOscXfKT62MaB7zNaZz0McXHav2ey0I0fvvnFlhgAQAeeeSRZFMtGh87JXPsF1mc+0Xn7fcbXRtvm5IODnV/ePnll4G87N737XLOfiAnU/N7ivqn33sk13E5ju6v3uZyteIcXSdva7penlQk9/1mssNewBNMqH+tsMIKyaakE7l6YZ6kRWOC+irUkxp4HSOvNSZ/elvV992vev7wNqDjcp9rKYY/iwW9zZtvvpmul8btDTfcMP1d19elvpJv+v1ZbemHP/xhsj366KNAOUmblpbk7mFD1dHSeONjrtqfP0+pNpkkhFBvs/5djT25ZVDtygWdiGgFQRAEQRAEQRB0mGFFtGaZZRZWX311oJ4KUzOUUF8g54v49GboC9tyC9QVdfI3W81S+eynZp1yi2eh/nbqkTHNxng6dn2uVXptJSbw2UX9PZf8oVv4AvVcREs+UJp8qM8w5Bayt8K/o1nJ3AJCZ7HFFgPKs2i6Dn79+znZg9I8Zn4nAAAgAElEQVTc7r333skmP6+00kpt70f+y7W/XlqIPRaoT7svlIbZU3/nZvh9FkqJWVZbbbVk8xnWfkPtyce63AyzIvCrrLJKsmlxu/dZbeciBW7zbY0DPiuobfe9ork+U60xy2ceNSvox6rUwb2Ook1+PdQmlYhlOLhfdK39/ue/0w/4sasN5aKxHuVSe/H2rL97dE/7zpUYgXo/9/uWnkO8TXpaaqFjHMl9cizYYost0rbGyi9+8YvJdtlllwHlqL36uicL0rnfeuutyaYkB37u7i9FKzwCoKiUK3+kEDruuOOSTYlH5p133mQ7+uijAXj88cfzJxv0NColon8dPf9BfWxzm9qN91u1EVdZqP15hEzf8Wdzb39qpx6V1rOnJ2DR/S4XvfbkTsLfZRTJctVGu/TmyBIEQRAEQRAEQdDHxItWEARBEARBEARBhxmWdPD1119PYWdJCJdbbrn097XWWqvhOwrLe+ILheB8UZnChS4/UHgvVwXd5WcuLVSI0RMTaLG4h6uVzMNDlrlkEDp+LayDejhxLGvBuJwsJzdT6NR9JflQu7WEhkqGIVyGlNvn1ltvDZT9rIW8/l2FkPsR1TRxGZvayHCkpOoPuRB2P9d+GglqdznZVKvEIC4NVB/w2m5DLZ7vBySTcImZZDi+UPiuu+4CygvktRg5J9P1Nqfxz9ucjzU6Bpf+qr17n15ooYUAuOCCC5LtlFNOAcq1qLS/XB2WXueGG24AYLfddks2JQTwGiztopo0UJeteB/oVSnbUHi7yt1LlKzB5cD6nLc5tS+XHeak1rmx09ukuP/++9N2ro5Wr0sHv/KVr6Ttm2++GSg/e0jK5AlyNO75fUpjgkv79Vzj5+5JNTSW+vWUbMzHJbXbn//858mm+l6+b9mC6Q/v1zk8McaMRm+OLEEQBEEQBEEQBH3MsKZ7ver24Ycf3vB3zbL4YvQlllgCqKf6hfrs54QJE5JNi6Rzs2I+S6Uo2OTJk5Ptj3/8Y9pWNfBWqXE18+qpTpV216Nv2vbZNc2yP/TQQ01/o5P4jJ/PJAmlG/cZUR2nz+o3S2WbS/kO+dnJXNRF19WrfO+www4Nn+vnBAWahfZFkooK+HVRRXSlL50WLfTMRVxmtIiW8KiJotS++DVHbtbb25fP3vYbp556aoNNY6zaF9Tb2Pbbb59sWijsM9+aWdYYDvVUuu4zj/KrLXpiBo0HvshYCoeTTjop2ZRu3qM9/Zay3FFaYh/TNJ56NKFV3xd+n1GE0vu+L/buB/yekYss6V7r5T/0HR87NTbm0jf7vSiXsCmXQMT9rPE2F8Ht1ei3Sl1A/Z7ux//AAw8A5ZTb2223HQArr7xysqkUz5577plsarf+HOSlS3QNPMollYonD9AzmJeYUIITfyZRhMzHjiCY3omIVhAEQRAEQRAEQYeJF60gCIIgCIIgCIIO09FYuSQiV155ZbJp+yc/+Uknf2rUbLXVVuN9CMPCZVU52Z8STLh0Qt/JyThytqFqlGg7Jy30mhtrrLEGUK8SP9S++60+TI6cvMolEq3kQ0oGILkl1GWxvboou9t4rThJVlpJzXL9wv3ntTimBzTGujxXsjNPhKO25HKoF154ASj3P33H+7b7VP3W23suQYmSbngSIsm4pxeUOMCll5K8e99XTa1W0kH3o8Zv34/7vN/w8xBqdy65l2wv18/9HpVLmOM0qz2opFB+DJ4kRvLjofY93qiNQV2a5xK922+/HSjXzNI9WAlcoL5Uw8fZSZMmAeUao74fjaW/+c1vkk31ulw6eOmllzbsR8ft8uV+rqEZBCNlxnyiC4IgCIIgCIIg6CK9ufozaMBn5jUj5TNF3/ve94DygljN3jWb7YP6bKFHnVqlHdc+PbX+NddcA8BFF12UbNVqteEYenXmcChyiUHOO++8ZFO6Z4+krL322gBcccUV2X3mkjzod3zGfEZCldyh3tZaRfc80YJmwL2d+uxtv+LtT/7w/qS2love+fnru4sttliyPfbYYw3f0SJ2/21PTKAIge9b0Z711lsv2RTRapVYp5fJHfvll1+ebEqM4VFAlbk466yzmu7bxwBdG2/vuTG4l/E2krvnKHqvEhkACy+8MFBOtqDolicDySWx8GitkrnkElp4O1UyBt+PJ9TpRbyUw3zzzQeU+7D646abbppsOj/3h3zs6e7Vpj3q59FyJeLwe5LKmPg4oX174pEFF1wQKD+n5BJ5BcH0TkS0giAIgiAIgiAIOky8aAVBEARBEARBEHSYkA72Cb6IVLIMlwpJjqdaYACLL744UK8cD82lWENJVWT3xcmSW/iCWEkK/BimPWaoSwr6hZx86Pzzz0+2PfbYAyhfD9U0Ouyww7L7lKQjl3Skn2sNjQYlawCYZ555gNayHpcXqY15EgG1yX7G20hOkrXkkksC5cQ0Gg/886pp+Pjjjyeb5GuqsQP5+kWeQEP9weVy2nb5Z+749d1+kRD6eClfXnLJJcm24447AmV5muRdrchdLyUxgXJyk37AZWoaw1yip3al5A2Qb0vyuRKEQL2d+ljsSSIkT/N2pd/z5A6qq+fXSIkjerW+o9cMvfnmm4F6n4f6fcclhrJJKgn1Wnd+f954442Bsi89iYtqonqtUvnOEznJh9dee22yLb300kC55qQ/iwTBjEJEtIIgCIIgCIIgCDpMRLT6BF9ArDTqHvnQjJJmrccTpTaH+uJYjzLcdtttY35Mo8FntRXV89TViqr4OebS5ztTpkwBYLnllks2zYp7dGFGwiMFq6yyCtDaj774WjOnHpHx6M30gCIEuQixJ5lRCm333wMPPACUoyaadfbP+cy+fsf9rEhMLhW5R95l8zTm/RbRyrU/T5mtJCAeOVBUz1Pd33333Q378Zl++c0juB6t7Qdy0Xkfy9RezjnnnI7/9iuvvDLk3zyCpsiNJ43SWOxRnV7iiSeeSNsbbLABAAsssECyqY16e3v22WeBcn9U4pGcCsD7vH9HY6lHy+Sn+eefP9nUr72vK1mG+si0vx0EMwoR0QqCIAiCIAiCIOgw8aIVBEEQBEEQBEHQYUI62CfceuutaVuhfV9A3EpiNZa4DEHyIZcZee2jfqBVHbInn3wSqC82hrq8Ys0110w2l39KAuYyN/ltrrnmGuUR9ycuhZVfWvneUcIGlwC5bGV6ICe5O+SQQwD46le/mmybb745ALPNNluyqWaWJ22Rz1566aVk8yQEkgx50htJgjyZgxbYn3jiicnmMiLRS+NUO7SSOKrvb7nllskm+Z8SDUBeOuhyLE82IrxOUT/gcjZJKV1SecQRR4z5MU3LCSecAJTrx0nq6RLxXpK4SdoIcMABBwAwceLEhs+dccYZaVv3Ih8/lTDEZZaS+btk1aWDGku93+pe7j6aOnUqABMmTEg2yeJdvt0vkuEg6CQR0QqCIAiCIAiCIOgwEdHqE55++um0rXS1HgFQ+ltH6XZ9VmuoFO4jxfen33n44YeT7eKLLwbKM5tKUdsvtJqFO/nkk4H6rB7AWWedBZSjWM6ZZ54JlP2ihAPXXXfdyA+2j5FPANZZZx2gnHSkFRdccEGDzVMjTw/kIkJKonL44Yc3/M2jDEp84ZGSWWaZBRi67IOi5j7jrSiOJ4Xotyh1pzjqqKOAetpwqPvsmmuuafrdSZMmpW2VNnjttdeS7corr+zUYY4Jfg9S1MOTqDTzR66ERjf43e9+B5TVIJ6CvhfxvnfuuecC8NxzzzV8ziNfvi1OOeUUAO64445kU+TbI/8egdLv3HfffQ1/v/DCCxt+w/etseqpp55KtohoBTMiEdEKgiAIgiAIgiDoMPGiFQRBEARBEARB0GEqwwnlViqVl4AnWn5wxmTBoijmHu6XwqdNCZ92nvBp5wmfdp7waecJn3aeEfkUwq9NCJ92h+j/nactnw7rRSsIgiAIgiAIgiBoTUgHgyAIgiAIgiAIOky8aAVBEARBEARBEHSYeNEKgiAIgiAIgiDoMF2ro1UZqGwDnAd8uKgWU9v4/OPAKkW1eHka+9+LavG+YfzusD7fZD97AZcX1eLZ0e6rm1QGKm8Ck6ldy/uBPYtq8UaTz58GXFRUi3MqA5VrgK8U1eL2sTjWfqEyUJkTUBGbDwBvAi8N/v+qRbX4d/aLwZCET8ePykDlG8Bu1Hz+FvBpYBL58XYrYOmiWnwns5+PAP8uqkW+ONx0QrTV8aEyUPkAcDwwEXgNeAE4sKgWDw5jH7MBuxXV4sfdOcr+Ivr+6Mj5r6gWt4xyn9fQ4rlreno264YPbd8foeanj3Vif92imwWLdwWuH/y32sXf6RZ7AVOAnn7RAv5RVIsVACoDlV8BnwGOG99DqlEZqLy9qBZvtv5kb1FUi1cA+fQw4O9Ftfiuf6YyUKkAlaJaNFaQ7QKVgco7imrx39af7E3Cp+NDZaCyBvAxYKWiWvyrMlCZC3jXUJ8vqsUFQEPl58pA5R3AR4C/A9P1w1a01bFn0J/nAacX1WKXQdvywPuBtl+0gNmA/YEZ/kUr+v7oGK7/gkZ62YdjOaZ25UWrMlB5H7A2sD5wIYMvWoNvn4cBLwPLAncAnyiq9dSHlYHKe4BzgXOLavGzafb7VWAnYCbgvKJaZF/gKgOV7wObAM8DuxTV4qXKQGUF4KfAzMAjwD5FtXg1Zwc2BFYBflUZqPwDWKOoFv8YpVvGguuACZWBykLUolbLAlQGKl8B3ldUi8OG+mJloLIrcAhQAS4uqsXXKgOVzwCLFtXiq4Of2YvaTNjnKwOVTwAHUOs0twD7F9XizcpA5e/AScBGwOeovWxPF1QGKotRuxHdCawIbFwZqKwPfI2a3y4oqsUhgzeml4tqMdvg93YBNiqqxX6D24dSm935S1Et1h/8/LHU+sy7gROKavHzykBlo8HP/h1YFPjwWJ7vWBA+7TofpOa3fwFoFrsyUAH4QmWgsiXwTmDHolpMnaaPnwb8k9p1eQZYE3hzsO9/oagW1431yYwn0Va7yvrAf4pq8VMZimpxd2WgUqkMVP4fsDlQAEcW1WLS4DPG+cDs1NrvoUW1OB/4DrBoZaByF/BH3btmUKLvj46h/PctYEvgPdRePD9dVItiMAp1C7W2PBuwb1Etrht8pj0VWB6YOvg9Bvf1E2oR3PcA5wz1TNvHDOXDx4HTqfnR2+B7gROpvR+8EzisqBbnDz7Tngm8d3C/n582uloZqEwETgZ2oPbsn9vPXsB2wPuAtwPrdeWsp6Fba7S2Bi4dDPm/UhmorGx/WxE4EFgaWARYy/72PmovZr/JvGRtAiwOrEpttnHlykBl3cxvvxe4vagWywB/oh5NOwP4WlEtJlCT2g1pL6rFOcDtwMeLarFCP7xkDd6sN6d2DsP97oeAY4ANqPl24qD083fAtvbRnYGzKgOVDw9urzUYTXsT+PjgZ94L3FJUi+WLajHdvGQZSwHfL6rF0tQero6kNrCuCKxVGai0CmFXgQ2LarE8dd9+CnixqBarUht0P1cZqCww+LdVqL3ETs8PWeHT7nE5MH9loPJgZaDy48pAxW8sLxfVYiXgJ8BXhvj+fMCaRbXYjtqE1PcHx8QZ4UErR7TV7qCJ12nZjto9aXlqk3f/rzJQ+SC1l4BtB9vv+sD3BqNiBwOPDLbRGfklC6Lvj5ah/PfDolpMHJzIfg+1iI14x2A/PpD6M+ZngTcG+3AV8OfhbxTVYhVgArBeZaAyoZsnNA4Mtw1+A7hq0IfrU+vv7wVeBDYe/PzOwAn+I5WByprU2ujWRbV4pMl+AFYCdiiqxZi8ZEH3XrR2Bc4a3D5r8P/FrUW1eHpQcnEXsJD97Xzg1KJanJHZ5yaD/90J/JnaDW/xzOfeoqZBBvglsHZloDIrMFtRLf40aD8dWHcoe9tn2Ru8Z3D27nbgSeAXI9jHROCaolq8NBhK/RWwblEtXgIerQxUVq/U1i0sBdxALeK3MnDb4G9vSO2lGWovXb8b1Rn1No8Udd30atQ688tFtfgP8Gtat58bgDMqA5X9qPe/TYC9B315C7XZMLXtm4pq8WRHz6D3CJ92iaJa/J1aX/0UtXVGkwZn9aCmHIDaA+5CQ+zit0Ufyn+7SLTVsWVtahOvbxbV4gVqk6cTqb3kfrsyULkHuAKYl5rMMBgk+v7oaOK/9SsDlVsqA5XJ1Canl7Gv5fy6LrVnUYpqcQ9wj31+p8pA5c/UnmuXoRaAmG4YQRvcBDh4cCy8hlqEfwFqUamfDfr8t5T99GFqkawtbawcaj9Qi3T/pWMn2QYdlw5WBipzUGt8y1UGKgW18FwxKPsD+Jd9/M1pjuEGYLPKQOXXRbWhknIFOLqoFicN85Cm94rMaY2WqAxU/kv5Jfrdo9j/WdTkmlOpyTWLwZnD04tq8fXM5/85nQ/Or7fxmbeotVfh/v8ktQe0jwF/rgxUVhz87P5FtbjSPsegdKid3+t3wqddZLA/XgNcM3ij2nPwTxqLpx2HnRnKV20QbbU73EtN8tMuHwfmBlYuqsV/BqVIo7nPTZdE3x8dGf99mlr0aZWiWjxVqa3h9HbXjl8BqAxUFqYWyZlY1JaxnMZ02IaH2QYrwPZFtXjA9zHo5xeoRbbfRi2iLZ6j5rcVqedUGGo/qzEO7bobEa0dgDOLarFgUS0WKqrF/MBjwDptfPdbwKvAjzJ/uwzYZ1CbTWWgMm9loDJP5nNvoz5g7wZcX1SLvwKvVgYqOobdgT8NZR/c/hvwP20ccy/yAjBPZaAyZ2WgMhPl0HaOW6mFreeqDFTeTi0CKT+cR00K6lHKK4Ed5P/KQGWOykBlwU6fRB9wC7XZrTkHpZu7UGtXb1FrV4tXBipvoyy/XKSoFjcD36TW1uel1rb3H9wHlYHKkoO67hmR8GkHGTxvj/yvADwxwt3185jYDaKtdo6rgJkqA5VPyTAoo3oN2LkyUHl7ZaAyN7XowK3ArNTklv+p1NbJ6f4TbXSQ6PujYwj/6cH95cFn0XYmB66l9ixKZaCyLLUXNYBZqD30/7UyUHk/taUf0xUjaIOXUVs/WBn8/oqD9lmB5wbH1t2pBXDEa8AWwNGVWh6IZvsZF7rxorUrtYdz53eU5YPN+CI1OdyxbiyqxeXUpBk3Db4Vn0O+478OrFoZqEyhFlk7fNC+JzWd5j3ULnYr+2nATysDlbv67aY3KGM5nNoN6Y/UolHNPv8cNW371cDdwB2DC4spqsWr1NLGL1hUi1sHbfdRW6R9+aDf/kht0eMMRVEtnqb2wHQNNRnszUW1uHjwz1+j1tlvBJ62r31/sP1OBq4uqsUUaslDHgLuGmy3P6G7GUF7lvBpx3kfcHploHLfYF9dmlpCopFwIbDt4JjYzsTZdE201c4xqGDZFtioMlB5pDJQuRc4mto9/x5q96WrgIOKavE8NXn7KoO+3IPBe1xRyxh5Q2WgMqVSS6IxIxN9f3QM5b+fUctIfRlwWxv7+QnwvspA5X5qz2V3QC3ZCzXJ4FRq7fyGTp9ADzDcNngENZngPYNjwBGD9h8De1YGKndTW8JSikoNyoo/BvxoMGo11H7GhUrRoNALgiAIgiAIgiAIRkO3kmEEQRAEQRAEQRDMsMSLVhAEQRAEQRAEQYeJF60gCIIgCIIgCIIOEy9aQRAEQRAEQRAEHWZYmY0qlUpHM2fMPPPMDbY33nijkz/R8vdGwzTH+nJRFHMPdx/t+rRTvnr722tZMd98s17qSvvuBd932qejuebN/CE/AswzzzwNn3/nO98JwL/+VS8b97a31ec13vGOWtfzZDQzzTQTAM8991zT4xruOY23T/33c9+VLxy1z7feemvIv2WOsfQvwH/+85+m32l2XM0Yy76fYyTt+r3vfS8Af/lLvVZjzmeO2rb8CPD3v/992L8thjHGjFk7bXZMuX3MMcccaVv91/uxfPnvf/+7wQb1sSHXtueeu37KDz/88JDHN5bj6Tve8Y5CY1MbvzEkzY65U/eesX6mYIQ+hbJfh9sGW51TM1/POeecafsf//gHUG6fs88+e9r+5z9r5Ypefvnlpr83XFodf1EU+QGpBe361Gn2/ON+7HI7avi9TjBe9/6x8FWP0JZPxzWF7Ic//OEG2x133DGmvzcapjnWkdanaItO+ep//qeWEf+1115r2Hcv+L7TPh3NNW/mD/kR4JOf/CQAd911V7J94AMfAOChhx7Kfkc3M38YW3TRRQE48sgjmx7XcM9pvH3qv5/7rh4u/WH1r3/9K1C/0Tvedh29/OpBFuCll14C4JVXXml6jOPt0+Gy9NJLp+12M8euscYaAPzyl79Mtne9611AeeLA+dznPgfU/Qhw7bXXDu9gjWGMMWPWTpsdU24fu+22W9r+73//C5QnVN797lrN0ccee6zBBvWxwV9Y5f/PfOYzybbFFlsMeXxjOZ7ONNNMTX+v3WuqffhDvdpuu/vwdpqbPBnrZwpG0U7dr8Ntg63OadlllwXyPtpjjz3S9uTJk4HyZNeOO+6YtqdOrVWGOeWUUxr2k7uOOUZzvYdLuz51mn3efd/ldtTwe51gvO79Y+GrHqEtn4Z0MAiCIAiCIAiCoMPMUEUR+4WVV155xN/daKON0rZmXl0KMNdccwF1yQDAxhtvDAw9C6HPenRBM2V/+tOfku3nP/85UJ79Hu65+OfHY1Yk95urrLJK2taMnM8KrrXWWgBstdVWDd/9v//7v7TtYXfNIOZkdb/4xS+SLScj1DGOpp0Ml5lnnnm0UZ8hefXVVwE47rjjkk3yNj//RRZZBIBnnnkm2R588MG0vdRSSwHl6MIVV1wBwD333JNs991335DHOt4+beUztcXbb7892XTMHjHdYIMNAFhxxRWT7amnngJgzz33TDa1Z8kKoT5GAGy33XYAvOc99ZrthxxyCAAXXXRRsl1wwQUAPPnkk22fS7+h8znqqKOS7cUXX2z4nPrxYostlmy5sdP7vuROZ5xxRsPvNTsW6H6bfeONNzpyLXOydUU7brrppmRbffXVAbjuuuuSTe3Po9mKxkJd2urRk4MOOgjI+6fX2mYnrqHvIxfJmjBhAgAHHHBAsp144olAOSI1ceLEtK1x5PTTT2/Yd04um4ts9Uut1tyzR6fayXg/13ST6e18ukFEtIIgCIIgCIIgCDrMsCJaPgM73LfYVjM2ndpfbj/tzha1ewzdmkEc7nHutNNOyTZp0qQhP3/ppZem7bXXXnvUv+9suummafvb3/42AMcff3yyfelLXxrxvrtNq7aiv+dm5JZffvm0reiLIoNDcc4556Rtre3YZZddGj7nkZ3vfve7Qx7reM1qN/utkcxuKYLy+OOPJ5uisT/72c+SbcMNN2z4rq9zed/73geUZ3PV3q+66qpkU8SwX3zqNJsd1tofgPXWWw+ABx54INk0RqywwgrJpuifr33x72id0euvv55sSpDh48qSSy4JwPzzz59szdrCeEcZRnOf8GugyLSvtdx+++0B+PGPf5xsn/3sZ9P2WWed1bAftX1XGkwPtIqy7LzzzgAMDAwk249+9CMA1llnnWRT//X9uVLjmGOOAeDggw9OttVWWw2oj7VQj7x0+nlkrGg3yvnxj38cqPdVqCtN9t9//2T70Ic+BMCss86abH7/PvvsswHYbLPNkk1j6jbbbDP8E2iCrsn999/f0f12mnbbzlCfk73V/adTx9Ntmp3PaPaXo1P357Hu3xHRCoIgCIIgCIIg6DDxohUEQRAEQRAEQdBhRp0Mo9Uiv9GE95rJS4YKIebCmJ2WOeY+14lQZLOQc+58tdi3FS5rGQ2XXHJJ2lYNmI997GMNn1tzzTXT9mGHHVb6F5qHmrsVBu90SF4yNajXGPHkAJJieWKGHXbYIW1feOGFQFlOuPDCCwMw33zzNT2WTkkORkunpLaSDnoSFeH1e5RwwROMfOITn2j4jsvgTjrpJCCfJr4V3W6f7Uqxc78pWRTUJVQuvZT8z32x7bbbAmWfKpnD888/n2w+Xkjedu+99yab5IGe/OXMM88EYMEFF0w2yeVuvfXWIc/N6dZ42ilWXXVVoFxHSwlGcrXgvO97+1PiBv+OpFvVajXZlCTm5JNPbnpcY9n3Oz12Sybo0jVJ4v1+oyQr3p59DHbJoLjlllsajqvdxAydlkONBS7l+8pXvgLAXnvtlWyStHodPSUTcomhJ2mZMmUKUE4gpKQaLidUUqxW0tfxlrnl6PQ1Hq/nx27RbiKs3Pm0sg2XTrWfsZauR0QrCIIgCIIgCIKgw3QlvXunZy16+W1/PDjiiCOA9hdlzjvvvE0/d/755wPlFM9eYFMzYG97W/29XAubfUG8Zrg08wvlRfTTHlezYx5r2m2zPvMnNEPt0b0//OEPwNCFYBUR9BlWRWp8JleMRZKX8UKRGE8wIjyN/m9+8xug7u9pUXmB/fbbr+FvKrw5HMZy9rXdGXRFk7xdKWGFz/CraLMnC3n/+98PwMMPP5xsjz76KFBORe6R19tuuw0oJyZ49tlngfIYofTbPqP9rW99CyhHeg888EAgXzKhWzPLndrvvvvuC5Sjf4oSekp8FYT2yPRvf/vbtK2x033lCRuExtHRHP9Y9H35OVeY1ouH63OzzTZbsv3ud78D6qUboF6m4KMf/WiyHXvssUC53MOuu+7a1vG5DzRG+zUcbvrxbkQihju+eyKsK6+8EqiXYgC4+uqrAbj77ruTTffx3//+9w2/6+OJp83XPd33rWQVnuBF3/nb3/7WcAydjmp0gm4mQxnrpGrdVl6MJGlT7nNjcc1H44tutomIaAVBEARBEARBEHSYeNEKgiAIgiAIgiDoMB2VDnZrkVurhWsjCRcONww4lvV1WuELh5shOcrqq6+ebDmZhCQuHvbX4mOoy+V8kfwHP/hBAJ555pmmxyBJkcsOJZsbSzol91huueUabLmEIJIH5WrHQF1m45KN9Y4qL9YAACAASURBVNdfHyjXOcrJL3qFTkto1lhjjbStJA0uR9K2189xXnvttQabZHVDyQ2npRf97Eji5/I/JVVQ7THHEy5IRujSrQUWWAAo1+CS9A1gmWWWadjPCy+8AJSlxlp8f9pppyXbiSeeCNTldVD373DlWqOh3UXarVhqqaUAuP3225Ntgw02aPic+rTLpl2OJV/+9a9/TTbfFqpx1C/4NZVPJbcEuPPOO4FynTYl/HBp79e//nWgLiGEeqIHv0e1Qj7345KM1es/KplRu4yVFDvXVpXc4tRTT002jYuSq0NeirrEEksAcOONNyab+qtLN3XPhvq9yO9J1113HVCXEEM9UdEss8ySbJJw67qPNcMdy3PPeK2e+zpdO6qbUtRO0im5Yjdllp2+l4/mGkVEKwiCIAiCIAiCoMN0JRnGSBhueuNu0uqNdbwXdHp65mZoZjA3e+spXhXxUkRlWjSD637RDPYBBxzQ9Bi0b19Mr1SwOToRJWk3HanT7gyNUgYfc8wxTfenRAFDXSulxj7vvPOSTdEr/duKXk9yIVrNCirqp1TZUE+a8MlPfjLZ1GbnnHPO7O9sueWWQNl/Tz/9NFCOQrRbeb4XUYRp8uTJyaaZe088oBTNHkn+yU9+AtR9C/WkAJdffnmy7bPPPmlbs/2PPPJIsmkGXck1oB6l8QQZwpPjNGMs+36rEgk6FkXuoe5LRVuHQpF/JS4B2HDDDRs+98Mf/jBtt0oPP+0xtlsCpFu0m+DAk4Cobfzv//5vsn3jG98AyhG/o48+GqinI4d6n/c27vcw8aUvfSltK8Lt0R9x2WWXZc9lvGh3tjyXeERJhJQwBOCVV14Byj5ccsklgXJ0atKkSQB84QtfSDZP+KQomLd52RZddNFk23rrrYGy/y+++GJg/CKz3UrgMNp7RC+0t5EymkRmw70O3YjudcL3I7lPRUQrCIIgCIIgCIKgw8SLVhAEQRAEQRAEQYfpGelgMzoVQhyqplGv0alQ5xlnnAGUK8aLOeaYI21LMnj44Ycnm2o6QV1K5JItLaI/++yzk80XvU+LyxWaSQc7ca1HUvehGTnJZCtU0+jPf/5zsrl8UvhC7ZlnnhkoS+hyx9AunQ7BdxI/JtUg8vYlyd/BBx+cbFpwfeutt2b3+c1vfhMo+y+XIKOXadVeVRPP243kqS6/UsIKtSmAvffeGygnBpEUUfWMoF6nD+oL3g877LBkUzINlw6qvtHCCy+cbEpEIhkT1KVf7SYnGS6d7vueJERyzBye9EbJBFyumpMOuszSJZ7in//855C/10t9upmfPaGF2tKKK67Y8Dlvz5/+9KeBen09qLdxT9TgCS283YmcZFCSbU/kovp8js6pGzWzRoOSr/jxq1bWmWeemWxKGnLvvfcmm2peqm4mwPXXXw/Axz/+8WT73Oc+l7aV/MkThqjf+/OCf0dorPJxvduybe//zfbbD/K9XpG4t5Jjt3ssw10eNNpr1Eoe3qnfaYeIaAVBEARBEARBEHSYMYtojfVs0A477JC2lfbU0+0OF5+RVCX2TpyTzxYMdybWF7BvsskmDX+/5JJLGmza93e+852Gv/mM2D/+8Y+0rdnWe+65J9m0SFwpo1uhtNSt6NYMYqf21W6SDc3wn3LKKcm22mqrpW2dp89ka0F8bpH3SOiFGdh2UNTF+6ciCUr/7AwVmZYvPRKQS3M8GroVJWw20+aL04UvOldU+Qc/+EGyKTFIzldKigH1mWhP+e4JB4RHB5QIRunboT4j7vsR3sa1cN+jPd1mNNdKC/+hPsZeeOGFDZ/ziIzGxp133jnZvva1rzV8x8tl5CJ8uWQY/YKSI3jSFvnII6GKwrovdO/xfiyFhfvJE0IoappLk+/3xlzimHbppkKg3X0qfbonWlFqfI86f/GLXwTKEX352JOwCE924SoCRQ2XXnrpZNOY7AlytttuOwBWWWWVZFt88cUbfqdZKn359/777x/yM60YzfNUjrEu6zOaxB3dut+3UgmMpl/kkruIkZQAcQVHToEwmmuoKPJI3iMiohUEQRAEQRAEQdBh4kUrCIIgCIIgCIKgw4xYOjiWC/X8txRiHEp6IonaWmutlWyq47DnnnsO+7ePP/54oLyoV9LBsSQXWm2Fzjv33VxCCl8U7NdQsozf//73ybbrrrsCZTmc9unyDi1i/vznP59sqtkxlgtSOyVJnHXWWdv6XE424Uj26YvohSc1EK2Ov1cWzo4EheI97L/ffvsB5UXfklZq8fe0aKG81y5rlgBnND7tBLmF247+5n1HvvJzVOIVR8luXCKl5Au+P8n6lIAA4LTTTkvbknnpevhv+74lO15qqaWSTf3c5ZtKWjCW0sFWNJO++PmoJpS3SfVfr4MleZS3n9tuuy1tazx+6623ki3X511O129onMydg8uBVYfxu9/9brJJMuRjo9qcS9pdWqi27ckyhEtvdZ1cUrfZZpsB8NJLLyXbeNfKHAodt8v7JOv147/qqquAsuSvWq0C5eQ0Qs85UF5WsMgiiwDw4IMPJpueK/xeuNBCCwHl5y5Jtzot326Xdu+JzT7XKklDu/fWTskXO73vdmglxxzu80XueNuVCZ5wwglpWwnZAK677jqgLGdVcqxW/tGYkGunkuoC7LTTTgCcddZZbR2rExGtIAiCIAiCIAiCDtPT6d1zb6K5GWqP2GgGR+mNoR5dOPTQQ5NNszo+i/7EE08A5XSkW2yxBVBPN+3H1c3I3bTMN998DbZWUS4tyvbZAi1W9UXemsHyhax+brnZ6qOOOgqA/fffP9mUztWTObQ7KzseUZeRzATlEhPkyM16+OJ2LdDOpRb2Npkj1/56MYVtu7+vmU+f4f/e974HwJe//OVkU+KVO++8M9k81btms33WezQJBcZyVrvZ4mZf+K5Z/Fw6cP9cLsmAxs5f/OIXySZfeRTa9/OZz3wGKM/2yacecdBY84lPfCLZnn/+eQB+9KMfJdt666035Hl2i9zsa7vX1JP43HDDDUA5+pSbBfVkD8Ij/4putSo3kot290vJhokTJwL58d+jTirt0C7eTq+55pq0rdIiuaQirZLj6O+t0kF3cxxo97rq76effnqy3XfffUA9+gQw++yzA/V+6TYfEzWb71Hxb3zjG2lbzz0+Nut5yhOdKIK+0korJZuib4qutTqnbtHNa9jNNjHctOmtjmU8kovl1GhOLpKV2/dll13WsI9PfepTaXu33XYDytFaRbRySeGc3Jiw++67A+WxSqo2fw5uVqrIiYhWEARBEARBEARBh4kXrSAIgiAIgiAIgg4zLOlgq8Xbww2jtvt5Dxfmwnw77rhj2lZYXItjoR7y8wXfqgXjNi1ydomhFu76Yudukwt1ezhfNoVTh0JSK8n8oL5wdqONNko21ba55ZZbks0lbVoIqdpQUK88v8IKKySbFuO6JMYlB9PSLclAq0rmown3S9an2mxQb3PbbLNNsl199dUN373pppvS9pprrgnk5UO5RdkjoZP+7aZPJZHKyX5cWpCr2eayIEm6vM5FbrzoFdlVuzVfvG7TZz/7WaBeVwjqiQJ84a76nY9bOSmaklz45y644IK0rX4u2RHUfe59W8fjY7VqGvmx+hgzHgx34bsnQZLM0ttk7r7giQWEf0d+c/mnbN4H1HZH0l671cbblSnp/ptrcy6t9HYl5FNPtqL9+P7WXnvttC3/5iS1yy67bNr2WlMjpVu1HodiueWWS9u77LILUL4/65y9LarPqf9CvT+69FXbbnPpda4+mdqlt2m120svvTTZVAvxoYceSjZJscar5tNwkzn4eHbggQcCsOCCCybblClTAPjkJz+ZbL/85S8BePbZZ5PN/ZerDanfaTcpRLv327FY3pI79pxtuDLBXIIW+RvK10HPUZLRAmy88cZAOfHSpEmTgHJiN7VdT+Si/fnzg5JwPfPMMw3H3IqIaAVBEARBEARBEHSYUYdpRpL+st1q0HqrzM2K+aJqT3yh2VZfxKbZtSOOOKJh3z4rq5lwn1XUTI+nU1bkayxmtrTfb33rWw1/axYtgnqSAU/g8NWvfhUoz7CI66+/Pm37gnhFsuaee+5k08JwR+k2fYFgq2McDzqd5EAziT5bn0vC8Pjjj6dtfdb7gI7HZyH7heH6NNd3vK1oltZnWj16JTxaIl/655SQYTR0azF1TiGQG0d8ofrPf/5zAHbeeedkU7/zSNQaa6wB5GedHS2q/9znPpdsPt6qbfv4p+vkfp46dSpQHne1H0+IkEu/PZYJHobbTn1GU/cH96kSBXnSgUcffbRhP/57ihK4mkKz5U6rpDi9jEoE5PqsR16b9c9c281FrPzvuQijJ2dqxlhHqtrFywmoH+61117JpoQVHq1bf/31gXJZAY2l3qb1jOD3c1dUSO1y8803J5uet/x+p3HJFS76u6fSl6JmLND9oN2kap6uXiUwFl100WRTf/Sxd2BgAIA///nPyabx2MdUHxO22moroO5byEd7Vl99dSA/brfLWLRpRYz8t3Lns8466wBw0kknJduxxx4LlKN/Ull4X1cyOE/Y5px77rlA+RpedNFFQPlZd8KECUA9yR3UlTJK/ARw3HHHAWUV0qqrrgqUVQ7tEhGtIAiCIAiCIAiCDhMvWkEQBEEQBEEQBB1mxNLBZtKLnBwqRy686CHdnGRw1113BcoL3LyujqQDs802W7JJqvHiiy8mm+p3uNRAv+35+RUSd2mH6kjcddddQ5xZ+7RKMCJctpdDIU6vpaFQt9cAk09dkiVcjqmwK9QXJbtEsxkudcpVoZ8e8LYrX3pV8hx+HRQW90Qu/cxwJXW5Nu/yNI0hkmZBXl7lC+Yl7XI5YTPalVWMhfyi2cLtAw44INkkw/F+JV89+OCDySZf5hKIeNuV1MLlfS7hlMzI5UY/+9nPgLK07SMf+QhQTkKksdNltC7t6AatkrYMF5edeZIAIYmRL4ZvJU+UXM79rHuTX9ehZHL9gNqfy8aE38ty0kKRk3f5/TonNT7//POTTf7/wQ9+kGwnnHBCy2MfT3Jtxhf4/7//9/8AuPzyy5NN7WmHHXZINsml/FkmJ/mVpM1lbJ58R+OrJ8+SjMvHDI0pXrOomUzY6daYqjaRW76Qe7b84he/mLb1LOSSQPlcsliAj370o0D5HCSH87HXn50kc3vssccavqMEIjA6yWC3aHat/G+SV/pzuBK5fOhDH0q2I488EijLY5Usx58jNRZ42/WxQPcYv1eqr/z2t79NNsnrl1xyyWSTPPSQQw5JtiuvvBIoyw532mknoFy7s136dyQPgiAIgiAIgiDoUbqSszwXqfIoVy7Jhb6Tm2nw2VQtsPTFc1//+tcbfsdnU6rVKlBexKbf0WJQgDPOOAOArbfeOtl8plJsuummQGciWu1y9tlnp20tdPc3ftk0MwJw0EEHAeVU5KLVbKnPFujatEpxr9kdTzqSQ9ffF86OZXX40XwnNyOn5B+eejTHxRdfnLZ1bc4555xk04ykz0JqFj2XkKHVOfVykoEc3hc1G+jtdI455mj4jqdy1yx6s1ny0dJJn7aKvqy22mpAOWqnvqNoEcCee+7Z8N1coiFFoPbZZ59kU5v0Wd9cSt7c7KpHHHLR3Fzaac1c+tiVi7p1knavVa6PeWIaJSA45ZRTkk0JhM4777yG7/o1cJ9qJtuv4csvvwzAww8/nGyKHIyEbvf9Vgli2k1TLf96G1ZU1Ntc7vkg96zg5PqAEkO5ukW0O953ilapyIUUNABLLLEEUD73hRdeGCiXe1HfzCWn8YjiMsssA5TvxVr0D/DHP/4RKI9BSuHuiSIUdfMol45bCQgA7rnnnobz61ZbbdU+hBKGTJw4seGYWqFkQo7Siuu6QDlaooiIP48efPDBQDniryRkft+TmsATN+hae3vXGHXNNde0dR7NmGmmmdK1VP/xhCnqr1/+8peTTWO+p7LfbbfdgPLYL5/7/Udjgj/36xylRINy9E/79FIISuXuqhg9LzzwwAPJpjHcj1VlJ7xkj9QN6jNQfn5rRkS0giAIgiAIgiAIOky8aAVBEARBEARBEHSYYUkHXeoymjBvs7CsL5TbbrvtgLqkD+pVxv/3f/832Tyxw5xzzgnkF2rm6pJ4eFnyBV+Q/P3vfx8oh+APPfTQIY+/Wyi07Pj5KMyaqzruEgyFdNuVdvhnc4uTndwiwdzvSFI0lpK2Vn9r1iZbyQgUsvfFrTkkBYN6m3VJhnA5XO74m1W371bihnaTtoym5pSfd0525nIFofEA6n3UfaoFrL/61a+GdSxjQSvpkGQSLgn56U9/CpT7oo91QtKRbbfdNtnU5lyqJ7mm17dxaZH27eOkro3bJOV2mYZwKafGA5d9uFyuk3Si/btvJZGR7Bdg8uTJAKy99trJpqQ33nZ9DL733nuBsrRIchmX0OTG8hzdlLe1Q87PzaTp/nndw7zv617mvs+Nkzl5u+PPEkJyzJx00BkLn7Z6ntIxuHRQiT2eeeaZZNOzkMtXlYTE65RpHPH+L8mVJ3jw/qo26ssPcseqOp+qIaXzg3Lb73QNy2ZssMEGQFmSrm1vL8cff3zDd9dcc02gvGRD/d/rY2kc9oRs8oUnZfJERerrfu86+eSTgXKdKEnUfD9KFOFJX9QPvM+pzay44orJpmfZ4TLPPPPw2c9+FqjXU8vVCXXJqZJIfO9730s2JaDQcz3U/edjoe57Lp3WdcjVx4W6D2688cZk073PpYpqf+5nJTTy+5mksN5O9PeR3K8iohUEQRAEQRAEQdBh4kUrCIIgCIIgCIKgwww766BCfaPJFCNpykILLZRsCit+8IMfTLYf/vCHQDmUrawsygoCZemgMo+5BEPhbP+cQuse1tZ3PASrrG8u1VHo17OPdFsG5/UIckgOkMti5tlbJMHYbLPN2v5tSQ48jHrVVVcB9fC8/91lAV7vQ+j6e+bIXqGZVG8o1G5anY9fB5GTY7YK8edkF81sYyHRbJdmx+JSIGVacptngBNeY2beeecFypmbPLOTaNd/440yAs4yyyzJJsmGn+NTTz0FlOvgSWLh4+AvfvELALbffvtkkxzCZRguSdb3c2OI+1bZBF2Spe96G9fYquxVUJdijEW9smbkfl/SQKhLYzzTmHzgEi2Rq98D9RqFXvNF10vSIGheg7AX22srdB/xY5cv/R6vtpTLEjxUnc2cVFHyOpd/SVLl9ZGES5+07262w1ZybP1N9YCgfj/1THSf/vSngXImNdWz8mcU+dXbpaR0ymYI5WcNLcW49tprk21gYAAoSzuffPJJoFwvaosttgDqdaNaoXZx//33t/X5VmhMczmj5Gjug8MPPxwoPxNJ1rf66qsnm87XswfrfFWPDOrPlsoa2A46d68P69vTsv7666ftq6++uuV+R8ODDz6YMinqec99qmdxl/fqOd4l4t/+9reBcm0qSQZd2q8Mtssuu2yy6Ttey/Duu+9O28r87P1C98Mf/ehHyaY+lcsWufnmmzfYXHaq3xgJEdEKgiAIgiAIgiDoMMOOaDWrTaBZJV98plkUr7qtt1ifOVVk6bjjjks2zdR4JEqJMfw4fCG3Znx9Jlxv1T7rqBnhX/7yl8m24447Nhz/IossAtRrAPj+NNvTTTQjkYt8+GyF6ovpeB2PROkaqRYG5KNbuRowfgy5xZCqM+Dkjvsb3/gGUK5Hk2MsZ7Wb/VarWSGdo88K+uJYkatFlOtPBx54YNref//92zqGbtPpRDi5fXh/UiIAn2WaOnVqw3d8UavarM8kDhVVaIexXLjdDB97DjvsMKDc1jRzp4XKkE8eoHbqf1Pf9v7ss3iaFfZ+rNqCV1xxRbJpFtojQIpIeFKH/fbbDyjXlRkP2q1L57Vh9t57b6DcZxVt9AXSUmrk6jhB3deKAkD9OnhkxiOZ0x5rr5NLTpE79hNPPBGAfffdN9nkXx8vW9XXbNbPPdGGlDM5csmccoxV5FW/c9NNNyWb6lr5WHnkkUcC9UgK1KMM/uykPuzXRrXc/LnhqKOOStu5pEQaPzyiJUWNt1mNBSuttFKyPfHEE0A5wUO3UFTPaXa9PELSLrpGXr9pOJGskdIsiuV0on0WRZGutT83TotHfHTf8CQXUur4PV1JR1RH0NG9AlqPe2rv3rY1XvvzrSJtPp5o7PBxQmO3v6PssssuAJx11llNjyVHRLSCIAiCIAiCIAg6TLxoBUEQBEEQBEEQdJhhSwfFRhttBJQXsSqUL2kY1MPVHqrT53xxu+R6vnjuggsuAOALX/hCsuUSVrjUT5JADwMqrL3zzjsnWy70qloGe+21V7JpQbL/nhJu5JIbDBeXZIlcuNdD9wrR+uJ31WlYd911G/bj3xUjSYbhPsid+yabbAKUazzk5JV+vcaTXDi6VTKM2267DYCJEycmm6RAyy+/fLLlpIM51JYcl3G1e4xjKSlqJrUazXF4e/74xz8OlGUGX//61xu+M2XKlLT9kY98BCDV/ID6Ymb17aEYb5/mfl+yH5dFSYLtn//Od74DlBNMKLmCSyk0Dnib23333YFyf/7973/fsB/v+5J0ubRDtaN8IbtkIV77JFfzZywk2NPSKiFK7nrovCXnhbqv5EeA6667Dmh9f/B7neSGPjb6wu9mx9Xtdpq7Rzm5Y8rJJiXvyX1eiVqgvmQgVysudy+DvLRQv+NJRYb6/nAY60QtOUnjhAkT0raSfnh70rOV1xKTlNclmfPPPz9QTkzmiQ5UF8+TtEg+5kl19Ntex0i/s+CCCyabnrvGK/FNs/tpN/tRu+fY7jGMd4IrTzqhPuX3b8lYTzrppGRTvSoluYO8ZDCHpPIu5fP7isYbH3N/8pOflL4LsM466wDwta99renv7bHHHkB96RPAr3/967aONUdEtIIgCIIgCIIgCDrMsKZ3ZplllhS10AzSoosumv6u9NZeDVpvmlqYBvXZUZ8l1QyMUoIC/PSnPwXKSSw0S6JFnFCeEdVCTZ/d+eY3v9lwLlqUmUsL69W07733XqCeOhrqEbJW1eU7iUcOc7N3mvX22SMtysydYys23XTTtK2Usj6znjsGtQW/Xn4dhM9KdJtc9KXdWaPcdz/xiU8A5VS6apO+6Pe0005ruu9cumKhdj/t7/QDrWbamvnefaE0zJ5kwPuA8NlZRbNXW221ZPNrMtxjHUtybU3H7qnXlQzI/agkPh7F13fdf7mkNrlIgW/rsx5l1faPf/zjZNMMu8++KyLjiZC0QN2TmDz22GN0g2aR15Fce0WbPJmI2qSnBldEqxV+XXW9fLwcy3FyNMjPHlG95557Gj6n88mNAZ7qXu3Uo3u5si2Orok/PygNt6eTVgRnNHQqGtMqwZBsnuZ72223Bcp96rLLLgPKKeulvPCkEzp3HyeUkMWfEfxaaCzwSIFSn3ukQCVbPKnJDTfcAJSjB1IW9OLYO1a/0YkI2niXwsglPnv88cfTtkebhMoG+LPjoYceCpRVFmqzfm9XdErPX1COVP/mN78Z8lhfeumltO0JXKbFE8co4YUnotI44veuZin4nYhoBUEQBEEQBEEQdJh40QqCIAiCIAiCIOgww5IOvv7660kaojCbJwBYa621Gr6zzz77AGU5ocKOHn7U4jlPfDHXXHMB5UrcCul5/Suv3eASxWnRgm2oywM9ZCnJghbt+uckS4K6bNEr13cbD93n5GaSTnjtiFwNnGa0qq3lsg3t08PW22yzDVCXfUE9bOuV5d2/40EudN+uvEhJRxxdm+FISSWVzS0aH6r+TjPGO5lDu0k7muHnLelBrl6b4/1XMhfvq7l6Uu0yHj71/aueleSCUJfn+tijMfPoo49ONo2nOQmH+1n+cym271syaZdcSCLmyTCUlMRlm1pQfPbZZyeb+orLi8dD+tKq7+eus6RQG2+8cbKpfcpPw8GlNhoT/dq0K/kebxlW7vc9CYOoVqtD7sPvLWojfs/Tdqt7me9H11AL4KGesEkJN8aTN954o636jQcddFCyKfmKyyo33HBDoCzB1bjncmuNCbkEBN4WJUWG+vOWy0KVoMivj54HfRy5/vrrgXI7fvTRR4c8z063404nuWg2TrSqy+fH0C+18IaLy9RzqN7bSM5f46wnzWmX008/PW3rt3Pjv9dCE14XUt/J1QhsRUS0giAIgiAIgiAIOsywIlp/+9vfuPLKK0s2n8XXLItX2FZUas8990w2LZz0SNSvfvUrALbaaqtkU8TLZ6lOPfVUAKZOnZpsXtV87bXXBvIL552PfexjQDmVqWZ6LrroomRT5MGrQevtOhfdGC6tZrWELyD0RdRiqaWWAspRQh2nL24fajExDC+SkptZVIpYXwg933zzAeVZgNFEGUZKNxeeyr9+XdQmczN4UF/omUs37OmjNRvT6fSwnaDTv5VrI75wO4cvmFUfcZ965GSkjHea91xigRyemOjVV18FytEp9W/NbEN9fPbEC96O5Uv3qfx81FFHJZuiaZ5G30sgjCedun5KS+wRLY2nWugNrfu+8ChYruSFrmG7dCKqnCN3j2rl09x9RvdaT9AgPDKtCIhHY2XzMcLvQbLn7o0jiTY2Y6wjiP58o8iRn7sSJimyBXWlkaflV5TRE7dIxfKHP/wh2VQuAuDYY48F4JRTTkm2FVZYASin+P7jH/8IlJ8/lITMkwzkGI+IbKd+s10lzEiScY13pFq0W4Ko0z4dq3T7Y3F/j4hWEARBEARBEARBh4kXrSAIgiAIgiAIgg4zLOlgqwrxCtG7hEW1VrzmisiFGpsls4B8mO+uu+5q+p0cAwMDw/5OO8fSCXL79cXokgC5/xTG1wJ6/05OxuFSAS1udXmiJ8aQLMOlhdr2GiU67pyk0iUfzerDjEW4vN1Fre0iGYfLX1rJh1TjTHLL3P5yxzfUMXY7/O3yodFIlJp9TnWXoO5Lo0YstQAAEuNJREFUt+XI9Qv3n0sLh8t4LFpu9ZvN/OdSHkl4XKb7wgsvAOX+p++4jNt9qrHD27YkdEcccUSyaZ9eR6vdGnad7PPtttPR+Dk3lrk8StKsVtJB93NOTu2LuJvR7Xbaqt5TjpxcTFJA1WGCugzzhBNOSDb5d8stt0w2JV4ZyieemGFaWsmPc4yFvKvdOlpeM0tJKbT8Auq1fLyO1i233AKUpYOS/B1//PHJpuukRC9QrhOk5AMuHf7kJz8JlOuXSha67LLLJtuvf/1rAL70pS81nFsv0a1kHNMzoxk/W90P2rW1u+/xThQGEdEKgiAIguD/t3fuQXZUdR7/fCUuYHgWz6gsYhLkVTFBQngoATY8lochLo/UohgCywKKhSusUJTe7awVoXBBQJQFRETZKAQoAgETAwyBQEJ45DGQLLVIACEhhAV20Q2F4ewffU7PuTM9d+6d2/cxM79PVSo9vz59+vTv/rr79Pn9zu8YhmEYhVOTRyuPVk3Sb/ZktlYTj8yHUf6JEydmspDi+brrrstkjzzyCFDuqcojTGSNR63yEmPEnoJQ55FHHpnJQvr8OJlIGP2K29DX5Nh66c/k7UrEugj1xCuCh+uJy4WkLAsWLMits9Io6xVXXJFtv/rqq2XnbQfqmaxaaZQp9rQEW+wrnXM80T14X+Jjwr3S1yhZO49m9pWkJoxyX3/99ZnsmGOOAco93Oeddx5QbpMh3XW89MWuu+6abYckBHGSgaDTa665JpONGDECKE+lPXv27IrtDjTKy1WEBzaP+fPnZ9vB1mLv1OTJk4HyBEp5xLYb9Bdz7rnnAnDvvfdmsoEy+p73zgn3d1jyA7rePbvvvnsmC/YV21xIghV7/uL3VYjQOPnkk3ucN/aKd3R0VH8RbUKc0CYklxo1alQme+KJJwB45ZVXMllY4iLW0Zw5c4DyVO0hGihOxx8vsbPHHnuUlYuJf4tgv3Gik+DlmjdvXiYLz5O8NPytimbp3o5Gt6XaNtSjl3Z+TjSyL5P3G7ZD38k8WoZhGIZhGIZhGAVjH1qGYRiGYRiGYRgFU1PoYN5E42rdrdWG+vUV5lFP/v5qJ9m1Y1hiHFYVwjIWL16cyYIbPw5XGT16NAAvvfRSJotXae9ObyFKQR67+0OIRqyrUC5v5fk4lCQOE2k0tf5+1ZaPw3lCmFYc3vn2229XPD6EdOStMh7WP6u3je1Cte2NQ3xCOGalSe5QvtZQsLE4cUMcEldNG9sp1KLa0Ic999wTKF8f69ZbbwXgrLPOymQhlPjYY4/NZGGdwBBqBOX6C/d0HMYVfps4XC5sx2GH1V5To3RexNp5eeHSDzzwQCY7+OCDgXLbDeFdfdUdJxKKdR4IiUrqmRzeKiqtYxmHXQfiZ154R8XJFk4//XQAbr755kw2fPjwHvXE4WzBZuMkESHpy7Rp0zJZuFfyaORE+rz+VF7fY+XKlZls++23B+DJJ5/MZOHei0MMw7soXt9twoQJQPlaVyHEPdblmjVrsu2DDjoI6AohhK5EWXEip3DuWBYSbEydOjWTTZo0Cch/P4brXbVqVY999dAOz/T+JI+op93tcM3dqSfBTNH99Gava2YeLcMwDMMwDMMwjIIpNL17taPCtY4S1ZNKspZ2VXu+as9bJGHCK3SNHsWjhiGl+tVXX93j2EaMfobV4/N0cMopp2TbYcQ8HhFfunRp4e2phVr1EY9qBw9LnB4/pHOOR6XzUurHdHZ2AuXpcAOlUinbDul1Y1qVgKbIc+V5NGKdhgnbfekx2Bd0eQhiW4sniHen6OdU0VR7ruAhjpPMhCQ1sacgPCPGjBmTyR577LEe9cVegeDFiRM3BI91XC7Yfux5P+SQQ3q0IXjIYk9uM72Jtf5+cbr6PMKE/9gjGJIOxHpesWJFth2uc/z48ZkspD6Pib217UZfv1Wepz4QJ0mp9HvEXptAnAQi9opUWlok9qAdf/zxQHkSpzyPVrXROfWQ15/K00dIiARdSwbEKdjDuyROhhPeIfH9uNtuuwHldhXebfGzI04nH1Lxx8s/BO9XqA+6vJCxl3uXXXYByp/RsTetO63ywlSKHGh2wqS+3jm1vvtbpdOi+9D9eVfXc2zR+jOPlmEYhmEYhmEYRsHYh5ZhGIZhGIZhGEbB1L2O1kCi3ScQV2pfHMJy4oknAuVu+rlz5zauYTVy5513ZtthVfg4NGHmzJlA/oTEdlzbKE7kkRdmEMLTwsR46ArZCOFTUB7+GUI28kKGdtxxxx6ydgwXjKl1zYq83z4OhQ2TtGO77+scQefxxO5qk2FUS6ttMoQMxaFZwf4uvvjiTBbsLp4Mv3r1agBOOumkTBbWvZkyZUomi+/fcL6QmAFg+vTpAJx22mmZLIQT3nTTTZksL6lLpZCyVt/7/QljOfTQQ4GuZzJ0hReHNcqgPHQwECcvyHsOhNCrVoew5q1LmEccPhls8dJLL81ks2bNKr5xNRKeMS+//HKLW1I9cTKMO+64AygPiQ7JWW677bZMFkJ9Q3noCieMw/dGjhwJlPcl4nDDsB0nuArv8riesB2HywZ7iJ8dlWhUMoxqz1utrN53ca3JHlr9zhmKFK1z82gZhmEYhmEYhmEUTL/TuxdNq1bibjXxhNhKI/avv/56th3S1VZKoRvTW9r2ZhA8bfFo55IlS3qUa9Qq3kX/9nntCymHJ06cmMnuuusuoNyLFRNGsMOoNXR5HPISFFRLM2296KUW4hHZvLTDlVIgQ35q8Xg0uN2o9t6PyfMIhdTiM2bM6LEvJGsAePzxx3vIFi5cCMA+++yTyeJlI4ItnnnmmZls7dq1AJx66qmZLE6W0V+Ktt1m3AvXXnstAOvWrctkIbV2fG/HhN84TjAQmDx5cra9YMGCXs/bju+0OOlB8JTGdHR09Hps/I6q5PWsl/Bcjj04lWi1lxXKl7i4++67gXwPaUiKAbBs2bIe+2+55RYAXnvttUwWEtrE/Ys4Oiaka49TvgdGjBjRQxbraNy4cUB5ev1G/rb9pR36tO387Gtkv3+oYB4twzAMwzAMwzCMgrEPLcMwDMMwDMMwjIJRLa5cSW8BvS9MM7TZ3Tm3U60HmU4rYjotHtNp8ZhOi8d0Wjym0+Lpl07B9FoB02ljsPu/eKrSaU0fWoZhGIZhGIZhGEbfWOigYRiGYRiGYRhGwdiHlmEYhmEYhmEYRsHYh5ZhGIZhGIZhGEbBNPdDS7oM6XmkFUjLkCYUWPfhSPcXVt9gIk/v0hqkHXPKfhnpkl7qORzpkEY3t9Uo0Q5KtMz/W6dEr0d//1Wr2zeokDZ5m3weaTnSd5BsAChG2sHraBnSOqTXo78r22Ol56J0M9I+vey7EOkT3WSXIJ2OdFKvxw1mzFYrY3bafki7Iv0G6SWkZ5AeQNqzxjq2Qzq/QS1sL6yP2ngaoWOpA+mAuss0iJoWLK4L6WDgBGB/nPvAd/Lbo9MqDcO5v/RdcABSq96dmwPMyalnGHA48D6QvwrvIMGV3NvAWAAl+hfgfVdyP4rLKJEAuZL7qBltUqJhrjQobfT/cG4sANLOwH8A2wClslKD+R7tC9dljyi1R1y5Pfaz3rNz5dJmwIXAr4E/R3uOAU4FrgTuB16ouw0DC7PVSpidthfpKtD3AL/Euale9nlgF+DFGmraDjgf+GnRTWwrrI/aeNpZxw2kmaNxI4ANOPcBAM5twLk3vGclQXoWaSXSXgBIw5FuQXoK6TmkyV7+GaTHfPlncz0s0nh/zMgK9UxDmoP0MPBQMxTQIvL1nnJBjt6nIf3Eb9+KdAPSEuAO4Fzg234U4kvNvpBWo0SjlOgFJbodeB4YoURfVaKVStSpRDN9uWFK9G503FQlujna7lSi5Ur0SFT+KiV6SolWKNHZXj5JiTqU6H5gZdMvuNk4tx44B/gmknLvUelipKV+NCzxsuFIc72XoRPpNC+/HOkFX7b+Dl+7I02MPAjPIW3t92yFNBtpNdLtvgNWPsInvY/0b0jLgcuATwKPoNRGkbYhfSGOBr4MXOnPMxJpLNJir+d7kLaP6r/Gl+tEOrB5ymgwZqv9x+y0WRwBfIhzN2QS55YDjyNd6a91ZWSDWyE9FPUJJvujLgdGev1c2eyLaCLWR208ven4+/5Z2Yl0Y7d7/wqvmxcJ/U5pS1JP7Sqke4AtszNIP0N6mtRrljT7AvNonkcL5gPfR3oRWAD8Fuce9fs24Nz+pO7pi4CzSR+iD+PcdKTtgKeQFgDrgaNwbiPSaGAW0OUOTI36OmAyzr2KNLOXegD2B8bg3H83+uJbSK16786ngUNwbhNFjlIOXPYCznAl97QSfRr4Aan9vQcsUKITgN9VOL4EHO5K7k0l2s7LzgHWu5I7UIk2BxYr0Xy/7wBgH1dyrzbkatoN5/5AOlK9s5d03aPS0aQdqAMBAXOQDgN2At7AueMBkLZF2gGYAuyFc87f+4Odi4Bv4NwipK2AjV4+DtgXeANYBBwKPN7t2OHAEpz7DgDSdOAInNvg908CHsK5J5DmAPfj3GxfdgVwAc49ijSD1MYv9Md9AufG+t/pFmC/oi+6ZZit9hez0+awH/BMjvwrpJ7HzwM7AkuRFgJvAVNw7n9IPQ2LvQ4vAfbLvLmDF+ujNp7edPwTnJsBgPQrUq/Xff6YYTh3INJxpPfsJOA84M84tzfSGODZ6ByX+WfwZsBDSGNwbkVTrq4XmufRcu594Aukncq3gN8iTfN77/b/PwN8xm8fDVyCtAzoALYA/hr4OHAT0krgTiCOwd4buBE4EZd1THurB+D3g8iA86ld7925E+c2NbKJA4yXXMk97bcnAA+7ktvgSu5D0lCiw/o4fhFwm/dahfvvaOBMJVoGLCEN1Rjt9z05ZD6y8onv0aP9v+dIH6x7keppJXCUH/n6Es69R/rhuxH4OdJXKA8tGqwsAq5C+hawXRRq8hTO/RHnPgKWkX+vbwLuqlD3scCDPaTStv5coUPyS8rvgVkAOLcQ2GaQf0SYrVaH2Wlr+SIwC+c24dybwKPAeNIBgZn+g3QB8CnSMMOhgfVRG0/vOj4CaYnX2ZGkAy6BPN0fRhoyjP+Iij+kTkV6lvTZuy/l+m8JzfRo4TvsHUCHV+jX/Z4P/P+bojYJ+Duc+8+yOlKvypukozEfo2s0DGAtqZGOIx0Vq1TPBOBPdV7RwKA2vXdnaOioeqrRx0ekdhfYItr+B9IPtBOAZ5VonC97viu5svAAJZpU5fkGD9JnSe1xvZfE1y/ghzj37znH7Q8cB/wA6SGcm0EaAvQ3wMnAN0kf4IMH6Ruk9gRwHM5djjSXVA+LkI7x+z6IjurtXt/Yx4DKgaSjiLXi+vh74GK2Wh1mp63ieVJ7qpbTST2uX8C5D5HWUP7uGvxYH7Xx9NTxPwJjgANw7jWvv9juqumnpkh7kHocx+PcO0i30gY23DyPlvQ570YNjAVeqXDEPNI5RCFWc5yXbwus9aNeXwM2i455Fzge+CHS4X3UMzSoXe+V+F9g6z5LDR2WAEf4LIXDgKnAoz5BxjtKNFqJPkYaFhT4rCu5xcD3gHdIRw3nAef7OlCizynRlgw1pJ2AG0jDCPI6OvOA6T7cCKRPIe2M9EnSMIJfk06A39+X2RbnHgC+TfrSG1w4dz3OjfX/3kAaiXMrce4KYCmpF6W/dN3r0r7A6qiD27Uv9ci8Q9ecza+RjpAHwvyPLwLv+fIDH7PV6jE7bRUPA5sjnZNJ0jCrd4HTkDbzdnwY8BRp32q9/8g6AtjdHzU03vvWR208+ToOH5gb/LOwmsGBhcDf+zr3I/1QgzQ50Z+A95B2Af62iGbXSzM9WlsB13mX/F+A/yJ1H57QS/l/BX4MrCBNofuyL/tT4C6kM0jnwpR/8Tv3JtIJwIM+fru3eoYKteq9EvcBs0kna16Ac48V18yBhyu5PyrR90hHZwTc50purt/9XdIH6HpSl/fmXn61Eu3hy893JdepRKtIQwWWKRH+mMkMDbb0IRMfJ7XPXwFX5ZZ0bj7S3sCTpO+k94GvAqNIJ71/BHxIOqK9NXAv0hakuv6nBl9HO3Ch7yB9RDqa/SBwcD/ruhH4HdIbwFzK5x3+hjQ05lukL8WvAzeQptn+A3BmVHYj0nOkv+/0fralXTBbLQaz02aQzvebAvwY6buknpU1pPPStgKWk3ru/hnn1iHdDtznvQxPA6t9PW8jLULqBB7EuYubfzFNwfqojac3Hb8LdALrSAdf+uJnwC+QVgGrCHMRnVvu7+PVwGukYcotR/mDcYZhGIYBSL8HzsC5tTUe1wFchMvmNBpG4zA7NQyjDWnuHC3DMAxjYOHcUa1ugmH0idmpYRhtiHm0DMMwDMMwDMMwCqaZCxYbhmEYhmEYhmEMCexDyzAMwzAMwzAMo2DsQ8swDMMwDMMwDKNg7EPLMAzDMAzDMAyjYOxDyzAMwzAMwzAMo2D+H4JH3iXyxG6HAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 1080x216 with 20 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "x_orig = ((x_original   ) * 255).astype('int')[:, :, :, 0]\n",
    "x_adv  = ((x_adv_samples) * 255).astype('int')[:, :, :, 0]\n",
    "\n",
    "y_pred_orig = model.predict(x_original,    verbose=0)\n",
    "y_pred_adv  = model.predict(x_adv_samples, verbose=0)\n",
    "\n",
    "fig    = plt.figure(figsize=(15, 3))\n",
    "cols   = 10\n",
    "rows   = 2\n",
    "images = list(x_orig[:cols])      + list(x_adv[:cols])\n",
    "preds  = list(y_pred_orig[:cols]) + list(y_pred_adv[:cols])\n",
    "labels = list(y[:cols])           + list(y[:cols])\n",
    "\n",
    "for i in range(0, len(images)):\n",
    "    ax = fig.add_subplot(rows, cols, i+1)\n",
    "    y_pred = np.argmax(preds[i])\n",
    "    y_orig = np.argmax(labels[i])\n",
    "    ax.set_xlabel(get_label(y_pred),\n",
    "                  color = \"green\" if y_pred == y_orig else \"red\")\n",
    "    ax.tick_params(axis='both', which='both',\n",
    "                   bottom=False, top=False,\n",
    "                   right=False, left=False,\n",
    "                   labelbottom=False, labelleft=False)\n",
    "    plt.imshow(images[i], cmap='gray')\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Summary and Next Steps <a id=\"summary\"></a>\n",
    "\n",
    "This notebook only looked at one adversarial robustness technique (FGM). The *ART* library contains many more attacks, metrics and defenses to help you understand and improve your model's robustness. You can use this notebook as a template to experiment with all aspects of *ART*. Find more state-of-the-art methods for attacking and defending classifiers here:\n",
    "\n",
    "https://github.com/IBM/adversarial-robustness-toolbox"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Acknowledgements\n",
    "\n",
    "Special thanks to [Anupama-Murthi](https://github.ibm.com/Anupama-Murthi) and [Vijay Arya](https://github.ibm.com/vijay-arya) who created the original notebook which we modified here to showcase how to use *ART* with *FfDL*. If you would like to try *[Watson Machine Learning (WML) Service](https://console.bluemix.net/catalog/services/machine-learning)* with *ART* check out Anupama and Vijay's notebook here:\n",
    "\n",
    "[https://github.ibm.com/robust-dlaas/ART-in-WML/Use ART to check robustness of deep learning models.ipynb](https://github.ibm.com/robust-dlaas/ART-in-WML/blob/master/Use%20ART%20to%20check%20robustness%20of%20deep%20learning%20models.ipynb)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Copyright © 2018 IBM. This notebook and its source code are released under the terms of the MIT License."
   ]
  }
 ],
 "metadata": {
  "anaconda-cloud": {},
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}