Introduction to TensorFlow with Keras.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div class=\"alert alert-block alert-info\">\n",
    "    <h1>Introduction to Tensorflow with Keras</h1>\n",
    "    <h3>Author: Kaushik Manjunatha </h2>\n",
    "    <h4>This is a simple tutorial to work with TensorFlow and Keras</h3>\n",
    "\n",
    "    \n",
    "</div>\n",
    "\n",
    "One of the most well known deep learning frameworks is tensorflow with a keras backend.\n",
    "\n",
    "One core advantage of deep learning frameworks is that they let us quickly piece together models consisting of different types of layers without having to worry about how to train the network using backpropagation.\n",
    "\n",
    "Tensorflow, as the name suggests, works on tensors. A [tensor](https://en.wikipedia.org/wiki/Tensor#As_multidimensional_arrays) can be interpreted as a multidimensional array or a matrix of higher order (e.g. a matrix with dimensions $d_0 \\times d_1 \\times \\dots \\times d_n$). Tensorflow knows how to train a model by using tensors and a method called [automatic differentiation](https://en.wikipedia.org/wiki/Automatic_differentiation).\n",
    "\n",
    "A tensorflow model is then a sequence of operations on a tensor. Examples for these operations are applying an activation function, multiplying a tensor by a weight matrix or adding tensors. Tensorflow builds a computation graph of these operations, allowing it to apply the [chain rule ](https://en.wikipedia.org/wiki/Chain_rule) to each of the operations in the computation graph to compute the gradient in respect to each trainable weight in the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Types of layers in Tensorflow with Keras\n",
    "\n",
    "Tensorflow comes with a set of predefined layers. Here a few:\n",
    "\n",
    "## [Core Layers](https://keras.io/api/layers/core_layers/)\n",
    "\n",
    "### [Input Object](https://keras.io/api/layers/core_layers/input/)\n",
    "Instantiates a tensor object on which operations can be performed. The input object has no trainable weights and is used for converting the input to our model into a tensor. The most important parameter here is *shape*, which tells us the dimensionality of the input to our model. **Attention:** The input object is the only part of the model where you need to specify the dimensionality of the input. All subsequent layers automatically figure out the dimensionality of their respective inputs.\n",
    "\n",
    "\n",
    "### [Dense Layer](https://keras.io/api/layers/core_layers/dense/)\n",
    "Creates a fully connected neural network layer. Here you can specify the output dimension (*units*) and the activation function (*activation*).\n",
    "\n",
    "### [Activation Layer](https://keras.io/api/layers/core_layers/activation/)\n",
    "Applies an activation function to an output of a model. Has no trainable weights.\n",
    "\n",
    "### [Embedding Layer](https://keras.io/api/layers/core_layers/embedding/)\n",
    "For NLP applications inputs are usually given as integer indexes. This is similar to one hot embeddings. In contrast to what we saw in previous assignments, the one hot embedding of a token used here is just an integer corresponding to the index of the one hot embedding that is $1$. Example: Traditional one hot embedding of a word might be $\\vec{w} = \\begin{pmatrix} 0 & 0 & 0 & 1 & 0 & 0\\end{pmatrix}^T$. Then the integer encoding of the word would be $w_{ie} = 3$.\n",
    "\n",
    "The embedding layer turns the integer encoded tokens into $n$-dimensional vectors. The parameters for the embedding layer are the input dimension (*input_dim*, size of vocabulary) and the output dimension (*output_dim*, size of the embeddings). Usually the weights of this layer are trainable. It is also possible to provide the weights using pretrained embeddings (e.g. Word2Vec, FastText) and set the weights to non trainable.\n",
    "\n",
    "---\n",
    "\n",
    "\n",
    "## [Recurrent Layers](https://keras.io/api/layers/recurrent_layers/)\n",
    "\n",
    "### [LSTM Layer](https://keras.io/api/layers/recurrent_layers/lstm/)\n",
    "\n",
    "A simple LSTM layer. The most important parameter is *units*, which defines the dimensionality of the output.  \n",
    "Other parameters include:\n",
    "- *return_sequences*: If this is set to *True*, the LSTM will return all outputs (one per input). Default is *False*, meaning the LSTM only returns the output of the last time step.\n",
    "- *return_states*: If this is set to *True*, the LSTM will return the last hidden state alongside the output of the LSTM. Default is *False*, meaning the LSTM will only return the output of the last timestep and not the last hidden state.\n",
    "\n",
    "### [TimeDistributed Layer](https://keras.io/api/layers/recurrent_layers/time_distributed/)\n",
    "\n",
    "This layer is a wrapper around a standard layer which allows us to apply that layer to a sequence of inputs.  \n",
    "Example: Say you have an LSTM that returns a sequence of outputs (one per input, *return_sequences* is set to *True*). Now you want to apply a fully connected dense layer to this sequences. Assume each output of the sequence has a length of $32$. This would mean we want a fully connected layer with $32$ inputs and apply it to each element of the sequence. For this we can create the dense layer the following way: ```dense_layer = TimeDistributed(Dense(32))```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1: XOR Problem\n",
    "\n",
    "### Defining the architecture of the model\n",
    "\n",
    "Let us create a simple neural network to solve the xor problem.\n",
    "\n",
    "This network should have inputs of size $2$, a single hidden layer and an output layer, producing outputs of size $1$.\n",
    "\n",
    "The following cell shows how to define this model with tensorflow / keras."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X ([0, 0], [0, 1], [1, 0], [1, 1])\n",
      "Y (0, 1, 1, 0)\n",
      "\n",
      "Model: \"XOR_Model\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "Input (InputLayer)           [(None, 2)]               0         \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (None, 32)                96        \n",
      "_________________________________________________________________\n",
      "Output (Dense)               (None, 1)                 33        \n",
      "=================================================================\n",
      "Total params: 129\n",
      "Trainable params: 129\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.layers import Input, Dense\n",
    "\n",
    "# Define a basic neural network for the xor problem\n",
    "training_data = [\n",
    "    ([0, 0], 0), # 0 xor 0 = 0\n",
    "    ([0, 1], 1), # 0 xor 1 = 1\n",
    "    ([1, 0], 1), # 1 xor 0 = 1\n",
    "    ([1, 1], 0)  # 1 xor 1 = 0\n",
    "]\n",
    "\n",
    "# Split the training data into two lists X (inputs) and Y (outputs)\n",
    "X, Y = zip(*training_data)\n",
    "print('X', X)\n",
    "print('Y', Y)\n",
    "print()\n",
    "# Our inputs have a dimensionality of 2, so create an input object for this\n",
    "# Notice that the shape is given as the tuple (2, ), not the integer 2\n",
    "\n",
    "input_object = Input(shape=(2,), name='Input')\n",
    "\n",
    "# Create a dense NN layer with 4 outputs\n",
    "hidden_layer = Dense(32, activation='relu')\n",
    "# Apply this to the inputs\n",
    "y_hidden = hidden_layer(input_object)\n",
    "\n",
    "# Create another dense NN layer with a single output\n",
    "output_layer = Dense(1, activation='sigmoid', name='Output')\n",
    "# Apply the output of the hidden layer to this layer\n",
    "y = output_layer(y_hidden)\n",
    "\n",
    "# Piece everything together into a model\n",
    "model = Model(input_object, y, name='XOR_Model')\n",
    "\n",
    "# Print a summary of our model\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Example 1: XOR Problem\n",
    "\n",
    "### Making the model trainable\n",
    "\n",
    "To make the model trainable we need to compile it. For this we need the following three parameters:\n",
    "\n",
    "- loss: The loss function of our network (e.g. cross-entropy, mean square error, ...)\n",
    "- optimizer: How we minimize the loss of the network (almost always Adam)\n",
    "- metrics: Which metrics you want to evaluate during training (usually Accuracy)\n",
    "\n",
    "After this we train our model using the method ```fit```. This method needs the following parameters:\n",
    "\n",
    "- x: The inputs to the model. Here you give all inputs at once\n",
    "- y: The desired outputs of the model. You give all at once\n",
    "- epochs: Number of epochs to train\n",
    "- batch_size: How many samples the model should look at for one update of the gradient"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input   Predicted   True_Output\n",
      "[0, 0]       0.41          0.00\n",
      "[0, 1]       0.69          1.00\n",
      "[1, 0]       0.82          1.00\n",
      "[1, 1]       0.20          0.00\n"
     ]
    }
   ],
   "source": [
    "# Compile the model with the loss, ...\n",
    "model.compile(\n",
    "    loss='mse',\n",
    "    optimizer='adam',\n",
    "    metrics=['accuracy', 'binary_accuracy']\n",
    ")\n",
    "\n",
    "# Train the model on the inputs\n",
    "history = model.fit(\n",
    "    x=X,\n",
    "    y=Y,\n",
    "    epochs=500,\n",
    "    batch_size=2,\n",
    "    verbose=0 # Do not print after each epoch (we have 500 epochs and don't want to clutter the output)\n",
    ")\n",
    "\n",
    "# Predict all outputs\n",
    "predictions = model.predict(X)\n",
    "\n",
    "print('Input   Predicted   True_Output')\n",
    "for xi, yp, yi in zip(X, predictions, Y):\n",
    "    print(f'{str(xi):8}{yp[0]:9.2f}{yi:14.2f}')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Plotting the model history\n",
    "\n",
    "The training history of the model is stored in the variable ```history```.\n",
    "\n",
    "We can access the loss, accuracy and additional metrics as:\n",
    "\n",
    "```history.history['loss'], history.history[metric], ...```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAEWCAYAAAB2X2wCAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjMsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+AADFEAAAgAElEQVR4nO3de3hddZ3v8fc392vTppe0TdqmpUWuQpmMVWSGADoWZgYFOVwFAbWDzyDw6NEDB56O4uBh1NGR0QMyggyI7YBKB4EZ4AAZROUqF0sLQym0Sa9p0qZJc2329/yx1k53dnZz2U2affm8nifP3nuttXd+vxA++fW7fuu3zN0REZH0lzPZDRARkfGhQBcRyRAKdBGRDKFAFxHJEAp0EZEMoUAXEckQCnTJGGZWa2ZuZnmjOPZyM3sumc8xs/9tZj851PaKjDcFukwKM3vfzHrNbEbc9tfCMK2dnJaNzN2/5e6fH+k4M2swsxGPExkvCnSZTO8BF0VfmNnxQPHkNSd1WED/f8qY6BdGJtN9wGUxrz8L3Bt7gJlVmNm9ZtZsZpvM7KZo0JlZrpl918x2mdlG4C8TvPcuM9tmZlvM7O/NLHcM7bvEzDaHn39jzOd+3cx+Fj4vMrOfmVmLme0xs5fMrMrMbgH+DPihmXWY2Q/D408Oj2kLH0+O+dwGM7vFzH4LdAJfMbNX4vr0FTNbM4Y+SBZRoMtkeh6YYmZHh0F7AfCzuGP+GagAFgGnEvwBuCLc9wXgr4ClQB1wXtx7/xXYDywOj/kLYCwlkFOADwBnACvN7OgEx3w2bN88YDpwFdDl7jcCvwGudvcyd7/azCqBR4HbwmO/BzxqZtNjPu9SYAVQHh63MO77fobgD6HIEAp0mWzRUfrHgbeALdEdMSF/g7u3u/v7wD8ShB7A+cA/uXuju7cC/yfmvVXAmcB17r7P3XcC3wcuHEPbvuHuXe7+OvA6cEKCY/oIwnmxu/e7+yvuvvcgn/eXwDvufp+773f3VWGf/zrmmHvc/c1wfw/wbwQhjpkdC9QCj4yhD5JFRpwNIDLB7gOeBRYSV24BZgAFwKaYbZuA6vD5XKAxbl/UAiAf2GZm0W05ccePZHvM806gLMEx9xGMzleb2VSCf2Hc6O59CY6dG9fGaJurY17Ht+9fgVVmdhPBH7IHwqAXGUIjdJlU7r6J4OToWcCv4nbvIhgBL4jZNp8Do/htBGEauy+qEegBZrj71PBrirsfO87t73P3b7j7McDJBCWg6HmB+KVMtzK4L9E2b4l5Peg97v480EtQj78YlVtkGAp0SQWfA053932xG929H3gAuMXMys1sAfBlDtTZHwCuMbMaM5sGXB/z3m3AE8A/mtkUM8sxsyPM7NTxbLiZnWZmx4flob0Ef4D6w907CGr/UY8BR5rZxWaWZ2YXAMcwcgnlXuCHwH53Tzh3XgQU6JIC3P1dd3/5ILu/BOwDNgLPAT8H7g73/QvwOEF9+w8MHeFfRlCyWQfsBn4BzBnXxsPs8HP3AuuB/+LAH5wfAOeZ2W4zu83dWwhG8F8BWoCvAX/l7rtG+B73Aceh0bmMwHSDC5HUZmbFwE7gJHd/Z7LbI6lLI3SR1PdF4CWFuYxEs1xEUpiZvQ8Y8KlJboqkAZVcREQyhEouIiIZYtJKLjNmzPDa2tqk3rtv3z5KS0vHt0EpTn3ODupzdjiUPr/yyiu73H1mon2TFui1tbW8/PLBZqoNr6Ghgfr6+vFtUIpTn7OD+pwdDqXPZhZ/tfEAlVxERDKEAl1EJEMo0EVEMoQCXUQkQyjQRUQyxIiBbmZ3m9lOM1t7kP1mZreZ2QYze8PMThr/ZoqIyEhGM0K/B1g+zP4zgSXh1wrg9kNvloiIjNWI89Dd/Vkzqx3mkE8C93qwhsDzZjbVzOaE61HLRPrjL6D57cluxYSp3fQ+RH472c04rNTn7FDeMQuoH/fPHY8Li6oZfNuspnDbkEA3sxUEo3iqqqpoaGhI6ht2dHQk/d50lajPf/5fV5HjfTiW+E1pbgHgB72EIjOpz9khf/7lE5Jh4xHoidIk4Ypf7n4ncCdAXV2dJ3ullK4si27cD3/+Vez0myalTRNN/52zQzb2uXWC+jwes1yaGHxfxxqCeyfKRHIHHEwTlUQkMB5p8DBwWTjb5cNAm+rnh4FHgkfLndx2iEjKGLHkYmarCKr3M8ysCfg7IB/A3e8guPHtWcAGoBO4YqIaKzEi4X2IczRCF5HAaGa5XDTCfgf+dtxaJKPjYaBrhC4iIQ3v0tXACF2BLiIBBXq60ghdROIo0NOVRugiEkeBnq40y0VE4ijQ05VmuYhIHKVBuhqooes/oYgElAbpSiUXEYmjQE9XOikqInEU6OlK0xZFJI4CPV1FwpKLRugiElKgpyudFBWROEqDdKUauojEUaCnK9XQRSSOAj1daYQuInEU6OlKI3QRiaNAT1fRWS46KSoiIaVBuopeKaq1XEQkpDRIVyq5iEgcBXq60klREYmjQE9XGqGLSBwFerrSCF1E4ijQ05WWzxWROAr0dKU7FolIHKVBulINXUTiKNDTlWroIhJHgZ6utHyuiMRRGqQrnRQVkTgK9HSlOxaJSBwFerpSyUVE4igN0pVOiopIHAV6utK0RRGJo0BPVxqhi0gcBXq60ghdROIo0NOVZrmISJxRBbqZLTezt81sg5ldn2D/fDN7xsxeNbM3zOys8W+qDKJZLiISZ8Q0MLNc4EfAmcAxwEVmdkzcYTcBD7j7UuBC4P+Od0MlTkSBLiKDjSYNPgRscPeN7t4LrAY+GXeMA1PC5xXA1vFroiTkKrmIyGDm7sMfYHYesNzdPx++vhRY5u5XxxwzB3gCmAaUAh9z91cSfNYKYAVAVVXVn6xevTqpRnd0dFBWVpbUe9NVfJ/nbf4lR2y8l2f/7AEiuYWT2LKJo//O2UF9HpvTTjvtFXevS7QvbxTvtwTb4v8KXATc4+7/aGYfAe4zs+Pco8PI8E3udwJ3AtTV1Xl9ff0ovv1QDQ0NJPvedDWkz8++DBvhz0+th7zMDHT9d84O6vP4GU3JpQmYF/O6hqEllc8BDwC4+++BImDGeDRQDkKLc4lInNEE+kvAEjNbaGYFBCc9H447ZjNwBoCZHU0Q6M3j2VCJowuLRCTOiIHu7vuBq4HHgfUEs1neNLObzezs8LCvAF8ws9eBVcDlPlJxXg6N9wMGlqgiJiLZaDQ1dNz9MeCxuG0rY56vAz46vk2TYUX6NToXkUE0iTldeb/q5yIyiAI9XWmELiJxFOjpyiO6SlREBlEipCuPqOQiIoOM6qSoJNDZCs/cAn1dh+XbfWD7NtjzwIENTS9Bjv4ei8gBCvRkbX4eXvoJlFVBbsGEf7tp3d3Q9c7gjYs/NuHfV0TShwI9WZH9weNnfgmzj5/wb/d8Fl4eLSJjo3+zJ0t3DBKRFKNAT5YuvReRFKNAT5YWxxKRFKNAT9bAHYO0loqIpAYFerJ0xyARSTEK9GTppKiIpBgFerJ0UlREUowCPVkaoYtIilGgJyuiGrqIpBYFerIGRuj6EYpIalAaJUs1dBFJMQr0ZKmGLiIpRoGeLI3QRSTFKNCTpRq6iKQYpVGy3INHlVxEJEUo0JOlkouIpBgFerK8HzAtziUiKUOBnqxIv0bnIpJSFOjJ8n7Vz0UkpSjQk6URuoikGAV6sjyiEbqIpBQFerIi/ZCjH5+IpA4lUrK8XxcViUhKUSIlSyUXEUkxCvRk6aSoiKQYBXqyNG1RRFKMAj1ZkYhG6CKSUkYV6Ga23MzeNrMNZnb9QY4538zWmdmbZvbz8W1mCtJJURFJMXkjHWBmucCPgI8DTcBLZvawu6+LOWYJcAPwUXffbWazJqrBKUM1dBFJMaMZYn4I2ODuG929F1gNfDLumC8AP3L33QDuvnN8m5mCVEMXkRQz4ggdqAYaY143AcvijjkSwMx+C+QCX3f3/4z/IDNbAawAqKqqoqGhIYkmQ0dHR9LvHS/H7thOSVc3Lx2mdqRCnw839Tk7qM/jZzSBnmh9WE/wOUuAeqAG+I2ZHefuewa9yf1O4E6Auro6r6+vH2t7AWhoaCDZ946b7f8CrXsPWztSos+HmfqcHdTn8TOakksTMC/mdQ2wNcEx/+7ufe7+HvA2QcBnrohOiopIahlNIr0ELDGzhWZWAFwIPBx3zBrgNAAzm0FQgtk4ng1NOR7RWi4iklJGTCR33w9cDTwOrAcecPc3zexmMzs7POxxoMXM1gHPAF9195aJanRK0ElREUkxo6mh4+6PAY/FbVsZ89yBL4df2UHTFkUkxahmkCyN0EUkxSjQk6VL/0UkxSjQk6VL/0UkxSiRkqUauoikGAV6slRDF5EUo0BPlkboIpJiFOjJUg1dRFKMEilZ7iq5iEhKUaAnK9KvS/9FJKUokZKlk6IikmIU6MnSSVERSTEK9GRphC4iKUaBnixd+i8iKUaBniyN0EUkxSjQk6VZLiKSYka1HnpKWfsrTnjt+7Bp2sGPMYMp1dDWdGDbEafDKdcN/9kdO+HX10LvvpHb0dmiEbqIpJT0C/RIPzmRPtjfc/BjtrwMkf2QXwqzj4eWd6CtceRA3/Y6vP0YVB0HBWXDH1tTB0cuH3v7RUQmSPoF+gf/B6+2zhz+jtnfWQL7dsKso+Bzj8OvVsDm50f+7Eh/8Hj2P0P1SePSXBGRwyUzi8DR2ScW8+iRkd/n/YPfLyKSRjIz0KNBHg3mnJwDo+/hRI9RbVxE0lBmBnp09smgEfooAl0jdBFJY5kZ6BYzMocgoDVCF5EMl5mBnrCGPpoRemTw+0VE0khmBvqQGnpucKn+SAZG6Jn5YxGRzJaZyRUN5NjHsdTQFegikoYyM7mGlFxyRjltUSUXEUlfmRnoFnMyNPqok6IikuEyM9BzYkbmoGmLIpIVMjPQE54UHc0IPTL4/SIiaSQzAz3RtEUc3Id/38AIPTN/LCKS2TIzuRKN0GHkUbpq6CKSxjIz0BPNcoGR6+iqoYtIGsvMQB+Y5RI320UjdBHJYJkZ6Alr6GiELiIZbVSBbmbLzextM9tgZtcPc9x5ZuZmVjd+TUxCoitFYRQj9Ogsl8z8OycimW3E5DKzXOBHwJnAMcBFZnZMguPKgWuAF8a7kWN2sJOiI10t6gp0EUlfo0muDwEb3H2ju/cCq4FPJjjum8C3ge5xbF9yDlpyGSnQ+4MwN5u4tomITJDRBHo10BjzuincNsDMlgLz3P2RcWxb8oZc+j/akku/ToiKSNoazU2iEw1XB67QMbMc4PvA5SN+kNkKYAVAVVUVDQ0No2pkvI6OjmHfe0xLK7OAxqatvNvQwJyt7/IB4He/e47ewukHfd+iTe9TjfGbJNs1kUbqcyZSn7OD+jx+RhPoTcC8mNc1wNaY1+XAcUCDBaWK2cDDZna2u78c+0HufidwJ0BdXZ3X19cn1eiGhgaGfW/zv0IzzFuwgHn19fCHzfDfcPKyD8HUeQd/X8+TsD1/+M+eJCP2OQOpz9lBfR4/oym5vAQsMbOFZlYAXAg8HN3p7m3uPsPda929FngeGBLmh1XS0xYjmrIoImlrxEB39/3A1cDjwHrgAXd/08xuNrOzJ7qBSTmUS/81w0VE0tRoSi64+2PAY3HbVh7k2PpDb9Yhil02N/ZxNLNcNEIXkTSV2cNRzXIRkSyS2YEenU8+lsW5VHIRkTSVmek1EORjLbnopKiIpK/MDPSoMZ8UjajkIiJpK7MDPZnVFnW3IhFJU5mdXkNG6COUXHRSVETSWGYHejJ3LFINXUTSVGYHejJ3LNIIXUTSVGYHui79F5EskpmB7uFikLr0X0SySIanly4sEpHskZnpNXBhUdwFRu6Jj49SyUVE0lhmBno8reUiIlkgOwJ9TBcWKdBFJD1lR6Dr0n8RyQLZEei69F9EskB2pJcuLBKRLJAdga47FolIFsiOQNcsFxHJApkZ6NH55tFHzXIRkSyQmYEez0Y7Qo/oSlERSVuZmV7xV4rmjLaGrkAXkfSVHemlkouIZIHsCHTdsUhEskDaBXrrvl52do4QzPF0xyIRyQJ5k92AsVr14ma+82wXd6xrYNHMUhbNLGPRjOCxdnoJlaUFQzulS/9FJAukXaCfs7Sa99/bSEdBORub9/HsO7vo3X9gxF6Un8P3SndzFvAf63axL9LE4qnGiYA/9z3slZ8e/MP3btGl/yKSttIu0OdOLeYvFxVQX/8nAPRHnK17uni3uYPG1k7e29XJmu1foGVHMTevXUjfH18HnP+Z90kW7WumpC+P0sJcSgvzKCnIo7Qgl5LCPIrzc8iZcyKccPHkdlBEJElpF+jxcnOMeZUlzKssidl6DHA6F/VH2NTayXvN+9jUeiwvtOxjU2snm1s6adzeSV+/D/qcuVOLmLc7l/mVb1A9tZi54Vf11GLmTC0iP1ejdxFJXWkf6MPJy83hiJllHDGzbMi+/oizra0rCPfdnTS2drG5NXj+/9bvYFdH76DjC3JzqJ1REoZ7cRj4RcytCEJ/doUCX0QmV0YH+nByc4yaaSXUTCtJuL+7r59tbd1s3dPFlt1dvLurg3d37mNbWxevNe5hd2ffoOPNYFZ54aBR/dyKooHXc6cWM60kH4te7CQiMs6yNtBHUpSfy8IZpSycUZpwf2fvfrbu6WZbW1cQ+nuC8N+6p4t1W/fy5Lodg07WBp+ZExP2QRln4PXUYuZUFFGUr1k2IpIcBXqSSgryWDyrjMWzhpZzANydln29AyG/NRr4bUH4v7V9J83tPUPeN6OsgDkVYTknJuy37+nnmL3dzCgrJCdHo3wRGUqBPkHMjBllhcwoK+SDNVMTHtOzv58dbT1sGQj9LraGZZ6Nzft47p1d7Os9MHf+5uefIj/XhgR+9HU0/EsL9Z9VJBvp//xJVJiXy/zpJcyfnriO7+7s7drP1rYuHv/Ni0yft3igtLOtrYsXNrayfW83/REf9L6K4vww7IuYX1nKwpmlzJtWHJ4zKFZZRyRDjSrQzWw58AMgF/iJu98at//LwOeB/UAzcKW7bxprY/r6+mhqaqK7u3vY4yoqKli/fv1YPz6tnbpkOkVFXdRNBSgKv4LQj3gwa2d/xGnvM57bFuH91m6adnfx2w0tdPUNvkJ2Rlkh8yuLWTC9NJjyOa14YOrn7ClF5KqkI5KWRgx0M8sFfgR8HGgCXjKzh919XcxhrwJ17t5pZl8Evg1cMNbGNDU1UV5eTm1t7bCzQdrb2ykvLx/rx6e10fTZ3WlpaWHB9HYWLjwegEjE2dHezZbdXTTt7qIpnKK5qXUfL77XyprXtgzcBwQgL8eYO7WYmmnFzJtWwrzKIOznh1+VpQWaqSOSokYzQv8QsMHdNwKY2Wrgk8BAoLv7MzHHPw98JpnGdHd3jxjmcnBmxvTp02lubh7YlpMT1NznVBRTVzv0Pb37I2zd0zUwF79pdyeNYfA/9dZOdnUMPnFbWpDLvMpguue8ymjoB89rppVQpvq9yKQxdx/+ALPzgOXu/vnw9aXAMne/+iDH/xDY7u5/n2DfCmAFQFVV1Z+sXr160P6KigoWL148YqP7+/vJzc2uOvBY+rxhwwba2trG5fv29Du7upzmzgg7O53mrgjN4eOuLqcnbr2zsnyYWZzDjBJjRnEOM4uNWSXGzOIcphcbeWMo53R0dFBWlngWUaZSn7PDofT5tNNOe8Xd6xLtG81wKtH/gQn/CpjZZ4A64NRE+939TuBOgLq6Oq+vrx+0f/369aMqpajkMryioiKWLl06wS0KSjyt+3pp3N1FY2snTbujI/3g+eubu+jtPzAXP8eCtXii5ZvYUs78yhKmxl141dDQQPzvSKZTn7PDRPV5NIHeBMyLeV0DbI0/yMw+BtwInOruQydYp4mysjI6OjomuxlpwcyYXlbI9LJCTpw3dGpmtH6/uaUzWFahNXjc3Jp4eYXywrxg1k8Y9t27+uDtncyvLKF6WjGFedn1rzKRsRpNoL8ELDGzhcAW4EJg0JKEZrYU+DFBaWbnuLdS0lJs/X7ZoulD9u/r2U/j7s4hgf/2jnaeemsnvfsj3LvuJSBYWqGqvIj5lSXUhLX7aPDPqyymqrxIF1xJ1hsx0N19v5ldDTxOMG3xbnd/08xuBl5294eB7wBlwIPhP5k3u/vZh9Kwb/z6TdZt3ZtwX7I19GPmTuHv/vrYMb9v06ZNXHnllTQ3NzNz5kx++tOfMn/+fB588EG+8Y1vkJubS0VFBc8++yxvvvkmV1xxBb29vUQiEX75y1+yZMmSMX/PbFBamMdRs6dw1OwpQ/ZFIs6/P/EMNUedOGgBtcbWTn7/bgsP7R08O6cgL4eaqcUDAT8o8KeVUFGSfxh7JjI5RjUlwd0fAx6L27Yy5vnHxrldKeXqq6/msssu47Of/Sx3330311xzDWvWrOHmm2/m8ccfp7q6mj179gBwxx13cO2113LJJZfQ29tLf/8Id0mShHJyjGlFOfxpbSV/Wls5ZH/P/n627O6icXewSmZTuFLm5tZOXmvcQ1vX4MXTphTlDdTs4+feV0/VxVaSGVJ2jtlwI+nDfVL097//Pb/61a8AuPTSS/na174GwEc/+lEuv/xyzj//fM4991wAPvKRj3DLLbfQ1NTEueeeq9H5BCnMyw1uP5hgaWSAtq6+8ORstJwTnLCNLefEmlleSM20Ymqnl1I7vZTqacFyCvMrS5hTUayLrSQtpGygp7LoTIw77riDF154gUcffZQTTzyR1157jYsvvphly5bx6KOP8olPfIKf/OQnnH766ZPc4uxTUZxPRXUFx1VXDNkXiTjNHT0DdfvoRVebWzt5YWMLD726ZdDxeTlG9bTiITNz5oVLKcTPzhGZLAr0UTj55JNZvXo1l156Kffffz+nnHIKAO+++y7Lli1j2bJl/PrXv6axsZG2tjYWLVrENddcw8aNG3njjTcU6CkmJ8eomlJE1ZSihOWc6Fr4W2KmYQY3P+niP9dup3Xf4Nk5ZYV51IRr5cRfbDVvWokWS5PDRr9pcTo7O6mpqRl4/eUvf5nbbruNK6+8ku985zsDJ0UBvvrVr/LOO+/g7pxxxhmccMIJ3HrrrfzsZz8jPz+f2bNns3LlyoN9K0lRI62F397dF9TtY+bfN4XB/7t3d9HZO/i8SWVpQbA4Wjiqj15VO29asDqm6vcyXhTocSKRSMLtTz/99JBt0bp6rBtuuIEbbrhh3NslqaO8KJ9j51Zw7Nyh5Zz4i61il1R4c0sbT7y5fdC9bAGqphQOrITZv7eX7SWbB2bn6F62MhYKdJFxNNLFVv0RZ8fe7qB2v6crZv2cTv6weTdbdvfxyMY/DhyfYzCnIlwsLaZuHy3pzCrX6phygAJd5DDKDVeznDu1OOH+p55+hiNPXEbj7k6aYhZLa2zt5DfvNLNj7+CLsPNzjeqpB+r30ZF+9PXMskKdsM0iCnSRFJKbYwPz4zli6P7uvn627Bm8FHI09J9cN3Q5hcK8nIGAjz5WTysOnyvwM40CXSSNFOXncsTMMo44yPz7zt79A7NzmuIWTXujaQ+7OwdfcBW9wvZAyJeEI/5gm0o66UWBLpJBSgryWFJVzpKqxBfedfQEgb9lTxD0sTc+eXLb3iEj/LwcY87UIuZWHLhheXU4O6c6vK9tSYFiJFXov4RIFikrzOMDs8v5wOzEgd/V2z8Q9k27D9y8fMueLl54L/E9bKeV5A/cqHxORTFzphYxpyJ4PreimKqKQq2UeZgo0BN46KGHOPfcc1m/fj1HHXXUZDdH5LApLshl8axyFs9KHPj7+yPsbO9hS0zQb9ndxba24B62L72/e8g6OgAzygrClTeDUf3siqKB57u6IvT1RzQ9cxwo0BNYtWoVp5xyCqtXr+brX//6hHyPbLzrkqS/vNycYWfpQFDH39bWzbY93WxrC8J+W1sXW/d0s6mlk99vbKG9e/+g93z12f9gZlkhc6YWM7eiiNkVQZknGvzRK3sL8hT6w0ndQP+P62H7HxPuKu7fD7lJNH328XDmrcMe0tHRwW9/+1ueeeYZzj777IFA//a3v819991HTk4OZ555JrfeeisbNmzgqquuorm5mdzcXB588EEaGxv57ne/yyOPPAIEKzXW1dVx+eWXU1tby5VXXskTTzzB1VdfTXt7O3feeSe9vb0sXryY++67j5KSEnbs2MFVV13Fxo0bAbj99ttZs2YN1dXVXHvttQDceOONVFVVcc0114z95yAygUoK8oY9cQtBLX/bniDsn3nxNaZULRgI/3d2dvDsfzezr3foSqUzygqZEwZ+7GPVlKDEM3tKEcUF2TtQSt1AnyRr1qxh+fLlHHnkkVRWVvKHP/yBHTt2sGbNGl544QVKSkpobW0F4JJLLuH666/nnHPOobu7m0gkQmNj47CfX1RUxHPPPQdAS0sLX/jCFwC46aabuOuuu/jSl77ENddcw6mnnspDDz1Ef38/HR0dTJkyhcsuu4xrr72WSCTC6tWrefHFFyf2hyEyQcoKD5y8jWzNp77+yCHHtHf3sb2tm21t3Qce9wah39jayYvvtSYs70wtyWf2lCDsZ4cj+/jn0zJ0QbXUDfRhRtJdE7h87qpVq7juuusAuPDCC1m1ahWRSIQrrriCkpISACorK2lvb2fLli2cc845QBDUo3HBBRcMPF+7di033XQTe/bsoaOjg0984hNAsMzAvffeCzBw84ycnBymT5/Oq6++yo4dO1i6dCnTpw+9C5BIpigvyqe8KP+gM3YgKO9sHxT4QXlne/h87Za9tOzrGXQzFAima1ZNKWT2lCJmTSmiqryI2RWFVE0pYlZ5EPpVUwrTbgZPerV2grW0tPD000+zdu1azIz+/n7MjE9/+tND/pp7/G9IKC8vb9B6MN3d3YP2l5YeWPDp8ssvZ82aNZxwwgncc889NDQ0DNu+z3/+89xzzz1s376dK6+8coy9E8k8JQV5w66LD9AXnsjd3tbNjr3dBx73Bo/rt+7lmb07hyyqBsF9bmdNKQwCvjwM/ymFYU2/kFnlRcyakjqzeBToMX7xi19w2WWX8eMf/3hg26mnnkplZSV33303F1988UDJpXBck+kAAAgDSURBVLKykpqaGtasWcOnPvUpenp66O/vZ8GCBaxbt46enh66u7t56qmnBpbbjdfe3s6cOXPo6+vj/vvvp7q6GoAzzjiD22+/neuuu47+/n727duHmXHOOeewcuVK+vr6+PnPf35YfiYi6S4/N4fqqcE8+oNxdzp69rNjbw87wqCPfb59bzcvvNfKzvbuIYurQVDmqQrDfWZ5GPTlhcHrskJmTQleT/RSygr0GKtWreL6668ftO3Tn/4069ev5+yzz6auro6CggLOOussvvWtb3HffffxN3/zN6xcuZL8/HwefPBBFi1axPnnn88HP/hBlixZwtKlSw/6/b75zW+ybNkyFixYwPHHH097ezsAP/jBD1ixYgV33XUXubm53H777Rx33HEUFBRw2mmnMXXqVM2QERlHZjZQ4lk86+CjfXdnd2ffQNDv3NvDzvYg/Lfv7WZnew/v7uyguaMnYfCXFOQyq7yQ5dX91E9EPw5WOphodXV1/vLLLw/atn79eo4++ugR33u4b0GXCtrb2yktLeWkk07iwQcfHPbWdqP9Oaa6hoYG6uvrJ7sZh5X6nBncnT2dfexsDwK/ub0neL63h+aOHo7Kb+Vvzzsjqc82s1fcvS7RPo3Q08Rbb73FBRdcwDnnnKP7lIqkODNjWmkB00oLEl6VO9L5smQp0NPEUUcdNTAvXUQkkZS77GqySkCZQj8/keyVUoFeVFRES0uLQilJ7k5LS8uo58SLSGZJqZJLTU0NTU1NNDc3D3tcd3d31oXWaPtcVFQ06CbXIpI9UirQ8/PzWbhw4YjHNTQ0DDsdMBNlY59FZGxSquQiIiLJU6CLiGQIBbqISIaYtCtFzawZ2JTk22cAu8axOelAfc4O6nN2OJQ+L3D3mYl2TFqgHwoze/lgl75mKvU5O6jP2WGi+qySi4hIhlCgi4hkiHQN9DsnuwGTQH3ODupzdpiQPqdlDV1ERIZK1xG6iIjEUaCLiGSItAt0M1tuZm+b2QYzu37kd6QHM7vbzHaa2dqYbZVm9qSZvRM+Tgu3m5ndFv4M3jCzkyav5ckzs3lm9oyZrTezN83s2nB7xvbbzIrM7EUzez3s8zfC7QvN7IWwz/9mZgXh9sLw9YZwf+1ktj9ZZpZrZq+a2SPh64zuL4CZvW9mfzSz18zs5XDbhP5up1Wgm1ku8CPgTOAY4CIzO2ZyWzVu7gGWx227HnjK3ZcAT4WvIej/kvBrBXD7YWrjeNsPfMXdjwY+DPxt+N8zk/vdA5zu7icAJwLLzezDwD8A3w/7vBv4XHj854Dd7r4Y+H54XDq6Flgf8zrT+xt1mrufGDPnfGJ/t909bb6AjwCPx7y+Abhhsts1jv2rBdbGvH4bmBM+nwO8HT7/MXBRouPS+Qv4d+Dj2dJvoAT4A7CM4KrBvHD7wO858DjwkfB5XnicTXbbx9jPmjC8TgceASyT+xvT7/eBGXHbJvR3O61G6EA10Bjzuinclqmq3H0bQPg4K9yecT+H8J/WS4EXyPB+h+WH14CdwJPAu8Aed98fHhLbr4E+h/vbgOmHt8WH7J+ArwGR8PV0Mru/UQ48YWavmNmKcNuE/m6n1Hroo2AJtmXjvMuM+jmYWRnwS+A6d99rlqh7waEJtqVdv929HzjRzKYCDwFHJzosfEzrPpvZXwE73f0VM6uPbk5waEb0N85H3X2rmc0CnjSzt4Y5dlz6nW4j9CZgXszrGmDrJLXlcNhhZnMAwsed4faM+TmYWT5BmN/v7r8KN2d8vwHcfQ/QQHD+YKqZRQdYsf0a6HO4vwJoPbwtPSQfBc42s/eB1QRll38ic/s7wN23ho87Cf5wf4gJ/t1Ot0B/CVgSniEvAC4EHp7kNk2kh4HPhs8/S1Bjjm6/LDwz/mGgLfrPuHRiwVD8LmC9u38vZlfG9tvMZoYjc8ysGPgYwcnCZ4DzwsPi+xz9WZwHPO1hkTUduPsN7l7j7rUE/78+7e6XkKH9jTKzUjMrjz4H/gJYy0T/bk/2iYMkTjScBfw3Qd3xxsluzzj2axWwDegj+Gv9OYLa4VPAO+FjZXisEcz2eRf4I1A32e1Pss+nEPyz8g3gtfDrrEzuN/BB4NWwz2uBleH2RcCLwAbgQaAw3F4Uvt4Q7l802X04hL7XA49kQ3/D/r0efr0ZzaqJ/t3Wpf8iIhki3UouIiJyEAp0EZEMoUAXEckQCnQRkQyhQBcRyRAKdMlYZtYfrnQX/Rq31TnNrNZiVsYUSQXpdum/yFh0ufuJk90IkcNFI3TJOuE61f8Qrkv+opktDrcvMLOnwvWonzKz+eH2KjN7KFzD/HUzOzn8qFwz+5dwXfMnwis/RSaNAl0yWXFcyeWCmH173f1DwA8J1hYhfH6vu38QuB+4Ldx+G/BfHqxhfhLBlX8QrF39I3c/FtgDfHqC+yMyLF0pKhnLzDrcvSzB9vcJbjKxMVwcbLu7TzezXQRrUPeF27e5+wwzawZq3L0n5jNqgSc9uFEBZva/gHx3//uJ75lIYhqhS7bygzw/2DGJ9MQ870fnpGSSKdAlW10Q8/j78PnvCFYEBLgEeC58/hTwRRi4OcWUw9VIkbHQiEIyWXF4Z6Co/3T36NTFQjN7gWBQc1G47RrgbjP7KtAMXBFuvxa408w+RzAS/yLBypgiKUU1dMk6YQ29zt13TXZbRMaTSi4iIhlCI3QRkQyhEbqISIZQoIuIZAgFuohIhlCgi4hkCAW6iEiG+P+ndzba1vXQUAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.title('Model history')\n",
    "plt.plot(history.history['loss'], label='Loss')\n",
    "plt.plot(history.history['accuracy'], label='Accuracy')\n",
    "\n",
    "\n",
    "plt.xlabel('Epoch')\n",
    "plt.legend()\n",
    "plt.grid()"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}