Tensorflow - Complete Tensorflow 2 and Keras Deep Learning Bootcamp - JP/06-NLP-and-Text-Data/00_Generating_Text_with_RNNs_Shakespears.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HWLJUpM048Ef"
   },
   "source": [
    "# Text Generation with Neural Networks\n",
    "\n",
    "In this notebook we will create a network that can generate text and it is being done character by character. \n",
    "\n",
    "Very awesome write up on this here: http://karpathy.github.io/2015/05/21/rnn-effectiveness/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "cUZKpyFO5Sql",
    "outputId": "15b4db35-88b7-4529-d1e5-6ac856e06f53"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Drive already mounted at /content/drive; to attempt to forcibly remount, call drive.mount(\"/content/drive\", force_remount=True).\n"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kp6Mu2MJ48Eq"
   },
   "source": [
    "+ [Step 1: The Data](#step1)\n",
    "    + [Understanding unique characters](#vocab)\n",
    "+ [Step 2: Text Processing](#step2)\n",
    "    + [Text Vectorization](#vectorization)\n",
    "        + [Character to Index](#chartoindex)\n",
    "        + [Index to Character](#indextochar)\n",
    "    + [Encoding the characters of the whole text](#encoding)\n",
    "+ [Step 3: Creating Batches](#step3)\n",
    "    + [First we need to understand the text we are using](#understand)\n",
    "    + [Training Sequences](#trainingsequences)\n",
    "        + [Create Training Sequences](#createtrainingsequences)\n",
    "        + [Creating Sequence Batches](#createsequencebatches)\n",
    "        + [Generating Training Batches](#generatingtrainingbatches)\n",
    "+ [Step 4: Creating the Model](#step4)\n",
    "    + [Setting up Loss Function](#setuplossfunction)\n",
    "    + [Setting up Model Function](#modelfunction)\n",
    "+ [Step 5: Training the modell](#step5)\n",
    "+ [Step 6: Generating text](#step6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "e8VlyAD348Er"
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "\n",
    "import tensorflow as tf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OeIIcG1l48Et"
   },
   "source": [
    "# <a name='step1'> Step 1: The Data </a>\n",
    "\n",
    "You can grab any free text you want from here: https://www.gutenberg.org/\n",
    "\n",
    "We'll choose all of shakespeare's works (which we have already downloaded for you), mainly for two reasons:\n",
    "\n",
    "1. Its a large corpus of text, its usually recommended you have at least a source of 1 million characters total to get realistic text generation.\n",
    "\n",
    "2. It has a very distinctive style. Since the text data uses old style english and is formatted in the style of a stage play, it will be very obvious to us if the model is able to reproduce similar results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "id": "A8bWneCK48Eu"
   },
   "outputs": [],
   "source": [
    "path_to_file = '/content/drive/MyDrive/TS2/shakespeare.txt'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "id": "PlC-Z8HV48Ev"
   },
   "outputs": [],
   "source": [
    "text = open(path_to_file, mode='r').read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 72
    },
    "id": "rxNoVImI48Ev",
    "outputId": "4315eb1c-c504-49e1-ed0b-ace759991718"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "\"\\n                     1\\n  From fairest creatures we desire increase,\\n  That thereby beauty's rose might never die,\\n  But as the riper should by time decease,\\n  His tender heir might bear his memory:\\n  But thou contracted to thine own bright eyes,\\n  Feed'st thy light's flame with self-substantial fuel,\\n  Making a famine where abundance lies,\\n  Thy self thy foe, to thy sweet self too cruel:\\n  Thou that art now the world's fresh ornament,\\n  And only herald to the gaudy spring,\\n  Within thine own bu\""
      ]
     },
     "execution_count": 16,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "text[:500]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SQ81b6FN48Ex"
   },
   "source": [
    "## <a name='vocab'>Understanding unique characters </a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Lv9O8CME48Ex",
    "outputId": "c5bc2e8d-2393-4c39-d401-60c40188a4ac"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\n', ' ', '!', '\"', '&']"
      ]
     },
     "execution_count": 17,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab = sorted(set(text))\n",
    "vocab[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zu4oPAsH48Ey",
    "outputId": "f4732494-fadf-4bee-ab1f-218e6715a6c2"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "84"
      ]
     },
     "execution_count": 18,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(vocab)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-4b_3ZRx48Ey"
   },
   "source": [
    "--------\n",
    "# <a name='step2'>Step 2: Text Processing</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OqnXHs0K48Ez"
   },
   "source": [
    "## <a name='vectorization'>Text Vectorization</a>\n",
    "\n",
    "We know a neural network can't take in the raw string data, we need to assign numbers to each character. Let's create two dictionaries that can go from numeric index to character and character to numeric index."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xw3xJHCE48Ez"
   },
   "source": [
    "### <a name='chartoindex'>Character to Index</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "id": "7-zjTNUb48Ez"
   },
   "outputs": [],
   "source": [
    "char_to_index = {char:index for index,char in enumerate(vocab)}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "9CGEVZjk48E0",
    "outputId": "853a416f-e104-4ef3-d8d6-067481a38a74"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'\\n': 0,\n",
       " ' ': 1,\n",
       " '!': 2,\n",
       " '\"': 3,\n",
       " '&': 4,\n",
       " \"'\": 5,\n",
       " '(': 6,\n",
       " ')': 7,\n",
       " ',': 8,\n",
       " '-': 9,\n",
       " '.': 10,\n",
       " '0': 11,\n",
       " '1': 12,\n",
       " '2': 13,\n",
       " '3': 14,\n",
       " '4': 15,\n",
       " '5': 16,\n",
       " '6': 17,\n",
       " '7': 18,\n",
       " '8': 19,\n",
       " '9': 20,\n",
       " ':': 21,\n",
       " ';': 22,\n",
       " '<': 23,\n",
       " '>': 24,\n",
       " '?': 25,\n",
       " 'A': 26,\n",
       " 'B': 27,\n",
       " 'C': 28,\n",
       " 'D': 29,\n",
       " 'E': 30,\n",
       " 'F': 31,\n",
       " 'G': 32,\n",
       " 'H': 33,\n",
       " 'I': 34,\n",
       " 'J': 35,\n",
       " 'K': 36,\n",
       " 'L': 37,\n",
       " 'M': 38,\n",
       " 'N': 39,\n",
       " 'O': 40,\n",
       " 'P': 41,\n",
       " 'Q': 42,\n",
       " 'R': 43,\n",
       " 'S': 44,\n",
       " 'T': 45,\n",
       " 'U': 46,\n",
       " 'V': 47,\n",
       " 'W': 48,\n",
       " 'X': 49,\n",
       " 'Y': 50,\n",
       " 'Z': 51,\n",
       " '[': 52,\n",
       " ']': 53,\n",
       " '_': 54,\n",
       " '`': 55,\n",
       " 'a': 56,\n",
       " 'b': 57,\n",
       " 'c': 58,\n",
       " 'd': 59,\n",
       " 'e': 60,\n",
       " 'f': 61,\n",
       " 'g': 62,\n",
       " 'h': 63,\n",
       " 'i': 64,\n",
       " 'j': 65,\n",
       " 'k': 66,\n",
       " 'l': 67,\n",
       " 'm': 68,\n",
       " 'n': 69,\n",
       " 'o': 70,\n",
       " 'p': 71,\n",
       " 'q': 72,\n",
       " 'r': 73,\n",
       " 's': 74,\n",
       " 't': 75,\n",
       " 'u': 76,\n",
       " 'v': 77,\n",
       " 'w': 78,\n",
       " 'x': 79,\n",
       " 'y': 80,\n",
       " 'z': 81,\n",
       " '|': 82,\n",
       " '}': 83}"
      ]
     },
     "execution_count": 20,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char_to_index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "-bNF06E248E0",
    "outputId": "96c2deaf-e7be-40b0-ed0c-f2fffb5091bc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "33"
      ]
     },
     "execution_count": 21,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "char_to_index['H']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "poFiZij748E1"
   },
   "source": [
    "### <a name='indextochar'>Index to Character</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "id": "Ww6RgMWd48E1"
   },
   "outputs": [],
   "source": [
    "index_to_char = np.array(vocab)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "FqSzFWov48E1",
    "outputId": "afd97dab-6ba8-4980-dc6f-af5517f2df4c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['\\n', ' ', '!', '\"', '&', \"'\", '(', ')', ',', '-', '.', '0', '1',\n",
       "       '2', '3', '4', '5', '6', '7', '8', '9', ':', ';', '<', '>', '?',\n",
       "       'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M',\n",
       "       'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z',\n",
       "       '[', ']', '_', '`', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i',\n",
       "       'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v',\n",
       "       'w', 'x', 'y', 'z', '|', '}'], dtype='<U1')"
      ]
     },
     "execution_count": 23,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_to_char"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 36
    },
    "id": "UGbgv33o48E2",
    "outputId": "7c50dcbc-c063-4a16-9738-a54c7e14a2af"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "'0'"
      ]
     },
     "execution_count": 24,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "index_to_char[11]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "-binYIF348E2"
   },
   "source": [
    "## <a name='encoding'>Encoding the characters of the whole text</a>\n",
    "+ direct mapping for each character by character"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "id": "z4nmBePJ48E2"
   },
   "outputs": [],
   "source": [
    "encoded_text = np.array([char_to_index[c] for c in text])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "QH6drQDL48E3",
    "outputId": "6889cac0-3d2e-40da-bb1d-6f29830109bc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  1,  1, ..., 30, 39, 29])"
      ]
     },
     "execution_count": 26,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Nd1zo_6y48E3",
    "outputId": "d432577e-3e0d-49f2-9bfa-557273b0ced1"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(5445609,)"
      ]
     },
     "execution_count": 27,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "encoded_text.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "kmvoUwFY48E3"
   },
   "source": [
    "We now have a mapping we can use to go back and forth from characters to numerics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 72
    },
    "id": "l8DqsqBt48E4",
    "outputId": "1051f337-633d-4ef6-cd7e-807ea8bdf315"
   },
   "outputs": [
    {
     "data": {
      "application/vnd.google.colaboratory.intrinsic+json": {
       "type": "string"
      },
      "text/plain": [
       "\"\\n                     1\\n  From fairest creatures we desire increase,\\n  That thereby beauty's rose might never die,\\n  But as the riper should by time decease,\\n  His tender heir might bear his memory:\\n  But thou contracted to thine own bright eyes,\\n  Feed'st thy light's flame with self-substantial fuel,\\n  Making a famine where abundance lies,\\n  Thy self thy foe, to thy sweet self too cruel:\\n  Thou that art now the world's fresh ornament,\\n  And only herald to the gaudy spring,\\n  Within thine own bu\""
      ]
     },
     "execution_count": 28,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's say for the sample text (Original Text)\n",
    "sample = text[:500]\n",
    "sample"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "sbZvZttE48E4",
    "outputId": "ee72e18b-6dd3-4e19-956d-552569969671"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 0,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,  1,\n",
       "        1,  1,  1,  1,  1, 12,  0,  1,  1, 31, 73, 70, 68,  1, 61, 56, 64,\n",
       "       73, 60, 74, 75,  1, 58, 73, 60, 56, 75, 76, 73, 60, 74,  1, 78, 60,\n",
       "        1, 59, 60, 74, 64, 73, 60,  1, 64, 69, 58, 73, 60, 56, 74, 60,  8,\n",
       "        0,  1,  1, 45, 63, 56, 75,  1, 75, 63, 60, 73, 60, 57, 80,  1, 57,\n",
       "       60, 56, 76, 75, 80,  5, 74,  1, 73, 70, 74, 60,  1, 68, 64, 62, 63,\n",
       "       75,  1, 69, 60, 77, 60, 73,  1, 59, 64, 60,  8,  0,  1,  1, 27, 76,\n",
       "       75,  1, 56, 74,  1, 75, 63, 60,  1, 73, 64, 71, 60, 73,  1, 74, 63,\n",
       "       70, 76, 67, 59,  1, 57, 80,  1, 75, 64, 68, 60,  1, 59, 60, 58, 60,\n",
       "       56, 74, 60,  8,  0,  1,  1, 33, 64, 74,  1, 75, 60, 69, 59, 60, 73,\n",
       "        1, 63, 60, 64, 73,  1, 68, 64, 62, 63, 75,  1, 57, 60, 56, 73,  1,\n",
       "       63, 64, 74,  1, 68, 60, 68, 70, 73, 80, 21,  0,  1,  1, 27, 76, 75,\n",
       "        1, 75, 63, 70, 76,  1, 58, 70, 69, 75, 73, 56, 58, 75, 60, 59,  1,\n",
       "       75, 70,  1, 75, 63, 64, 69, 60,  1, 70, 78, 69,  1, 57, 73, 64, 62,\n",
       "       63, 75,  1, 60, 80, 60, 74,  8,  0,  1,  1, 31, 60, 60, 59,  5, 74,\n",
       "       75,  1, 75, 63, 80,  1, 67, 64, 62, 63, 75,  5, 74,  1, 61, 67, 56,\n",
       "       68, 60,  1, 78, 64, 75, 63,  1, 74, 60, 67, 61,  9, 74, 76, 57, 74,\n",
       "       75, 56, 69, 75, 64, 56, 67,  1, 61, 76, 60, 67,  8,  0,  1,  1, 38,\n",
       "       56, 66, 64, 69, 62,  1, 56,  1, 61, 56, 68, 64, 69, 60,  1, 78, 63,\n",
       "       60, 73, 60,  1, 56, 57, 76, 69, 59, 56, 69, 58, 60,  1, 67, 64, 60,\n",
       "       74,  8,  0,  1,  1, 45, 63, 80,  1, 74, 60, 67, 61,  1, 75, 63, 80,\n",
       "        1, 61, 70, 60,  8,  1, 75, 70,  1, 75, 63, 80,  1, 74, 78, 60, 60,\n",
       "       75,  1, 74, 60, 67, 61,  1, 75, 70, 70,  1, 58, 73, 76, 60, 67, 21,\n",
       "        0,  1,  1, 45, 63, 70, 76,  1, 75, 63, 56, 75,  1, 56, 73, 75,  1,\n",
       "       69, 70, 78,  1, 75, 63, 60,  1, 78, 70, 73, 67, 59,  5, 74,  1, 61,\n",
       "       73, 60, 74, 63,  1, 70, 73, 69, 56, 68, 60, 69, 75,  8,  0,  1,  1,\n",
       "       26, 69, 59,  1, 70, 69, 67, 80,  1, 63, 60, 73, 56, 67, 59,  1, 75,\n",
       "       70,  1, 75, 63, 60,  1, 62, 56, 76, 59, 80,  1, 74, 71, 73, 64, 69,\n",
       "       62,  8,  0,  1,  1, 48, 64, 75, 63, 64, 69,  1, 75, 63, 64, 69, 60,\n",
       "        1, 70, 78, 69,  1, 57, 76])"
      ]
     },
     "execution_count": 29,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Let's see the encoded form of the above sample text, this is direct mapping character by character\n",
    "encoded_sample = encoded_text[:500]\n",
    "encoded_sample"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "sNbBhmFt48E4"
   },
   "source": [
    "---------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "NtTpbCRf48E5"
   },
   "source": [
    "# <a name='step3'>Step 3: Creating Batches</a>\n",
    "\n",
    "+ Overall what we are trying to achieve is to have the model predict the next highest probability character given a historical sequence of characters. \n",
    "+ Its up to us (the user) to choose how long that historic sequence. \n",
    "    + Too short a sequence and we don't have enough information (e.g. given the letter \"a\" , what is the next character) , \n",
    "    + too long a sequence and training will take too long and most likely overfit to sequence characters that are irrelevant to characters farther out. \n",
    "+ While there is no correct sequence length choice, you should consider \n",
    "    + the text itself, \n",
    "    + how long normal phrases are in it, and \n",
    "    + a reasonable idea of what characters/words are relevant to each other."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "uohp13Yp48E5"
   },
   "source": [
    "## <a name='understand'>First we need to understand the text we are using</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dWchdZOb48E5",
    "outputId": "4af8d9dc-c9a3-4623-87b0-aca0d481a736"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "                     1\n",
      "  From fairest creatures we desire increase,\n",
      "  That thereby beauty's rose might never die,\n",
      "  But as the riper should by time decease,\n",
      "  His tender heir might bear his memory:\n",
      "  But thou contracted to thine own bright eyes,\n",
      "  Feed'st thy light's flame with self-substantial fuel,\n",
      "  Making a famine where abundance lies,\n",
      "  Thy self thy foe, to thy sweet self too cruel:\n",
      "  Thou that art now the world's fresh ornament,\n",
      "  And only herald to the gaudy spring,\n",
      "  Within thine own bu\n"
     ]
    }
   ],
   "source": [
    "print(text[:500])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "j6FHHqMr48E6",
    "outputId": "1a64009d-81b5-4b09-a6ff-58a94126c401"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "42"
      ]
     },
     "execution_count": 31,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines = 'From fairest creatures we desire increase,'\n",
    "len(lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JBqwUoNu48E6"
   },
   "source": [
    "As Shakespears writing's fortunatley include rhymes between the lines, we need to pick up at least 3 lines to pick up those patterns.\n",
    "\n",
    "Example: check out the ending words. They do rhyme, such as `increase, decrease` , `die , eyes, lies`\n",
    "\n",
    "There are about 134 characters for 3 lines, one line is about 42 characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "06HKIqKP48E6",
    "outputId": "ee2c21ab-b029-4b93-ed81-5a8c847aedf8"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "134"
      ]
     },
     "execution_count": 32,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines = '''\n",
    " From fairest creatures we desire increase,\n",
    "  That thereby beauty's rose might never die,\n",
    "  But as the riper should by time decease,\n",
    "'''\n",
    "\n",
    "len(lines)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nLA0Zo3X48E7"
   },
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "SSGFsKcb48E7"
   },
   "source": [
    "## <a name='trainingsequences'>Training Sequences</a>\n",
    "\n",
    "The actual text data will be the text sequence shifted one character forward. For example:\n",
    "\n",
    "**Sequence In: \"Hello my nam\"\n",
    "Sequence Out: \"ello my name\"**\n",
    "\n",
    "\n",
    "We can use the `tf.data.Dataset.from_tensor_slices` function to convert a text vector into a stream of character indices."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "_LqYz3M548E7"
   },
   "source": [
    "**We will use about 120 characters for the sequence length.**"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "id": "9RP5gyFE48E8"
   },
   "outputs": [],
   "source": [
    "sequence_length = 120"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "id": "2tiAr7Qq48E9"
   },
   "outputs": [],
   "source": [
    "# calculate how many sequences for the whole text, becuse of 0 indexing we will add 1\n",
    "total_num_sequence = len(text) // (sequence_length+1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "O19TlUy548E9",
    "outputId": "54ec7a81-1bc7-4837-a939-68375c163a25"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "45005"
      ]
     },
     "execution_count": 35,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_num_sequence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "TBrEn2VP48E9"
   },
   "source": [
    "### <a name='createtrainingsequences'>Create Training Sequences</a>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "id": "d0yDuPHW48E-"
   },
   "outputs": [],
   "source": [
    "# Create Training Sequences\n",
    "char_dataset = tf.data.Dataset.from_tensor_slices(encoded_text)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "1NLwdbZQ48E-",
    "outputId": "2842972a-ec88-4491-fb9a-684c33628f76"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "tensorflow.python.data.ops.dataset_ops.TensorSliceDataset"
      ]
     },
     "execution_count": 37,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(char_dataset)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "xQGwwPG448E-",
    "outputId": "b6484b35-26b8-4fff-8396-ab01bc6f40c6"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      " \n",
      "1\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "F\n",
      "r\n",
      "o\n",
      "m\n",
      " \n",
      "f\n",
      "a\n",
      "i\n",
      "r\n",
      "e\n",
      "s\n",
      "t\n",
      " \n",
      "c\n",
      "r\n",
      "e\n",
      "a\n",
      "t\n",
      "u\n",
      "r\n",
      "e\n",
      "s\n",
      " \n",
      "w\n",
      "e\n",
      " \n",
      "d\n",
      "e\n",
      "s\n",
      "i\n",
      "r\n",
      "e\n",
      " \n",
      "i\n",
      "n\n",
      "c\n",
      "r\n",
      "e\n",
      "a\n",
      "s\n",
      "e\n",
      ",\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "T\n",
      "h\n",
      "a\n",
      "t\n",
      " \n",
      "t\n",
      "h\n",
      "e\n",
      "r\n",
      "e\n",
      "b\n",
      "y\n",
      " \n",
      "b\n",
      "e\n",
      "a\n",
      "u\n",
      "t\n",
      "y\n",
      "'\n",
      "s\n",
      " \n",
      "r\n",
      "o\n",
      "s\n",
      "e\n",
      " \n",
      "m\n",
      "i\n",
      "g\n",
      "h\n",
      "t\n",
      " \n",
      "n\n",
      "e\n",
      "v\n",
      "e\n",
      "r\n",
      " \n",
      "d\n",
      "i\n",
      "e\n",
      ",\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "B\n",
      "u\n",
      "t\n",
      " \n",
      "a\n",
      "s\n",
      " \n",
      "t\n",
      "h\n",
      "e\n",
      " \n",
      "r\n",
      "i\n",
      "p\n",
      "e\n",
      "r\n",
      " \n",
      "s\n",
      "h\n",
      "o\n",
      "u\n",
      "l\n",
      "d\n",
      " \n",
      "b\n",
      "y\n",
      " \n",
      "t\n",
      "i\n",
      "m\n",
      "e\n",
      " \n",
      "d\n",
      "e\n",
      "c\n",
      "e\n",
      "a\n",
      "s\n",
      "e\n",
      ",\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "H\n",
      "i\n",
      "s\n",
      " \n",
      "t\n",
      "e\n",
      "n\n",
      "d\n",
      "e\n",
      "r\n",
      " \n",
      "h\n",
      "e\n",
      "i\n",
      "r\n",
      " \n",
      "m\n",
      "i\n",
      "g\n",
      "h\n",
      "t\n",
      " \n",
      "b\n",
      "e\n",
      "a\n",
      "r\n",
      " \n",
      "h\n",
      "i\n",
      "s\n",
      " \n",
      "m\n",
      "e\n",
      "m\n",
      "o\n",
      "r\n",
      "y\n",
      ":\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "B\n",
      "u\n",
      "t\n",
      " \n",
      "t\n",
      "h\n",
      "o\n",
      "u\n",
      " \n",
      "c\n",
      "o\n",
      "n\n",
      "t\n",
      "r\n",
      "a\n",
      "c\n",
      "t\n",
      "e\n",
      "d\n",
      " \n",
      "t\n",
      "o\n",
      " \n",
      "t\n",
      "h\n",
      "i\n",
      "n\n",
      "e\n",
      " \n",
      "o\n",
      "w\n",
      "n\n",
      " \n",
      "b\n",
      "r\n",
      "i\n",
      "g\n",
      "h\n",
      "t\n",
      " \n",
      "e\n",
      "y\n",
      "e\n",
      "s\n",
      ",\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "F\n",
      "e\n",
      "e\n",
      "d\n",
      "'\n",
      "s\n",
      "t\n",
      " \n",
      "t\n",
      "h\n",
      "y\n",
      " \n",
      "l\n",
      "i\n",
      "g\n",
      "h\n",
      "t\n",
      "'\n",
      "s\n",
      " \n",
      "f\n",
      "l\n",
      "a\n",
      "m\n",
      "e\n",
      " \n",
      "w\n",
      "i\n",
      "t\n",
      "h\n",
      " \n",
      "s\n",
      "e\n",
      "l\n",
      "f\n",
      "-\n",
      "s\n",
      "u\n",
      "b\n",
      "s\n",
      "t\n",
      "a\n",
      "n\n",
      "t\n",
      "i\n",
      "a\n",
      "l\n",
      " \n",
      "f\n",
      "u\n",
      "e\n",
      "l\n",
      ",\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "M\n",
      "a\n",
      "k\n",
      "i\n",
      "n\n",
      "g\n",
      " \n",
      "a\n",
      " \n",
      "f\n",
      "a\n",
      "m\n",
      "i\n",
      "n\n",
      "e\n",
      " \n",
      "w\n",
      "h\n",
      "e\n",
      "r\n",
      "e\n",
      " \n",
      "a\n",
      "b\n",
      "u\n",
      "n\n",
      "d\n",
      "a\n",
      "n\n",
      "c\n",
      "e\n",
      " \n",
      "l\n",
      "i\n",
      "e\n",
      "s\n",
      ",\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "T\n",
      "h\n",
      "y\n",
      " \n",
      "s\n",
      "e\n",
      "l\n",
      "f\n",
      " \n",
      "t\n",
      "h\n",
      "y\n",
      " \n",
      "f\n",
      "o\n",
      "e\n",
      ",\n",
      " \n",
      "t\n",
      "o\n",
      " \n",
      "t\n",
      "h\n",
      "y\n",
      " \n",
      "s\n",
      "w\n",
      "e\n",
      "e\n",
      "t\n",
      " \n",
      "s\n",
      "e\n",
      "l\n",
      "f\n",
      " \n",
      "t\n",
      "o\n",
      "o\n",
      " \n",
      "c\n",
      "r\n",
      "u\n",
      "e\n",
      "l\n",
      ":\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "T\n",
      "h\n",
      "o\n",
      "u\n",
      " \n",
      "t\n",
      "h\n",
      "a\n",
      "t\n",
      " \n",
      "a\n",
      "r\n",
      "t\n",
      " \n",
      "n\n",
      "o\n",
      "w\n",
      " \n",
      "t\n",
      "h\n",
      "e\n",
      " \n",
      "w\n",
      "o\n",
      "r\n",
      "l\n",
      "d\n",
      "'\n",
      "s\n",
      " \n",
      "f\n",
      "r\n",
      "e\n",
      "s\n",
      "h\n",
      " \n",
      "o\n",
      "r\n",
      "n\n",
      "a\n",
      "m\n",
      "e\n",
      "n\n",
      "t\n",
      ",\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "A\n",
      "n\n",
      "d\n",
      " \n",
      "o\n",
      "n\n",
      "l\n",
      "y\n",
      " \n",
      "h\n",
      "e\n",
      "r\n",
      "a\n",
      "l\n",
      "d\n",
      " \n",
      "t\n",
      "o\n",
      " \n",
      "t\n",
      "h\n",
      "e\n",
      " \n",
      "g\n",
      "a\n",
      "u\n",
      "d\n",
      "y\n",
      " \n",
      "s\n",
      "p\n",
      "r\n",
      "i\n",
      "n\n",
      "g\n",
      ",\n",
      "\n",
      "\n",
      " \n",
      " \n",
      "W\n",
      "i\n",
      "t\n",
      "h\n",
      "i\n",
      "n\n",
      " \n",
      "t\n",
      "h\n",
      "i\n",
      "n\n",
      "e\n",
      " \n",
      "o\n",
      "w\n",
      "n\n",
      " \n",
      "b\n",
      "u\n"
     ]
    }
   ],
   "source": [
    "# grabbing data from training sequences - demo for first 500 characters\n",
    "for item in char_dataset.take(500):\n",
    "    print(index_to_char[item.numpy()])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "DEcOnKxU48E_"
   },
   "source": [
    "### <a name='createsequencebatches'>Creating Sequence Batches</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "E-m14suo48E_"
   },
   "source": [
    "The **batch** method converts these individual character calls into sequences we can feed in as a batch. We use `seq_len+1` because of zero indexing. Here is what `drop_remainder` means:\n",
    "\n",
    "+ drop_remainder: (Optional.) A `tf.bool` scalar `tf.Tensor`, representing\n",
    "    whether the last batch should be dropped in the case it has fewer than\n",
    "    `batch_size` elements; the default behavior is not to drop the smaller\n",
    "    batch.\n",
    "\n",
    "\n",
    "Why we have remainder in the first place?\n",
    "+ when we divide length of the whole text by 120 (sequence length), there are some remainder left. This is because the length of whole text is not perfectly divisible by 120, which is expected. We will choose to discard those remaining chacters which are not part of any sequence group. This won't have impact on our training as our text is already quite large."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "id": "xlTj7h8_48FA"
   },
   "outputs": [],
   "source": [
    "sequences = char_dataset.batch(sequence_length+1, drop_remainder=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "NGGcTBHO48FB",
    "outputId": "c052c723-e4f9-45a5-b2d2-e6f49be02a19"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BatchDataset shapes: (121,), types: tf.int64>"
      ]
     },
     "execution_count": 40,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sequences"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "nc7GxtN248FB"
   },
   "source": [
    "Now that we have our sequences, we will perform the following steps for each one to create our target text sequences:\n",
    "\n",
    "1. Grab the input text sequence\n",
    "2. Assign the target text sequence as the input text sequence shifted by one step forward\n",
    "3. Group them together as a tuple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "id": "Nf35m5Hi48FB"
   },
   "outputs": [],
   "source": [
    "def create_sequence_targets(seq):\n",
    "    input_txt = seq[:-1] # Hello my nam\n",
    "    target_txt = seq[1:] # ello my name\n",
    "    return (input_txt, target_txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "id": "pqlzGaw448FC"
   },
   "outputs": [],
   "source": [
    "dataset = sequences.map(tf.autograph.experimental.do_not_convert(create_sequence_targets))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "UA1NYXvt48FC"
   },
   "source": [
    "#### Demo of how it looks like for one sequence of first batch\n",
    "+ we can see input_txt is starting with 0 and end with 75.\n",
    "+ for target_txt is shfiting to next 1 character, starting with 1 and ending with 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "skiBZ66b48FC",
    "outputId": "476f4027-8ecb-4131-b0dd-5e7a8a41fbe4"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[ 0  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1 12  0\n",
      "  1  1 31 73 70 68  1 61 56 64 73 60 74 75  1 58 73 60 56 75 76 73 60 74\n",
      "  1 78 60  1 59 60 74 64 73 60  1 64 69 58 73 60 56 74 60  8  0  1  1 45\n",
      " 63 56 75  1 75 63 60 73 60 57 80  1 57 60 56 76 75 80  5 74  1 73 70 74\n",
      " 60  1 68 64 62 63 75  1 69 60 77 60 73  1 59 64 60  8  0  1  1 27 76 75]\n",
      "\n",
      "                     1\n",
      "  From fairest creatures we desire increase,\n",
      "  That thereby beauty's rose might never die,\n",
      "  But\n",
      "\n",
      "\n",
      "[ 1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1  1 12  0  1\n",
      "  1 31 73 70 68  1 61 56 64 73 60 74 75  1 58 73 60 56 75 76 73 60 74  1\n",
      " 78 60  1 59 60 74 64 73 60  1 64 69 58 73 60 56 74 60  8  0  1  1 45 63\n",
      " 56 75  1 75 63 60 73 60 57 80  1 57 60 56 76 75 80  5 74  1 73 70 74 60\n",
      "  1 68 64 62 63 75  1 69 60 77 60 73  1 59 64 60  8  0  1  1 27 76 75  1]\n",
      "                     1\n",
      "  From fairest creatures we desire increase,\n",
      "  That thereby beauty's rose might never die,\n",
      "  But \n"
     ]
    }
   ],
   "source": [
    "for input_txt, target_txt in dataset.take(1): # get the one sequence batch\n",
    "    # for input text\n",
    "    print(input_txt.numpy())\n",
    "    print(\"\".join(index_to_char[input_txt.numpy()]))\n",
    "    \n",
    "    print('\\n')\n",
    "    \n",
    "    # for target text\n",
    "    print(target_txt.numpy())\n",
    "    print(\"\".join(index_to_char[target_txt.numpy()]))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "gmig2ZZt48FD"
   },
   "source": [
    "### <a name='generatingtrainingbatches'>Generating Training Batches</a>\n",
    "\n",
    "Now that we have the actual sequences, we will create the batches, we want to shuffle these sequences into a random order, so the model doesn't overfit to any section of the text, but can instead generate characters given any seed text."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "id": "9y4LGtze48FD"
   },
   "outputs": [],
   "source": [
    "# Batch size\n",
    "batch_size = 128\n",
    "\n",
    "# Buffer size to shuffle the dataset so it doesn't attempt to shuffle the entire sequence in memory. \n",
    "# Instead, it maintains a buffer in which it shuffles elements\n",
    "buffer_size = 1000\n",
    "\n",
    "dataset = dataset.shuffle(buffer_size).batch(batch_size, drop_remainder=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "dJoTu6mC48FD",
    "outputId": "7aef7453-0d4a-414c-fc42-6b19f4b21bdc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<BatchDataset shapes: ((128, 120), (128, 120)), types: (tf.int64, tf.int64)>"
      ]
     },
     "execution_count": 45,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dataset\n",
    "\n",
    "# we can see input sequence and target sequence"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "qUzTRBOE48FE"
   },
   "source": [
    "-------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "QR16m-mm48FE"
   },
   "source": [
    "# <a name='step4'>Step 4: Creating the Model</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "pasRyRAz48FE"
   },
   "source": [
    "We will use an LSTM based model with a few extra features, including an embedding layer to start off with and **two** LSTM layers. We based this model architecture off the [DeepMoji](https://deepmoji.mit.edu/) and the original source code can be found [here](https://github.com/bfelbo/DeepMoji).\n",
    "\n",
    "The embedding layer will serve as the input layer, which essentially creates a lookup table that maps the numbers indices of each character to a vector with \"`embedding dim`\" number of dimensions. As you can imagine, the larger this embedding size, the more complex the training. This is similar to the idea behind word2vec, where words are mapped to some n-dimensional space. Embedding before feeding straight into the LSTM usually leads to more realisitic results."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "id": "ADJOSPlO48FF"
   },
   "outputs": [],
   "source": [
    "# Length of the vocabulary in chars\n",
    "vocab_size = len(vocab)\n",
    "\n",
    "# The embedding dimension (We might want to choose same scale of vocab size)\n",
    "embed_dim = 64\n",
    "\n",
    "# Number of RNN units\n",
    "rnn_neurons = 1026"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "nfy8UkLG48FF",
    "outputId": "7e4e1cb0-a21b-4e0a-ed33-ba0a1919b1a6"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "84"
      ]
     },
     "execution_count": 53,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab_size "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VnPO8wJ448FF"
   },
   "source": [
    "## <a name='setuplossfunction'> Setting up Loss Function</a>\n",
    "\n",
    "For our loss we will use sparse categorical crossentropy, which we can import from Keras. **We will also set this as `logits=True` as our data are `one-hot encoded`**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "id": "CTWgu_4E48FG"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.losses import sparse_categorical_crossentropy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "id": "N5MhEnFO48FG"
   },
   "outputs": [],
   "source": [
    "def sparse_cat_loss(y_true, y_pred):\n",
    "    return sparse_categorical_crossentropy(y_true, y_pred, from_logits=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "Q7TCiQ3W48FG"
   },
   "source": [
    "https://datascience.stackexchange.com/questions/41921/sparse-categorical-crossentropy-vs-categorical-crossentropy-keras-accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "VLy3jU3c48FG"
   },
   "source": [
    "## <a name='modelfunction'> Setting up Model Function</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "HcP9BJfA48FH"
   },
   "source": [
    "Now let's create a function that easily adapts to different variables as shown above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "id": "_Bm12Rij48FH"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import Sequential\n",
    "from tensorflow.keras.layers import Dense, LSTM, Embedding, Dropout, GRU"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "id": "ybDlKJaZ48FH"
   },
   "outputs": [],
   "source": [
    "def create_model(vocab_size, embed_dim, rnn_neurons, batch_size):\n",
    "    model = Sequential()\n",
    "    \n",
    "    model.add(Embedding(vocab_size, embed_dim, batch_input_shape=[batch_size, None]))\n",
    "    \n",
    "    model.add(GRU(rnn_neurons,\n",
    "                          return_sequences=True,\n",
    "                          stateful=True,\n",
    "                          recurrent_initializer='glorot_uniform'))\n",
    "    \n",
    "    model.add(Dense(vocab_size))\n",
    "    \n",
    "    model.compile(optimizer='adam', loss=sparse_cat_loss)\n",
    "    \n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "id": "oz9-dmpH48FH"
   },
   "outputs": [],
   "source": [
    "model = create_model(vocab_size, embed_dim, rnn_neurons, batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "uhMkKmYN48FI",
    "outputId": "3564e9c1-4c2b-4533-fcce-e75a38681bd9"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding (Embedding)        (128, None, 64)           5376      \n",
      "_________________________________________________________________\n",
      "gru (GRU)                    (128, None, 1026)         3361176   \n",
      "_________________________________________________________________\n",
      "dense (Dense)                (128, None, 84)           86268     \n",
      "=================================================================\n",
      "Total params: 3,452,820\n",
      "Trainable params: 3,452,820\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "EkxJheNr48FI"
   },
   "source": [
    "----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "bPok-9c848FI"
   },
   "source": [
    "# <a name='step5'>Step 5: Training the model</a>\n",
    "\n",
    "Let's make sure everything is ok with our model before we spend too much time training! Let's pass in a batch to confirm the model currently predicts random characters without any training.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "3sSk5z0Y48FI",
    "outputId": "9ada24b1-2f3b-4b61-8947-79acd79513d0"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(128, 120, 84)  <=== (batch_size, sequence_length, vocab_size)\n"
     ]
    }
   ],
   "source": [
    "for input_example_batch, target_example_batch in dataset.take(1):\n",
    "\n",
    "  # Predict off some random batch\n",
    "  example_batch_predictions = model(input_example_batch)\n",
    "\n",
    "  # Display the dimensions of the predictions\n",
    "  print(example_batch_predictions.shape, \" <=== (batch_size, sequence_length, vocab_size)\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "KTklWCLG8ztK",
    "outputId": "ec26247c-93dd-4cfd-b838-9c8eef37676d"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "TensorShape([128, 120, 84])"
      ]
     },
     "execution_count": 64,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_batch_predictions.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "rqSfqsDl88rh",
    "outputId": "cf69637b-0c89-4003-98ff-9fe76984d8ed"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(120, 84), dtype=float32, numpy=\n",
       "array([[ 0.00744317, -0.00090096, -0.0094114 , ...,  0.00486787,\n",
       "        -0.00405866, -0.00121064],\n",
       "       [ 0.00855853, -0.00498477, -0.00884936, ...,  0.00592785,\n",
       "        -0.00123757, -0.00812292],\n",
       "       [ 0.00321869, -0.00558345, -0.00278171, ...,  0.00023974,\n",
       "        -0.00402449,  0.00210256],\n",
       "       ...,\n",
       "       [ 0.0038353 ,  0.00550529,  0.00038914, ..., -0.00625554,\n",
       "        -0.00011505, -0.01148627],\n",
       "       [ 0.00331539, -0.00225345,  0.0048323 , ...,  0.00185895,\n",
       "        -0.0020827 , -0.01181064],\n",
       "       [ 0.00576036, -0.00651682, -0.00320568, ...,  0.00465379,\n",
       "         0.00060987, -0.01383383]], dtype=float32)>"
      ]
     },
     "execution_count": 65,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_batch_predictions[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tAtqwfGZ9IId"
   },
   "source": [
    "These are just probabilities for each characters.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "id": "xxWvIXJ19Cbh"
   },
   "outputs": [],
   "source": [
    "sampled_indices = tf.random.categorical(example_batch_predictions[0], num_samples=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "gUfpL1Y29u6f",
    "outputId": "2f15b188-fba2-4f13-9b50-49b271cbbc5c"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tf.Tensor: shape=(120, 1), dtype=int64, numpy=\n",
       "array([[34],\n",
       "       [44],\n",
       "       [39],\n",
       "       [39],\n",
       "       [68],\n",
       "       [ 9],\n",
       "       [38],\n",
       "       [57],\n",
       "       [25],\n",
       "       [49],\n",
       "       [ 5],\n",
       "       [20],\n",
       "       [57],\n",
       "       [69],\n",
       "       [65],\n",
       "       [18],\n",
       "       [27],\n",
       "       [39],\n",
       "       [73],\n",
       "       [ 4],\n",
       "       [12],\n",
       "       [ 9],\n",
       "       [63],\n",
       "       [20],\n",
       "       [21],\n",
       "       [20],\n",
       "       [ 8],\n",
       "       [20],\n",
       "       [77],\n",
       "       [55],\n",
       "       [ 1],\n",
       "       [44],\n",
       "       [45],\n",
       "       [11],\n",
       "       [32],\n",
       "       [73],\n",
       "       [61],\n",
       "       [58],\n",
       "       [42],\n",
       "       [59],\n",
       "       [83],\n",
       "       [39],\n",
       "       [79],\n",
       "       [49],\n",
       "       [ 0],\n",
       "       [66],\n",
       "       [21],\n",
       "       [28],\n",
       "       [51],\n",
       "       [45],\n",
       "       [39],\n",
       "       [77],\n",
       "       [71],\n",
       "       [71],\n",
       "       [57],\n",
       "       [22],\n",
       "       [72],\n",
       "       [51],\n",
       "       [ 9],\n",
       "       [16],\n",
       "       [53],\n",
       "       [59],\n",
       "       [41],\n",
       "       [69],\n",
       "       [72],\n",
       "       [78],\n",
       "       [17],\n",
       "       [53],\n",
       "       [ 5],\n",
       "       [41],\n",
       "       [58],\n",
       "       [57],\n",
       "       [32],\n",
       "       [57],\n",
       "       [15],\n",
       "       [57],\n",
       "       [83],\n",
       "       [ 3],\n",
       "       [43],\n",
       "       [81],\n",
       "       [47],\n",
       "       [65],\n",
       "       [66],\n",
       "       [16],\n",
       "       [74],\n",
       "       [ 1],\n",
       "       [72],\n",
       "       [51],\n",
       "       [52],\n",
       "       [61],\n",
       "       [22],\n",
       "       [37],\n",
       "       [52],\n",
       "       [11],\n",
       "       [29],\n",
       "       [ 3],\n",
       "       [83],\n",
       "       [ 3],\n",
       "       [24],\n",
       "       [18],\n",
       "       [42],\n",
       "       [62],\n",
       "       [ 6],\n",
       "       [28],\n",
       "       [16],\n",
       "       [75],\n",
       "       [53],\n",
       "       [82],\n",
       "       [27],\n",
       "       [71],\n",
       "       [43],\n",
       "       [ 8],\n",
       "       [37],\n",
       "       [ 4],\n",
       "       [23],\n",
       "       [29],\n",
       "       [27],\n",
       "       [34],\n",
       "       [72],\n",
       "       [78]])>"
      ]
     },
     "execution_count": 67,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampled_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "id": "DulCOrnp9wEr"
   },
   "outputs": [],
   "source": [
    "# Reformat to not be a lists of lists\n",
    "sampled_indices = tf.squeeze(sampled_indices,axis=-1).numpy()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "UfD-fJj09xmD",
    "outputId": "b5a72dc8-88ed-42ff-c356-d163850831dc"
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([34, 44, 39, 39, 68,  9, 38, 57, 25, 49,  5, 20, 57, 69, 65, 18, 27,\n",
       "       39, 73,  4, 12,  9, 63, 20, 21, 20,  8, 20, 77, 55,  1, 44, 45, 11,\n",
       "       32, 73, 61, 58, 42, 59, 83, 39, 79, 49,  0, 66, 21, 28, 51, 45, 39,\n",
       "       77, 71, 71, 57, 22, 72, 51,  9, 16, 53, 59, 41, 69, 72, 78, 17, 53,\n",
       "        5, 41, 58, 57, 32, 57, 15, 57, 83,  3, 43, 81, 47, 65, 66, 16, 74,\n",
       "        1, 72, 51, 52, 61, 22, 37, 52, 11, 29,  3, 83,  3, 24, 18, 42, 62,\n",
       "        6, 28, 16, 75, 53, 82, 27, 71, 43,  8, 37,  4, 23, 29, 27, 34, 72,\n",
       "       78])"
      ]
     },
     "execution_count": 69,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sampled_indices"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "HINtlaGV9zJ_",
    "outputId": "48e0e730-3648-4572-d2a5-ef5f613180e3"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Given the input seq: \n",
      "\n",
      "d him from me now.\n",
      "    Yet him for this, my love no whit disdaineth,\n",
      "    Suns of the world may stain, when heaven's sun \n",
      "\n",
      "\n",
      "Next Char Predictions: \n",
      "\n",
      "ISNNm-Mb?X'9bnj7BNr&1-h9:9,9v` ST0GrfcQd}NxX\n",
      "k:CZTNvppb;qZ-5]dPnqw6]'PcbGb4b}\"RzVjk5s qZ[f;L[0D\"}\">7Qg(C5t]|BpR,L&<DBIqw\n"
     ]
    }
   ],
   "source": [
    "print(\"Given the input seq: \\n\")\n",
    "print(\"\".join(index_to_char[input_example_batch[0]]))\n",
    "print('\\n')\n",
    "print(\"Next Char Predictions: \\n\")\n",
    "print(\"\".join(index_to_char[sampled_indices ]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "wcLpsOC997dB"
   },
   "source": [
    "We can see these are just random predictions of characters as we haven't trained our model yet.\n",
    "\n",
    "After confirming the dimensions are working, let's train our network!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {
    "id": "vLCXU1D790s7"
   },
   "outputs": [],
   "source": [
    "epochs = 30"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "zWvr8B6O-A4o",
    "outputId": "edea112a-7ccb-466f-f124-7e50e0db346d"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1/30\n",
      "351/351 [==============================] - 43s 119ms/step - loss: 3.1474\n",
      "Epoch 2/30\n",
      "351/351 [==============================] - 43s 122ms/step - loss: 1.8723\n",
      "Epoch 3/30\n",
      "351/351 [==============================] - 43s 124ms/step - loss: 1.5653\n",
      "Epoch 4/30\n",
      "351/351 [==============================] - 45s 128ms/step - loss: 1.4094\n",
      "Epoch 5/30\n",
      "351/351 [==============================] - 45s 129ms/step - loss: 1.3279\n",
      "Epoch 6/30\n",
      "351/351 [==============================] - 46s 132ms/step - loss: 1.2776\n",
      "Epoch 7/30\n",
      "351/351 [==============================] - 45s 129ms/step - loss: 1.2428\n",
      "Epoch 8/30\n",
      "351/351 [==============================] - 47s 132ms/step - loss: 1.2154\n",
      "Epoch 9/30\n",
      "351/351 [==============================] - 45s 129ms/step - loss: 1.1911\n",
      "Epoch 10/30\n",
      "351/351 [==============================] - 46s 129ms/step - loss: 1.1706\n",
      "Epoch 11/30\n",
      "351/351 [==============================] - 45s 129ms/step - loss: 1.1527\n",
      "Epoch 12/30\n",
      "351/351 [==============================] - 45s 128ms/step - loss: 1.1375\n",
      "Epoch 13/30\n",
      "351/351 [==============================] - 46s 132ms/step - loss: 1.1212\n",
      "Epoch 14/30\n",
      "351/351 [==============================] - 45s 129ms/step - loss: 1.1083\n",
      "Epoch 15/30\n",
      "351/351 [==============================] - 47s 132ms/step - loss: 1.0947\n",
      "Epoch 16/30\n",
      "351/351 [==============================] - 46s 130ms/step - loss: 1.0822\n",
      "Epoch 17/30\n",
      "351/351 [==============================] - 45s 129ms/step - loss: 1.0711\n",
      "Epoch 18/30\n",
      "351/351 [==============================] - 45s 129ms/step - loss: 1.0594\n",
      "Epoch 19/30\n",
      "351/351 [==============================] - 45s 128ms/step - loss: 1.0498\n",
      "Epoch 20/30\n",
      "351/351 [==============================] - 46s 132ms/step - loss: 1.0414\n",
      "Epoch 21/30\n",
      "351/351 [==============================] - 45s 129ms/step - loss: 1.0336\n",
      "Epoch 22/30\n",
      "351/351 [==============================] - 46s 132ms/step - loss: 1.0274\n",
      "Epoch 23/30\n",
      "351/351 [==============================] - 46s 129ms/step - loss: 1.0223\n",
      "Epoch 24/30\n",
      "351/351 [==============================] - 45s 129ms/step - loss: 1.0180\n",
      "Epoch 25/30\n",
      "351/351 [==============================] - 45s 129ms/step - loss: 1.0142\n",
      "Epoch 26/30\n",
      "351/351 [==============================] - 45s 128ms/step - loss: 1.0102\n",
      "Epoch 27/30\n",
      "351/351 [==============================] - 46s 132ms/step - loss: 1.0077\n",
      "Epoch 28/30\n",
      "351/351 [==============================] - 46s 130ms/step - loss: 1.0059\n",
      "Epoch 29/30\n",
      "351/351 [==============================] - 47s 132ms/step - loss: 1.0038\n",
      "Epoch 30/30\n",
      "351/351 [==============================] - 46s 130ms/step - loss: 1.0038\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.keras.callbacks.History at 0x7f0e43f51f98>"
      ]
     },
     "execution_count": 73,
     "metadata": {
      "tags": []
     },
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit(dataset, epochs = epochs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "IzJL9qMR-6iP"
   },
   "source": [
    "--------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "o7TdvAi1-0Xs"
   },
   "source": [
    "# <a name='step6'>Step 6: Generating text</a>\n",
    "\n",
    "Currently our model only expects 128 sequences at a time. We can create a new model that only expects a batch_size=1. We can create a new model with this batch size, then load our saved models weights. Then call .build() on the model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "id": "aw35yLKm-NCE"
   },
   "outputs": [],
   "source": [
    "model.save('shakespeare_gen.h5')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "id": "Ja-NmujU-5WK"
   },
   "outputs": [],
   "source": [
    "from tensorflow.keras.models import load_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {
    "id": "NefwX-9__Tdu"
   },
   "outputs": [],
   "source": [
    "model = create_model(vocab_size, embed_dim, rnn_neurons, batch_size=1)\n",
    "\n",
    "model.load_weights('shakespeare_gen.h5')\n",
    "\n",
    "model.build(tf.TensorShape([1, None]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "D0yBg3iE_dlm",
    "outputId": "755b90e4-e861-4bb3-bbed-15caa30ac75f"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Model: \"sequential_1\"\n",
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "embedding_1 (Embedding)      (1, None, 64)             5376      \n",
      "_________________________________________________________________\n",
      "gru_1 (GRU)                  (1, None, 1026)           3361176   \n",
      "_________________________________________________________________\n",
      "dense_1 (Dense)              (1, None, 84)             86268     \n",
      "=================================================================\n",
      "Total params: 3,452,820\n",
      "Trainable params: 3,452,820\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {
    "id": "4kAnfDGs_doF"
   },
   "outputs": [],
   "source": [
    "def generate_text(model, start_seed,gen_size=100,temp=1.0):\n",
    "  '''\n",
    "  model: Trained Model to Generate Text\n",
    "  start_seed: Intial Seed text in string form\n",
    "  gen_size: Number of characters to generate\n",
    "\n",
    "  Basic idea behind this function is to take in some seed text, format it so\n",
    "  that it is in the correct shape for our network, then loop the sequence as\n",
    "  we keep adding our own predicted characters. Similar to our work in the RNN\n",
    "  time series problems.\n",
    "  '''\n",
    "\n",
    "  # Number of characters to generate\n",
    "  num_generate = gen_size\n",
    "\n",
    "  # Vecotrizing starting seed text\n",
    "  input_eval = [char_to_index[s] for s in start_seed]\n",
    "\n",
    "  # Expand to match batch format shape\n",
    "  input_eval = tf.expand_dims(input_eval, 0)\n",
    "\n",
    "  # Empty list to hold resulting generated text\n",
    "  text_generated = []\n",
    "\n",
    "  # Temperature effects randomness in our resulting text\n",
    "  # The term is derived from entropy/thermodynamics.\n",
    "  # The temperature is used to effect probability of next characters.\n",
    "  # Higher probability == lesss surprising/ more expected\n",
    "  # Lower temperature == more surprising / less expected\n",
    " \n",
    "  temperature = temp\n",
    "\n",
    "  # Here batch size == 1\n",
    "  model.reset_states()\n",
    "\n",
    "  for i in range(num_generate):\n",
    "\n",
    "      # Generate Predictions\n",
    "      predictions = model(input_eval)\n",
    "\n",
    "      # Remove the batch shape dimension\n",
    "      predictions = tf.squeeze(predictions, 0)\n",
    "\n",
    "      # Use a cateogircal disitribution to select the next character\n",
    "      predictions = predictions / temperature\n",
    "      predicted_id = tf.random.categorical(predictions, num_samples=1)[-1,0].numpy()\n",
    "\n",
    "      # Pass the predicted charracter for the next input\n",
    "      input_eval = tf.expand_dims([predicted_id], 0)\n",
    "\n",
    "      # Transform back to character letter\n",
    "      text_generated.append(index_to_char[predicted_id])\n",
    "\n",
    "  return (start_seed + ''.join(text_generated))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "PkJXQZxA_dqm",
    "outputId": "1b81d179-a363-4574-9e2d-d58ed5bf4a4a"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "loves; and anon-\n",
      "    She had no sintine, Greek,\n",
      "    Whose witchcraster hath bounds me blind,\n",
      "  With this, this letter, that they ride so well?\n",
      "  CLOWN. I met a fool and to say so; the first time when she died the\n",
      "    you would tarry under the Prince; that there was to be found, and\n",
      "    both the maid for tailing any left hand,\n",
      "    We'll encompass out of you to my request,\n",
      "  Must be against kind and one that keeps the Tears of my forsune.\n",
      "    Love cannot have you free and hold it\n",
      "    The enemies to me. If we be his\n",
      "    sister- how and ye whose searcher hast thou forg'd i' th' care.\n",
      "  TAMORA. My husar, half heard me; all, true one,\n",
      "    Were nothing benefits   Thou are those credies upon you? You have done\n",
      "    How hus more fresh! my mother, what thou art?     ospers.\n",
      "\n",
      "  Con, I have avoided\n",
      "    As advesture seems down and safe to thee;\n",
      "    So IOr I'll corrupt the dead in this. All's with her;\n",
      "    And I envy arives you crowned in holy execution;\n",
      "    two and twenty shogors and the reing A perse,\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(generate_text(model,\"love\",gen_size=1000))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yRd1e21m_dtJ"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [],
   "name": "00_Generating_Text_with_RNNs.ipynb",
   "provenance": []
  },
  "kernelspec": {
   "display_name": "venv-datascience",
   "language": "python",
   "name": "venv-datascience"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}