23.NLP irish song generation.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Gaelic ( irish ) Song Generation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "from tensorflow.keras.preprocessing.text import Tokenizer\n",
    "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
    "from tensorflow.keras.layers import LSTM, GRU, Embedding, Bidirectional, Dense\n",
    "from tensorflow.keras.models import Model\n",
    "from tensorflow.keras.models import Sequential"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "# the irish song is separated by \\n for the new line\n",
    "data = \"In the town of Athy one Jeremy Lanigan \\n Battered away til he hadnt a pound. \\nHis father died and made him a man again \\n Left him a farm and ten acres of ground. \\nHe gave a grand party for friends and relations \\nWho didnt forget him when come to the wall, \\nAnd if youll but listen Ill make your eyes glisten \\nOf the rows and the ructions of Lanigans Ball. \\nMyself to be sure got free invitation, \\nFor all the nice girls and boys I might ask, \\nAnd just in a minute both friends and relations \\nWere dancing round merry as bees round a cask. \\nJudy ODaly, that nice little milliner, \\nShe tipped me a wink for to give her a call, \\nAnd I soon arrived with Peggy McGilligan \\nJust in time for Lanigans Ball. \\nThere were lashings of punch and wine for the ladies, \\nPotatoes and cakes; there was bacon and tea, \\nThere were the Nolans, Dolans, OGradys \\nCourting the girls and dancing away. \\nSongs they went round as plenty as water, \\nThe harp that once sounded in Taras old hall,\\nSweet Nelly Gray and The Rat Catchers Daughter,\\nAll singing together at Lanigans Ball. \\nThey were doing all kinds of nonsensical polkas \\nAll round the room in a whirligig. \\nJulia and I, we banished their nonsense \\nAnd tipped them the twist of a reel and a jig. \\nAch mavrone, how the girls got all mad at me \\nDanced til youd think the ceiling would fall. \\nFor I spent three weeks at Brooks Academy \\nLearning new steps for Lanigans Ball. \\nThree long weeks I spent up in Dublin, \\nThree long weeks to learn nothing at all,\\n Three long weeks I spent up in Dublin, \\nLearning new steps for Lanigans Ball. \\nShe stepped out and I stepped in again, \\nI stepped out and she stepped in again, \\nShe stepped out and I stepped in again, \\nLearning new steps for Lanigans Ball. \\nBoys were all merry and the girls they were hearty \\nAnd danced all around in couples and groups, \\nTil an accident happened, young Terrance McCarthy \\nPut his right leg through miss Finnertys hoops. \\nPoor creature fainted and cried Meelia murther, \\nCalled for her brothers and gathered them all. \\nCarmody swore that hed go no further \\nTil he had satisfaction at Lanigans Ball. \\nIn the midst of the row miss Kerrigan fainted, \\nHer cheeks at the same time as red as a rose. \\nSome of the lads declared she was painted, \\nShe took a small drop too much, I suppose. \\nHer sweetheart, Ned Morgan, so powerful and able, \\nWhen he saw his fair colleen stretched out by the wall, \\nTore the left leg from under the table \\nAnd smashed all the Chaneys at Lanigans Ball. \\nBoys, oh boys, twas then there were runctions. \\nMyself got a lick from big Phelim McHugh. \\nI soon replied to his introduction \\nAnd kicked up a terrible hullabaloo. \\nOld Casey, the piper, was near being strangled. \\nThey squeezed up his pipes, bellows, chanters and all. \\nThe girls, in their ribbons, they got all entangled \\nAnd that put an end to Lanigans Ball.\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['in the town of athy one jeremy lanigan ',\n",
       " ' battered away til he hadnt a pound. ',\n",
       " 'his father died and made him a man again ',\n",
       " ' left him a farm and ten acres of ground. ',\n",
       " 'he gave a grand party for friends and relations ',\n",
       " 'who didnt forget him when come to the wall, ',\n",
       " 'and if youll but listen ill make your eyes glisten ',\n",
       " 'of the rows and the ructions of lanigans ball. ',\n",
       " 'myself to be sure got free invitation, ',\n",
       " 'for all the nice girls and boys i might ask, ',\n",
       " 'and just in a minute both friends and relations ',\n",
       " 'were dancing round merry as bees round a cask. ',\n",
       " 'judy odaly, that nice little milliner, ',\n",
       " 'she tipped me a wink for to give her a call, ',\n",
       " 'and i soon arrived with peggy mcgilligan ',\n",
       " 'just in time for lanigans ball. ',\n",
       " 'there were lashings of punch and wine for the ladies, ',\n",
       " 'potatoes and cakes; there was bacon and tea, ',\n",
       " 'there were the nolans, dolans, ogradys ',\n",
       " 'courting the girls and dancing away. ',\n",
       " 'songs they went round as plenty as water, ',\n",
       " 'the harp that once sounded in taras old hall,',\n",
       " 'sweet nelly gray and the rat catchers daughter,',\n",
       " 'all singing together at lanigans ball. ',\n",
       " 'they were doing all kinds of nonsensical polkas ',\n",
       " 'all round the room in a whirligig. ',\n",
       " 'julia and i, we banished their nonsense ',\n",
       " 'and tipped them the twist of a reel and a jig. ',\n",
       " 'ach mavrone, how the girls got all mad at me ',\n",
       " 'danced til youd think the ceiling would fall. ',\n",
       " 'for i spent three weeks at brooks academy ',\n",
       " 'learning new steps for lanigans ball. ',\n",
       " 'three long weeks i spent up in dublin, ',\n",
       " 'three long weeks to learn nothing at all,',\n",
       " ' three long weeks i spent up in dublin, ',\n",
       " 'learning new steps for lanigans ball. ',\n",
       " 'she stepped out and i stepped in again, ',\n",
       " 'i stepped out and she stepped in again, ',\n",
       " 'she stepped out and i stepped in again, ',\n",
       " 'learning new steps for lanigans ball. ',\n",
       " 'boys were all merry and the girls they were hearty ',\n",
       " 'and danced all around in couples and groups, ',\n",
       " 'til an accident happened, young terrance mccarthy ',\n",
       " 'put his right leg through miss finnertys hoops. ',\n",
       " 'poor creature fainted and cried meelia murther, ',\n",
       " 'called for her brothers and gathered them all. ',\n",
       " 'carmody swore that hed go no further ',\n",
       " 'til he had satisfaction at lanigans ball. ',\n",
       " 'in the midst of the row miss kerrigan fainted, ',\n",
       " 'her cheeks at the same time as red as a rose. ',\n",
       " 'some of the lads declared she was painted, ',\n",
       " 'she took a small drop too much, i suppose. ',\n",
       " 'her sweetheart, ned morgan, so powerful and able, ',\n",
       " 'when he saw his fair colleen stretched out by the wall, ',\n",
       " 'tore the left leg from under the table ',\n",
       " 'and smashed all the chaneys at lanigans ball. ',\n",
       " 'boys, oh boys, twas then there were runctions. ',\n",
       " 'myself got a lick from big phelim mchugh. ',\n",
       " 'i soon replied to his introduction ',\n",
       " 'and kicked up a terrible hullabaloo. ',\n",
       " 'old casey, the piper, was near being strangled. ',\n",
       " 'they squeezed up his pipes, bellows, chanters and all. ',\n",
       " 'the girls, in their ribbons, they got all entangled ',\n",
       " 'and that put an end to lanigans ball.']"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = data.lower().split('\\n')\n",
    "data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'and': 1, 'the': 2, 'a': 3, 'in': 4, 'all': 5, 'i': 6, 'for': 7, 'of': 8, 'lanigans': 9, 'ball': 10, 'were': 11, 'at': 12, 'to': 13, 'she': 14, 'stepped': 15, 'his': 16, 'girls': 17, 'as': 18, 'they': 19, 'til': 20, 'he': 21, 'again': 22, 'got': 23, 'boys': 24, 'round': 25, 'that': 26, 'her': 27, 'there': 28, 'three': 29, 'weeks': 30, 'up': 31, 'out': 32, 'him': 33, 'was': 34, 'spent': 35, 'learning': 36, 'new': 37, 'steps': 38, 'long': 39, 'away': 40, 'left': 41, 'friends': 42, 'relations': 43, 'when': 44, 'wall': 45, 'myself': 46, 'nice': 47, 'just': 48, 'dancing': 49, 'merry': 50, 'tipped': 51, 'me': 52, 'soon': 53, 'time': 54, 'old': 55, 'their': 56, 'them': 57, 'danced': 58, 'dublin': 59, 'an': 60, 'put': 61, 'leg': 62, 'miss': 63, 'fainted': 64, 'from': 65, 'town': 66, 'athy': 67, 'one': 68, 'jeremy': 69, 'lanigan': 70, 'battered': 71, 'hadnt': 72, 'pound': 73, 'father': 74, 'died': 75, 'made': 76, 'man': 77, 'farm': 78, 'ten': 79, 'acres': 80, 'ground': 81, 'gave': 82, 'grand': 83, 'party': 84, 'who': 85, 'didnt': 86, 'forget': 87, 'come': 88, 'if': 89, 'youll': 90, 'but': 91, 'listen': 92, 'ill': 93, 'make': 94, 'your': 95, 'eyes': 96, 'glisten': 97, 'rows': 98, 'ructions': 99, 'be': 100, 'sure': 101, 'free': 102, 'invitation': 103, 'might': 104, 'ask': 105, 'minute': 106, 'both': 107, 'bees': 108, 'cask': 109, 'judy': 110, 'odaly': 111, 'little': 112, 'milliner': 113, 'wink': 114, 'give': 115, 'call': 116, 'arrived': 117, 'with': 118, 'peggy': 119, 'mcgilligan': 120, 'lashings': 121, 'punch': 122, 'wine': 123, 'ladies': 124, 'potatoes': 125, 'cakes': 126, 'bacon': 127, 'tea': 128, 'nolans': 129, 'dolans': 130, 'ogradys': 131, 'courting': 132, 'songs': 133, 'went': 134, 'plenty': 135, 'water': 136, 'harp': 137, 'once': 138, 'sounded': 139, 'taras': 140, 'hall': 141, 'sweet': 142, 'nelly': 143, 'gray': 144, 'rat': 145, 'catchers': 146, 'daughter': 147, 'singing': 148, 'together': 149, 'doing': 150, 'kinds': 151, 'nonsensical': 152, 'polkas': 153, 'room': 154, 'whirligig': 155, 'julia': 156, 'we': 157, 'banished': 158, 'nonsense': 159, 'twist': 160, 'reel': 161, 'jig': 162, 'ach': 163, 'mavrone': 164, 'how': 165, 'mad': 166, 'youd': 167, 'think': 168, 'ceiling': 169, 'would': 170, 'fall': 171, 'brooks': 172, 'academy': 173, 'learn': 174, 'nothing': 175, 'hearty': 176, 'around': 177, 'couples': 178, 'groups': 179, 'accident': 180, 'happened': 181, 'young': 182, 'terrance': 183, 'mccarthy': 184, 'right': 185, 'through': 186, 'finnertys': 187, 'hoops': 188, 'poor': 189, 'creature': 190, 'cried': 191, 'meelia': 192, 'murther': 193, 'called': 194, 'brothers': 195, 'gathered': 196, 'carmody': 197, 'swore': 198, 'hed': 199, 'go': 200, 'no': 201, 'further': 202, 'had': 203, 'satisfaction': 204, 'midst': 205, 'row': 206, 'kerrigan': 207, 'cheeks': 208, 'same': 209, 'red': 210, 'rose': 211, 'some': 212, 'lads': 213, 'declared': 214, 'painted': 215, 'took': 216, 'small': 217, 'drop': 218, 'too': 219, 'much': 220, 'suppose': 221, 'sweetheart': 222, 'ned': 223, 'morgan': 224, 'so': 225, 'powerful': 226, 'able': 227, 'saw': 228, 'fair': 229, 'colleen': 230, 'stretched': 231, 'by': 232, 'tore': 233, 'under': 234, 'table': 235, 'smashed': 236, 'chaneys': 237, 'oh': 238, 'twas': 239, 'then': 240, 'runctions': 241, 'lick': 242, 'big': 243, 'phelim': 244, 'mchugh': 245, 'replied': 246, 'introduction': 247, 'kicked': 248, 'terrible': 249, 'hullabaloo': 250, 'casey': 251, 'piper': 252, 'near': 253, 'being': 254, 'strangled': 255, 'squeezed': 256, 'pipes': 257, 'bellows': 258, 'chanters': 259, 'ribbons': 260, 'entangled': 261, 'end': 262}\n",
      "\n",
      "\n",
      "263\n"
     ]
    }
   ],
   "source": [
    "# just tokenizing the words and generating the sequences\n",
    "tokenizer = Tokenizer()\n",
    "tokenizer.fit_on_texts(data)\n",
    "\n",
    "\n",
    "total_words = len(tokenizer.word_index)+1\n",
    "print(tokenizer.word_index)\n",
    "print('\\n')\n",
    "print(total_words)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'in the town of athy one jeremy lanigan '"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[4, 2, 66, 8, 67, 68, 69, 70]"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tokenizer.texts_to_sequences([data[0]])[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_sequences = []\n",
    "for line in data :\n",
    "    seq = tokenizer.texts_to_sequences([line])[0]\n",
    "    for i in range(1,len(seq)):\n",
    "        n_gram_sequence = seq[:i+1]\n",
    "        input_sequences.append(n_gram_sequence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[[4, 2],\n",
       " [4, 2, 66],\n",
       " [4, 2, 66, 8],\n",
       " [4, 2, 66, 8, 67],\n",
       " [4, 2, 66, 8, 67, 68],\n",
       " [4, 2, 66, 8, 67, 68, 69],\n",
       " [4, 2, 66, 8, 67, 68, 69, 70],\n",
       " [71, 40],\n",
       " [71, 40, 20],\n",
       " [71, 40, 20, 21],\n",
       " [71, 40, 20, 21, 72],\n",
       " [71, 40, 20, 21, 72, 3],\n",
       " [71, 40, 20, 21, 72, 3, 73],\n",
       " [16, 74],\n",
       " [16, 74, 75],\n",
       " [16, 74, 75, 1],\n",
       " [16, 74, 75, 1, 76],\n",
       " [16, 74, 75, 1, 76, 33],\n",
       " [16, 74, 75, 1, 76, 33, 3],\n",
       " [16, 74, 75, 1, 76, 33, 3, 77],\n",
       " [16, 74, 75, 1, 76, 33, 3, 77, 22],\n",
       " [41, 33],\n",
       " [41, 33, 3],\n",
       " [41, 33, 3, 78],\n",
       " [41, 33, 3, 78, 1],\n",
       " [41, 33, 3, 78, 1, 79],\n",
       " [41, 33, 3, 78, 1, 79, 80],\n",
       " [41, 33, 3, 78, 1, 79, 80, 8],\n",
       " [41, 33, 3, 78, 1, 79, 80, 8, 81],\n",
       " [21, 82],\n",
       " [21, 82, 3],\n",
       " [21, 82, 3, 83],\n",
       " [21, 82, 3, 83, 84],\n",
       " [21, 82, 3, 83, 84, 7],\n",
       " [21, 82, 3, 83, 84, 7, 42],\n",
       " [21, 82, 3, 83, 84, 7, 42, 1],\n",
       " [21, 82, 3, 83, 84, 7, 42, 1, 43],\n",
       " [85, 86],\n",
       " [85, 86, 87],\n",
       " [85, 86, 87, 33],\n",
       " [85, 86, 87, 33, 44],\n",
       " [85, 86, 87, 33, 44, 88],\n",
       " [85, 86, 87, 33, 44, 88, 13],\n",
       " [85, 86, 87, 33, 44, 88, 13, 2],\n",
       " [85, 86, 87, 33, 44, 88, 13, 2, 45],\n",
       " [1, 89],\n",
       " [1, 89, 90],\n",
       " [1, 89, 90, 91],\n",
       " [1, 89, 90, 91, 92],\n",
       " [1, 89, 90, 91, 92, 93],\n",
       " [1, 89, 90, 91, 92, 93, 94],\n",
       " [1, 89, 90, 91, 92, 93, 94, 95],\n",
       " [1, 89, 90, 91, 92, 93, 94, 95, 96],\n",
       " [1, 89, 90, 91, 92, 93, 94, 95, 96, 97],\n",
       " [8, 2],\n",
       " [8, 2, 98],\n",
       " [8, 2, 98, 1],\n",
       " [8, 2, 98, 1, 2],\n",
       " [8, 2, 98, 1, 2, 99],\n",
       " [8, 2, 98, 1, 2, 99, 8],\n",
       " [8, 2, 98, 1, 2, 99, 8, 9],\n",
       " [8, 2, 98, 1, 2, 99, 8, 9, 10],\n",
       " [46, 13],\n",
       " [46, 13, 100],\n",
       " [46, 13, 100, 101],\n",
       " [46, 13, 100, 101, 23],\n",
       " [46, 13, 100, 101, 23, 102],\n",
       " [46, 13, 100, 101, 23, 102, 103],\n",
       " [7, 5],\n",
       " [7, 5, 2],\n",
       " [7, 5, 2, 47],\n",
       " [7, 5, 2, 47, 17],\n",
       " [7, 5, 2, 47, 17, 1],\n",
       " [7, 5, 2, 47, 17, 1, 24],\n",
       " [7, 5, 2, 47, 17, 1, 24, 6],\n",
       " [7, 5, 2, 47, 17, 1, 24, 6, 104],\n",
       " [7, 5, 2, 47, 17, 1, 24, 6, 104, 105],\n",
       " [1, 48],\n",
       " [1, 48, 4],\n",
       " [1, 48, 4, 3],\n",
       " [1, 48, 4, 3, 106],\n",
       " [1, 48, 4, 3, 106, 107],\n",
       " [1, 48, 4, 3, 106, 107, 42],\n",
       " [1, 48, 4, 3, 106, 107, 42, 1],\n",
       " [1, 48, 4, 3, 106, 107, 42, 1, 43],\n",
       " [11, 49],\n",
       " [11, 49, 25],\n",
       " [11, 49, 25, 50],\n",
       " [11, 49, 25, 50, 18],\n",
       " [11, 49, 25, 50, 18, 108],\n",
       " [11, 49, 25, 50, 18, 108, 25],\n",
       " [11, 49, 25, 50, 18, 108, 25, 3],\n",
       " [11, 49, 25, 50, 18, 108, 25, 3, 109],\n",
       " [110, 111],\n",
       " [110, 111, 26],\n",
       " [110, 111, 26, 47],\n",
       " [110, 111, 26, 47, 112],\n",
       " [110, 111, 26, 47, 112, 113],\n",
       " [14, 51],\n",
       " [14, 51, 52],\n",
       " [14, 51, 52, 3],\n",
       " [14, 51, 52, 3, 114],\n",
       " [14, 51, 52, 3, 114, 7],\n",
       " [14, 51, 52, 3, 114, 7, 13],\n",
       " [14, 51, 52, 3, 114, 7, 13, 115],\n",
       " [14, 51, 52, 3, 114, 7, 13, 115, 27],\n",
       " [14, 51, 52, 3, 114, 7, 13, 115, 27, 3],\n",
       " [14, 51, 52, 3, 114, 7, 13, 115, 27, 3, 116],\n",
       " [1, 6],\n",
       " [1, 6, 53],\n",
       " [1, 6, 53, 117],\n",
       " [1, 6, 53, 117, 118],\n",
       " [1, 6, 53, 117, 118, 119],\n",
       " [1, 6, 53, 117, 118, 119, 120],\n",
       " [48, 4],\n",
       " [48, 4, 54],\n",
       " [48, 4, 54, 7],\n",
       " [48, 4, 54, 7, 9],\n",
       " [48, 4, 54, 7, 9, 10],\n",
       " [28, 11],\n",
       " [28, 11, 121],\n",
       " [28, 11, 121, 8],\n",
       " [28, 11, 121, 8, 122],\n",
       " [28, 11, 121, 8, 122, 1],\n",
       " [28, 11, 121, 8, 122, 1, 123],\n",
       " [28, 11, 121, 8, 122, 1, 123, 7],\n",
       " [28, 11, 121, 8, 122, 1, 123, 7, 2],\n",
       " [28, 11, 121, 8, 122, 1, 123, 7, 2, 124],\n",
       " [125, 1],\n",
       " [125, 1, 126],\n",
       " [125, 1, 126, 28],\n",
       " [125, 1, 126, 28, 34],\n",
       " [125, 1, 126, 28, 34, 127],\n",
       " [125, 1, 126, 28, 34, 127, 1],\n",
       " [125, 1, 126, 28, 34, 127, 1, 128],\n",
       " [28, 11],\n",
       " [28, 11, 2],\n",
       " [28, 11, 2, 129],\n",
       " [28, 11, 2, 129, 130],\n",
       " [28, 11, 2, 129, 130, 131],\n",
       " [132, 2],\n",
       " [132, 2, 17],\n",
       " [132, 2, 17, 1],\n",
       " [132, 2, 17, 1, 49],\n",
       " [132, 2, 17, 1, 49, 40],\n",
       " [133, 19],\n",
       " [133, 19, 134],\n",
       " [133, 19, 134, 25],\n",
       " [133, 19, 134, 25, 18],\n",
       " [133, 19, 134, 25, 18, 135],\n",
       " [133, 19, 134, 25, 18, 135, 18],\n",
       " [133, 19, 134, 25, 18, 135, 18, 136],\n",
       " [2, 137],\n",
       " [2, 137, 26],\n",
       " [2, 137, 26, 138],\n",
       " [2, 137, 26, 138, 139],\n",
       " [2, 137, 26, 138, 139, 4],\n",
       " [2, 137, 26, 138, 139, 4, 140],\n",
       " [2, 137, 26, 138, 139, 4, 140, 55],\n",
       " [2, 137, 26, 138, 139, 4, 140, 55, 141],\n",
       " [142, 143],\n",
       " [142, 143, 144],\n",
       " [142, 143, 144, 1],\n",
       " [142, 143, 144, 1, 2],\n",
       " [142, 143, 144, 1, 2, 145],\n",
       " [142, 143, 144, 1, 2, 145, 146],\n",
       " [142, 143, 144, 1, 2, 145, 146, 147],\n",
       " [5, 148],\n",
       " [5, 148, 149],\n",
       " [5, 148, 149, 12],\n",
       " [5, 148, 149, 12, 9],\n",
       " [5, 148, 149, 12, 9, 10],\n",
       " [19, 11],\n",
       " [19, 11, 150],\n",
       " [19, 11, 150, 5],\n",
       " [19, 11, 150, 5, 151],\n",
       " [19, 11, 150, 5, 151, 8],\n",
       " [19, 11, 150, 5, 151, 8, 152],\n",
       " [19, 11, 150, 5, 151, 8, 152, 153],\n",
       " [5, 25],\n",
       " [5, 25, 2],\n",
       " [5, 25, 2, 154],\n",
       " [5, 25, 2, 154, 4],\n",
       " [5, 25, 2, 154, 4, 3],\n",
       " [5, 25, 2, 154, 4, 3, 155],\n",
       " [156, 1],\n",
       " [156, 1, 6],\n",
       " [156, 1, 6, 157],\n",
       " [156, 1, 6, 157, 158],\n",
       " [156, 1, 6, 157, 158, 56],\n",
       " [156, 1, 6, 157, 158, 56, 159],\n",
       " [1, 51],\n",
       " [1, 51, 57],\n",
       " [1, 51, 57, 2],\n",
       " [1, 51, 57, 2, 160],\n",
       " [1, 51, 57, 2, 160, 8],\n",
       " [1, 51, 57, 2, 160, 8, 3],\n",
       " [1, 51, 57, 2, 160, 8, 3, 161],\n",
       " [1, 51, 57, 2, 160, 8, 3, 161, 1],\n",
       " [1, 51, 57, 2, 160, 8, 3, 161, 1, 3],\n",
       " [1, 51, 57, 2, 160, 8, 3, 161, 1, 3, 162],\n",
       " [163, 164],\n",
       " [163, 164, 165],\n",
       " [163, 164, 165, 2],\n",
       " [163, 164, 165, 2, 17],\n",
       " [163, 164, 165, 2, 17, 23],\n",
       " [163, 164, 165, 2, 17, 23, 5],\n",
       " [163, 164, 165, 2, 17, 23, 5, 166],\n",
       " [163, 164, 165, 2, 17, 23, 5, 166, 12],\n",
       " [163, 164, 165, 2, 17, 23, 5, 166, 12, 52],\n",
       " [58, 20],\n",
       " [58, 20, 167],\n",
       " [58, 20, 167, 168],\n",
       " [58, 20, 167, 168, 2],\n",
       " [58, 20, 167, 168, 2, 169],\n",
       " [58, 20, 167, 168, 2, 169, 170],\n",
       " [58, 20, 167, 168, 2, 169, 170, 171],\n",
       " [7, 6],\n",
       " [7, 6, 35],\n",
       " [7, 6, 35, 29],\n",
       " [7, 6, 35, 29, 30],\n",
       " [7, 6, 35, 29, 30, 12],\n",
       " [7, 6, 35, 29, 30, 12, 172],\n",
       " [7, 6, 35, 29, 30, 12, 172, 173],\n",
       " [36, 37],\n",
       " [36, 37, 38],\n",
       " [36, 37, 38, 7],\n",
       " [36, 37, 38, 7, 9],\n",
       " [36, 37, 38, 7, 9, 10],\n",
       " [29, 39],\n",
       " [29, 39, 30],\n",
       " [29, 39, 30, 6],\n",
       " [29, 39, 30, 6, 35],\n",
       " [29, 39, 30, 6, 35, 31],\n",
       " [29, 39, 30, 6, 35, 31, 4],\n",
       " [29, 39, 30, 6, 35, 31, 4, 59],\n",
       " [29, 39],\n",
       " [29, 39, 30],\n",
       " [29, 39, 30, 13],\n",
       " [29, 39, 30, 13, 174],\n",
       " [29, 39, 30, 13, 174, 175],\n",
       " [29, 39, 30, 13, 174, 175, 12],\n",
       " [29, 39, 30, 13, 174, 175, 12, 5],\n",
       " [29, 39],\n",
       " [29, 39, 30],\n",
       " [29, 39, 30, 6],\n",
       " [29, 39, 30, 6, 35],\n",
       " [29, 39, 30, 6, 35, 31],\n",
       " [29, 39, 30, 6, 35, 31, 4],\n",
       " [29, 39, 30, 6, 35, 31, 4, 59],\n",
       " [36, 37],\n",
       " [36, 37, 38],\n",
       " [36, 37, 38, 7],\n",
       " [36, 37, 38, 7, 9],\n",
       " [36, 37, 38, 7, 9, 10],\n",
       " [14, 15],\n",
       " [14, 15, 32],\n",
       " [14, 15, 32, 1],\n",
       " [14, 15, 32, 1, 6],\n",
       " [14, 15, 32, 1, 6, 15],\n",
       " [14, 15, 32, 1, 6, 15, 4],\n",
       " [14, 15, 32, 1, 6, 15, 4, 22],\n",
       " [6, 15],\n",
       " [6, 15, 32],\n",
       " [6, 15, 32, 1],\n",
       " [6, 15, 32, 1, 14],\n",
       " [6, 15, 32, 1, 14, 15],\n",
       " [6, 15, 32, 1, 14, 15, 4],\n",
       " [6, 15, 32, 1, 14, 15, 4, 22],\n",
       " [14, 15],\n",
       " [14, 15, 32],\n",
       " [14, 15, 32, 1],\n",
       " [14, 15, 32, 1, 6],\n",
       " [14, 15, 32, 1, 6, 15],\n",
       " [14, 15, 32, 1, 6, 15, 4],\n",
       " [14, 15, 32, 1, 6, 15, 4, 22],\n",
       " [36, 37],\n",
       " [36, 37, 38],\n",
       " [36, 37, 38, 7],\n",
       " [36, 37, 38, 7, 9],\n",
       " [36, 37, 38, 7, 9, 10],\n",
       " [24, 11],\n",
       " [24, 11, 5],\n",
       " [24, 11, 5, 50],\n",
       " [24, 11, 5, 50, 1],\n",
       " [24, 11, 5, 50, 1, 2],\n",
       " [24, 11, 5, 50, 1, 2, 17],\n",
       " [24, 11, 5, 50, 1, 2, 17, 19],\n",
       " [24, 11, 5, 50, 1, 2, 17, 19, 11],\n",
       " [24, 11, 5, 50, 1, 2, 17, 19, 11, 176],\n",
       " [1, 58],\n",
       " [1, 58, 5],\n",
       " [1, 58, 5, 177],\n",
       " [1, 58, 5, 177, 4],\n",
       " [1, 58, 5, 177, 4, 178],\n",
       " [1, 58, 5, 177, 4, 178, 1],\n",
       " [1, 58, 5, 177, 4, 178, 1, 179],\n",
       " [20, 60],\n",
       " [20, 60, 180],\n",
       " [20, 60, 180, 181],\n",
       " [20, 60, 180, 181, 182],\n",
       " [20, 60, 180, 181, 182, 183],\n",
       " [20, 60, 180, 181, 182, 183, 184],\n",
       " [61, 16],\n",
       " [61, 16, 185],\n",
       " [61, 16, 185, 62],\n",
       " [61, 16, 185, 62, 186],\n",
       " [61, 16, 185, 62, 186, 63],\n",
       " [61, 16, 185, 62, 186, 63, 187],\n",
       " [61, 16, 185, 62, 186, 63, 187, 188],\n",
       " [189, 190],\n",
       " [189, 190, 64],\n",
       " [189, 190, 64, 1],\n",
       " [189, 190, 64, 1, 191],\n",
       " [189, 190, 64, 1, 191, 192],\n",
       " [189, 190, 64, 1, 191, 192, 193],\n",
       " [194, 7],\n",
       " [194, 7, 27],\n",
       " [194, 7, 27, 195],\n",
       " [194, 7, 27, 195, 1],\n",
       " [194, 7, 27, 195, 1, 196],\n",
       " [194, 7, 27, 195, 1, 196, 57],\n",
       " [194, 7, 27, 195, 1, 196, 57, 5],\n",
       " [197, 198],\n",
       " [197, 198, 26],\n",
       " [197, 198, 26, 199],\n",
       " [197, 198, 26, 199, 200],\n",
       " [197, 198, 26, 199, 200, 201],\n",
       " [197, 198, 26, 199, 200, 201, 202],\n",
       " [20, 21],\n",
       " [20, 21, 203],\n",
       " [20, 21, 203, 204],\n",
       " [20, 21, 203, 204, 12],\n",
       " [20, 21, 203, 204, 12, 9],\n",
       " [20, 21, 203, 204, 12, 9, 10],\n",
       " [4, 2],\n",
       " [4, 2, 205],\n",
       " [4, 2, 205, 8],\n",
       " [4, 2, 205, 8, 2],\n",
       " [4, 2, 205, 8, 2, 206],\n",
       " [4, 2, 205, 8, 2, 206, 63],\n",
       " [4, 2, 205, 8, 2, 206, 63, 207],\n",
       " [4, 2, 205, 8, 2, 206, 63, 207, 64],\n",
       " [27, 208],\n",
       " [27, 208, 12],\n",
       " [27, 208, 12, 2],\n",
       " [27, 208, 12, 2, 209],\n",
       " [27, 208, 12, 2, 209, 54],\n",
       " [27, 208, 12, 2, 209, 54, 18],\n",
       " [27, 208, 12, 2, 209, 54, 18, 210],\n",
       " [27, 208, 12, 2, 209, 54, 18, 210, 18],\n",
       " [27, 208, 12, 2, 209, 54, 18, 210, 18, 3],\n",
       " [27, 208, 12, 2, 209, 54, 18, 210, 18, 3, 211],\n",
       " [212, 8],\n",
       " [212, 8, 2],\n",
       " [212, 8, 2, 213],\n",
       " [212, 8, 2, 213, 214],\n",
       " [212, 8, 2, 213, 214, 14],\n",
       " [212, 8, 2, 213, 214, 14, 34],\n",
       " [212, 8, 2, 213, 214, 14, 34, 215],\n",
       " [14, 216],\n",
       " [14, 216, 3],\n",
       " [14, 216, 3, 217],\n",
       " [14, 216, 3, 217, 218],\n",
       " [14, 216, 3, 217, 218, 219],\n",
       " [14, 216, 3, 217, 218, 219, 220],\n",
       " [14, 216, 3, 217, 218, 219, 220, 6],\n",
       " [14, 216, 3, 217, 218, 219, 220, 6, 221],\n",
       " [27, 222],\n",
       " [27, 222, 223],\n",
       " [27, 222, 223, 224],\n",
       " [27, 222, 223, 224, 225],\n",
       " [27, 222, 223, 224, 225, 226],\n",
       " [27, 222, 223, 224, 225, 226, 1],\n",
       " [27, 222, 223, 224, 225, 226, 1, 227],\n",
       " [44, 21],\n",
       " [44, 21, 228],\n",
       " [44, 21, 228, 16],\n",
       " [44, 21, 228, 16, 229],\n",
       " [44, 21, 228, 16, 229, 230],\n",
       " [44, 21, 228, 16, 229, 230, 231],\n",
       " [44, 21, 228, 16, 229, 230, 231, 32],\n",
       " [44, 21, 228, 16, 229, 230, 231, 32, 232],\n",
       " [44, 21, 228, 16, 229, 230, 231, 32, 232, 2],\n",
       " [44, 21, 228, 16, 229, 230, 231, 32, 232, 2, 45],\n",
       " [233, 2],\n",
       " [233, 2, 41],\n",
       " [233, 2, 41, 62],\n",
       " [233, 2, 41, 62, 65],\n",
       " [233, 2, 41, 62, 65, 234],\n",
       " [233, 2, 41, 62, 65, 234, 2],\n",
       " [233, 2, 41, 62, 65, 234, 2, 235],\n",
       " [1, 236],\n",
       " [1, 236, 5],\n",
       " [1, 236, 5, 2],\n",
       " [1, 236, 5, 2, 237],\n",
       " [1, 236, 5, 2, 237, 12],\n",
       " [1, 236, 5, 2, 237, 12, 9],\n",
       " [1, 236, 5, 2, 237, 12, 9, 10],\n",
       " [24, 238],\n",
       " [24, 238, 24],\n",
       " [24, 238, 24, 239],\n",
       " [24, 238, 24, 239, 240],\n",
       " [24, 238, 24, 239, 240, 28],\n",
       " [24, 238, 24, 239, 240, 28, 11],\n",
       " [24, 238, 24, 239, 240, 28, 11, 241],\n",
       " [46, 23],\n",
       " [46, 23, 3],\n",
       " [46, 23, 3, 242],\n",
       " [46, 23, 3, 242, 65],\n",
       " [46, 23, 3, 242, 65, 243],\n",
       " [46, 23, 3, 242, 65, 243, 244],\n",
       " [46, 23, 3, 242, 65, 243, 244, 245],\n",
       " [6, 53],\n",
       " [6, 53, 246],\n",
       " [6, 53, 246, 13],\n",
       " [6, 53, 246, 13, 16],\n",
       " [6, 53, 246, 13, 16, 247],\n",
       " [1, 248],\n",
       " [1, 248, 31],\n",
       " [1, 248, 31, 3],\n",
       " [1, 248, 31, 3, 249],\n",
       " [1, 248, 31, 3, 249, 250],\n",
       " [55, 251],\n",
       " [55, 251, 2],\n",
       " [55, 251, 2, 252],\n",
       " [55, 251, 2, 252, 34],\n",
       " [55, 251, 2, 252, 34, 253],\n",
       " [55, 251, 2, 252, 34, 253, 254],\n",
       " [55, 251, 2, 252, 34, 253, 254, 255],\n",
       " [19, 256],\n",
       " [19, 256, 31],\n",
       " [19, 256, 31, 16],\n",
       " [19, 256, 31, 16, 257],\n",
       " [19, 256, 31, 16, 257, 258],\n",
       " [19, 256, 31, 16, 257, 258, 259],\n",
       " [19, 256, 31, 16, 257, 258, 259, 1],\n",
       " [19, 256, 31, 16, 257, 258, 259, 1, 5],\n",
       " [2, 17],\n",
       " [2, 17, 4],\n",
       " [2, 17, 4, 56],\n",
       " [2, 17, 4, 56, 260],\n",
       " [2, 17, 4, 56, 260, 19],\n",
       " [2, 17, 4, 56, 260, 19, 23],\n",
       " [2, 17, 4, 56, 260, 19, 23, 5],\n",
       " [2, 17, 4, 56, 260, 19, 23, 5, 261],\n",
       " [1, 26],\n",
       " [1, 26, 61],\n",
       " [1, 26, 61, 60],\n",
       " [1, 26, 61, 60, 262],\n",
       " [1, 26, 61, 60, 262, 13],\n",
       " [1, 26, 61, 60, 262, 13, 9],\n",
       " [1, 26, 61, 60, 262, 13, 9, 10]]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "11"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "max_sequence_len = max([len(i) for i in input_sequences])\n",
    "max_sequence_len"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "list"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(input_sequences)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0,   0,   0, ...,   0,   4,   2],\n",
       "       [  0,   0,   0, ...,   4,   2,  66],\n",
       "       [  0,   0,   0, ...,   2,  66,   8],\n",
       "       ...,\n",
       "       [  0,   0,   0, ...,  60, 262,  13],\n",
       "       [  0,   0,   0, ..., 262,  13,   9],\n",
       "       [  0,   0,   0, ...,  13,   9,  10]])"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_sequences = np.array(pad_sequences(input_sequences,maxlen=11,padding='pre'))\n",
    "input_sequences"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[  0,   0,   0, ...,   0,   0,   4],\n",
       "       [  0,   0,   0, ...,   0,   4,   2],\n",
       "       [  0,   0,   0, ...,   4,   2,  66],\n",
       "       ...,\n",
       "       [  0,   0,   0, ...,  61,  60, 262],\n",
       "       [  0,   0,   0, ...,  60, 262,  13],\n",
       "       [  0,   0,   0, ..., 262,  13,   9]])"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xs , labels = input_sequences[:,:-1], input_sequences[:,-1]\n",
    "xs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([  2,  66,   8,  67,  68,  69,  70,  40,  20,  21,  72,   3,  73,\n",
       "        74,  75,   1,  76,  33,   3,  77,  22,  33,   3,  78,   1,  79,\n",
       "        80,   8,  81,  82,   3,  83,  84,   7,  42,   1,  43,  86,  87,\n",
       "        33,  44,  88,  13,   2,  45,  89,  90,  91,  92,  93,  94,  95,\n",
       "        96,  97,   2,  98,   1,   2,  99,   8,   9,  10,  13, 100, 101,\n",
       "        23, 102, 103,   5,   2,  47,  17,   1,  24,   6, 104, 105,  48,\n",
       "         4,   3, 106, 107,  42,   1,  43,  49,  25,  50,  18, 108,  25,\n",
       "         3, 109, 111,  26,  47, 112, 113,  51,  52,   3, 114,   7,  13,\n",
       "       115,  27,   3, 116,   6,  53, 117, 118, 119, 120,   4,  54,   7,\n",
       "         9,  10,  11, 121,   8, 122,   1, 123,   7,   2, 124,   1, 126,\n",
       "        28,  34, 127,   1, 128,  11,   2, 129, 130, 131,   2,  17,   1,\n",
       "        49,  40,  19, 134,  25,  18, 135,  18, 136, 137,  26, 138, 139,\n",
       "         4, 140,  55, 141, 143, 144,   1,   2, 145, 146, 147, 148, 149,\n",
       "        12,   9,  10,  11, 150,   5, 151,   8, 152, 153,  25,   2, 154,\n",
       "         4,   3, 155,   1,   6, 157, 158,  56, 159,  51,  57,   2, 160,\n",
       "         8,   3, 161,   1,   3, 162, 164, 165,   2,  17,  23,   5, 166,\n",
       "        12,  52,  20, 167, 168,   2, 169, 170, 171,   6,  35,  29,  30,\n",
       "        12, 172, 173,  37,  38,   7,   9,  10,  39,  30,   6,  35,  31,\n",
       "         4,  59,  39,  30,  13, 174, 175,  12,   5,  39,  30,   6,  35,\n",
       "        31,   4,  59,  37,  38,   7,   9,  10,  15,  32,   1,   6,  15,\n",
       "         4,  22,  15,  32,   1,  14,  15,   4,  22,  15,  32,   1,   6,\n",
       "        15,   4,  22,  37,  38,   7,   9,  10,  11,   5,  50,   1,   2,\n",
       "        17,  19,  11, 176,  58,   5, 177,   4, 178,   1, 179,  60, 180,\n",
       "       181, 182, 183, 184,  16, 185,  62, 186,  63, 187, 188, 190,  64,\n",
       "         1, 191, 192, 193,   7,  27, 195,   1, 196,  57,   5, 198,  26,\n",
       "       199, 200, 201, 202,  21, 203, 204,  12,   9,  10,   2, 205,   8,\n",
       "         2, 206,  63, 207,  64, 208,  12,   2, 209,  54,  18, 210,  18,\n",
       "         3, 211,   8,   2, 213, 214,  14,  34, 215, 216,   3, 217, 218,\n",
       "       219, 220,   6, 221, 222, 223, 224, 225, 226,   1, 227,  21, 228,\n",
       "        16, 229, 230, 231,  32, 232,   2,  45,   2,  41,  62,  65, 234,\n",
       "         2, 235, 236,   5,   2, 237,  12,   9,  10, 238,  24, 239, 240,\n",
       "        28,  11, 241,  23,   3, 242,  65, 243, 244, 245,  53, 246,  13,\n",
       "        16, 247, 248,  31,   3, 249, 250, 251,   2, 252,  34, 253, 254,\n",
       "       255, 256,  31,  16, 257, 258, 259,   1,   5,  17,   4,  56, 260,\n",
       "        19,  23,   5, 261,  26,  61,  60, 262,  13,   9,  10])"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "labels"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "453"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "262"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(tokenizer.word_index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0., 0., 1., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.,\n",
       "       0., 0., 0., 0., 0., 0., 0., 0.], dtype=float32)"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ys = tf.keras.utils.to_categorical(labels, num_classes=len(tokenizer.word_index)+1) # doing one hot encoding\n",
    "ys[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(453, 263)"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ys.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "4\n"
     ]
    }
   ],
   "source": [
    "print(tokenizer.word_index['in'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "10"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(xs[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(453, 10)"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xs.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "263"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "total_words"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 453 samples\n",
      "Epoch 1/500\n",
      "453/453 [==============================] - 23s 51ms/sample - loss: 5.5675 - accuracy: 0.0132\n",
      "Epoch 2/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 5.5380 - accuracy: 0.0574\n",
      "Epoch 3/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 5.4598 - accuracy: 0.0574\n",
      "Epoch 4/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 5.2611 - accuracy: 0.0530\n",
      "Epoch 5/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 5.1200 - accuracy: 0.0508\n",
      "Epoch 6/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 5.0611 - accuracy: 0.0508\n",
      "Epoch 7/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 5.0262 - accuracy: 0.0508\n",
      "Epoch 8/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 4.9933 - accuracy: 0.0508\n",
      "Epoch 9/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 4.9649 - accuracy: 0.0552\n",
      "Epoch 10/500\n",
      "453/453 [==============================] - 0s 991us/sample - loss: 4.9292 - accuracy: 0.0552\n",
      "Epoch 11/500\n",
      "453/453 [==============================] - 0s 566us/sample - loss: 4.8939 - accuracy: 0.0706\n",
      "Epoch 12/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 4.8560 - accuracy: 0.0706\n",
      "Epoch 13/500\n",
      "453/453 [==============================] - 0s 649us/sample - loss: 4.8072 - accuracy: 0.0861\n",
      "Epoch 14/500\n",
      "453/453 [==============================] - 0s 724us/sample - loss: 4.7613 - accuracy: 0.0927\n",
      "Epoch 15/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 4.7113 - accuracy: 0.0971\n",
      "Epoch 16/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 4.6561 - accuracy: 0.0993\n",
      "Epoch 17/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 4.5994 - accuracy: 0.0971\n",
      "Epoch 18/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 4.5505 - accuracy: 0.0861\n",
      "Epoch 19/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 4.5020 - accuracy: 0.0971\n",
      "Epoch 20/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 4.4457 - accuracy: 0.1126\n",
      "Epoch 21/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 4.4081 - accuracy: 0.1192\n",
      "Epoch 22/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 4.3656 - accuracy: 0.1170\n",
      "Epoch 23/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 4.3131 - accuracy: 0.1302\n",
      "Epoch 24/500\n",
      "453/453 [==============================] - 0s 972us/sample - loss: 4.2729 - accuracy: 0.1391\n",
      "Epoch 25/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 4.2407 - accuracy: 0.1501\n",
      "Epoch 26/500\n",
      "453/453 [==============================] - 0s 926us/sample - loss: 4.2053 - accuracy: 0.1369\n",
      "Epoch 27/500\n",
      "453/453 [==============================] - 0s 878us/sample - loss: 4.1802 - accuracy: 0.1589\n",
      "Epoch 28/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 4.1263 - accuracy: 0.1545\n",
      "Epoch 29/500\n",
      "453/453 [==============================] - 0s 976us/sample - loss: 4.0788 - accuracy: 0.1722\n",
      "Epoch 30/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 4.0319 - accuracy: 0.1722\n",
      "Epoch 31/500\n",
      "453/453 [==============================] - 0s 893us/sample - loss: 3.9969 - accuracy: 0.1722\n",
      "Epoch 32/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.9608 - accuracy: 0.1744\n",
      "Epoch 33/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.9192 - accuracy: 0.1788\n",
      "Epoch 34/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.8799 - accuracy: 0.1788\n",
      "Epoch 35/500\n",
      "453/453 [==============================] - 0s 739us/sample - loss: 3.8355 - accuracy: 0.1876\n",
      "Epoch 36/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 3.7996 - accuracy: 0.1898\n",
      "Epoch 37/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 3.7620 - accuracy: 0.1965\n",
      "Epoch 38/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.7170 - accuracy: 0.2119\n",
      "Epoch 39/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.6760 - accuracy: 0.2296\n",
      "Epoch 40/500\n",
      "453/453 [==============================] - 0s 897us/sample - loss: 3.6375 - accuracy: 0.2296\n",
      "Epoch 41/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.5988 - accuracy: 0.2230\n",
      "Epoch 42/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.5628 - accuracy: 0.2318\n",
      "Epoch 43/500\n",
      "453/453 [==============================] - 0s 876us/sample - loss: 3.5399 - accuracy: 0.2450\n",
      "Epoch 44/500\n",
      "453/453 [==============================] - 0s 982us/sample - loss: 3.4978 - accuracy: 0.2494\n",
      "Epoch 45/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.4631 - accuracy: 0.2406\n",
      "Epoch 46/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.4306 - accuracy: 0.2693\n",
      "Epoch 47/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.4095 - accuracy: 0.2671\n",
      "Epoch 48/500\n",
      "453/453 [==============================] - 0s 990us/sample - loss: 3.3702 - accuracy: 0.2848\n",
      "Epoch 49/500\n",
      "453/453 [==============================] - 0s 986us/sample - loss: 3.3287 - accuracy: 0.3002\n",
      "Epoch 50/500\n",
      "453/453 [==============================] - 0s 974us/sample - loss: 3.2973 - accuracy: 0.3091\n",
      "Epoch 51/500\n",
      "453/453 [==============================] - 0s 548us/sample - loss: 3.2654 - accuracy: 0.3091\n",
      "Epoch 52/500\n",
      "453/453 [==============================] - 0s 651us/sample - loss: 3.2421 - accuracy: 0.3333\n",
      "Epoch 53/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.2046 - accuracy: 0.3289\n",
      "Epoch 54/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 3.1734 - accuracy: 0.3400\n",
      "Epoch 55/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.1327 - accuracy: 0.3598\n",
      "Epoch 56/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.0921 - accuracy: 0.3576\n",
      "Epoch 57/500\n",
      "453/453 [==============================] - 0s 872us/sample - loss: 3.0694 - accuracy: 0.3687\n",
      "Epoch 58/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 3.0333 - accuracy: 0.3819\n",
      "Epoch 59/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 3.0055 - accuracy: 0.3819\n",
      "Epoch 60/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.9781 - accuracy: 0.4128\n",
      "Epoch 61/500\n",
      "453/453 [==============================] - 0s 879us/sample - loss: 2.9358 - accuracy: 0.4106\n",
      "Epoch 62/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.9117 - accuracy: 0.4172\n",
      "Epoch 63/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.8998 - accuracy: 0.4260\n",
      "Epoch 64/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.8751 - accuracy: 0.4084\n",
      "Epoch 65/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.8320 - accuracy: 0.4062\n",
      "Epoch 66/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.8006 - accuracy: 0.4238\n",
      "Epoch 67/500\n",
      "453/453 [==============================] - 0s 839us/sample - loss: 2.7690 - accuracy: 0.4393\n",
      "Epoch 68/500\n",
      "453/453 [==============================] - 0s 618us/sample - loss: 2.7392 - accuracy: 0.4636\n",
      "Epoch 69/500\n",
      "453/453 [==============================] - 0s 630us/sample - loss: 2.7105 - accuracy: 0.4768\n",
      "Epoch 70/500\n",
      "453/453 [==============================] - 0s 879us/sample - loss: 2.6796 - accuracy: 0.4945\n",
      "Epoch 71/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.6551 - accuracy: 0.4901\n",
      "Epoch 72/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.6293 - accuracy: 0.4812\n",
      "Epoch 73/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.6087 - accuracy: 0.4945\n",
      "Epoch 74/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.5846 - accuracy: 0.5099\n",
      "Epoch 75/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.5598 - accuracy: 0.5276\n",
      "Epoch 76/500\n",
      "453/453 [==============================] - 0s 869us/sample - loss: 2.5419 - accuracy: 0.5232\n",
      "Epoch 77/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.5317 - accuracy: 0.5099\n",
      "Epoch 78/500\n",
      "453/453 [==============================] - 0s 795us/sample - loss: 2.4965 - accuracy: 0.5254\n",
      "Epoch 79/500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "453/453 [==============================] - 1s 1ms/sample - loss: 2.4723 - accuracy: 0.5342\n",
      "Epoch 80/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 2.4453 - accuracy: 0.5342\n",
      "Epoch 81/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.4474 - accuracy: 0.5298\n",
      "Epoch 82/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.4226 - accuracy: 0.5408\n",
      "Epoch 83/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.3919 - accuracy: 0.54300s - loss: 2.3528 - accura\n",
      "Epoch 84/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.3678 - accuracy: 0.5607\n",
      "Epoch 85/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.3885 - accuracy: 0.5430\n",
      "Epoch 86/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.3520 - accuracy: 0.5541\n",
      "Epoch 87/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.3101 - accuracy: 0.5784\n",
      "Epoch 88/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.2754 - accuracy: 0.5850\n",
      "Epoch 89/500\n",
      "453/453 [==============================] - ETA: 0s - loss: 2.2404 - accuracy: 0.58 - 0s 1ms/sample - loss: 2.2427 - accuracy: 0.5938\n",
      "Epoch 90/500\n",
      "453/453 [==============================] - 0s 766us/sample - loss: 2.2166 - accuracy: 0.6026\n",
      "Epoch 91/500\n",
      "453/453 [==============================] - 0s 656us/sample - loss: 2.1879 - accuracy: 0.6159\n",
      "Epoch 92/500\n",
      "453/453 [==============================] - 0s 729us/sample - loss: 2.1647 - accuracy: 0.6093\n",
      "Epoch 93/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.1389 - accuracy: 0.6159\n",
      "Epoch 94/500\n",
      "453/453 [==============================] - 0s 796us/sample - loss: 2.1129 - accuracy: 0.6380\n",
      "Epoch 95/500\n",
      "453/453 [==============================] - 0s 648us/sample - loss: 2.0919 - accuracy: 0.6402\n",
      "Epoch 96/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 2.0696 - accuracy: 0.6291\n",
      "Epoch 97/500\n",
      "453/453 [==============================] - 0s 889us/sample - loss: 2.0427 - accuracy: 0.6358\n",
      "Epoch 98/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.0243 - accuracy: 0.65120s - loss: 1.8572 - accu\n",
      "Epoch 99/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 2.0042 - accuracy: 0.65120s - loss: 1.9794 - accuracy: 0.\n",
      "Epoch 100/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.9852 - accuracy: 0.6512\n",
      "Epoch 101/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.9617 - accuracy: 0.6600\n",
      "Epoch 102/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.9433 - accuracy: 0.6667\n",
      "Epoch 103/500\n",
      "453/453 [==============================] - 0s 989us/sample - loss: 1.9245 - accuracy: 0.6821\n",
      "Epoch 104/500\n",
      "453/453 [==============================] - 0s 997us/sample - loss: 1.9051 - accuracy: 0.6887 - loss: 1.8949 - accuracy: 0.69\n",
      "Epoch 105/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.8871 - accuracy: 0.6932\n",
      "Epoch 106/500\n",
      "453/453 [==============================] - 0s 800us/sample - loss: 1.8667 - accuracy: 0.6954\n",
      "Epoch 107/500\n",
      "453/453 [==============================] - 0s 648us/sample - loss: 1.8457 - accuracy: 0.6954\n",
      "Epoch 108/500\n",
      "453/453 [==============================] - 0s 631us/sample - loss: 1.8354 - accuracy: 0.6954\n",
      "Epoch 109/500\n",
      "453/453 [==============================] - 0s 764us/sample - loss: 1.8219 - accuracy: 0.7064\n",
      "Epoch 110/500\n",
      "453/453 [==============================] - 0s 648us/sample - loss: 1.8066 - accuracy: 0.6998\n",
      "Epoch 111/500\n",
      "453/453 [==============================] - 0s 625us/sample - loss: 1.7910 - accuracy: 0.7219\n",
      "Epoch 112/500\n",
      "453/453 [==============================] - 0s 631us/sample - loss: 1.7662 - accuracy: 0.7130\n",
      "Epoch 113/500\n",
      "453/453 [==============================] - 0s 622us/sample - loss: 1.7564 - accuracy: 0.7174\n",
      "Epoch 114/500\n",
      "453/453 [==============================] - 0s 601us/sample - loss: 1.7390 - accuracy: 0.7263\n",
      "Epoch 115/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 1.7075 - accuracy: 0.7285\n",
      "Epoch 116/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 1.6998 - accuracy: 0.7285\n",
      "Epoch 117/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 1.6734 - accuracy: 0.7241\n",
      "Epoch 118/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 1.6671 - accuracy: 0.7285\n",
      "Epoch 119/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 1.6628 - accuracy: 0.7307\n",
      "Epoch 120/500\n",
      "453/453 [==============================] - 0s 647us/sample - loss: 1.6273 - accuracy: 0.7506\n",
      "Epoch 121/500\n",
      "453/453 [==============================] - 0s 788us/sample - loss: 1.6011 - accuracy: 0.7594\n",
      "Epoch 122/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.5804 - accuracy: 0.76600s - loss: 1.5829 - accuracy: \n",
      "Epoch 123/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.5606 - accuracy: 0.7704\n",
      "Epoch 124/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.5479 - accuracy: 0.7682\n",
      "Epoch 125/500\n",
      "453/453 [==============================] - 0s 850us/sample - loss: 1.5309 - accuracy: 0.7770\n",
      "Epoch 126/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.5108 - accuracy: 0.7770\n",
      "Epoch 127/500\n",
      "453/453 [==============================] - ETA: 0s - loss: 1.4720 - accuracy: 0.78 - 0s 1ms/sample - loss: 1.4949 - accuracy: 0.7792\n",
      "Epoch 128/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.4834 - accuracy: 0.7837\n",
      "Epoch 129/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.4640 - accuracy: 0.7815\n",
      "Epoch 130/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.4492 - accuracy: 0.7881\n",
      "Epoch 131/500\n",
      "453/453 [==============================] - 0s 858us/sample - loss: 1.4396 - accuracy: 0.7815\n",
      "Epoch 132/500\n",
      "453/453 [==============================] - 0s 633us/sample - loss: 1.4192 - accuracy: 0.7969\n",
      "Epoch 133/500\n",
      "453/453 [==============================] - 0s 647us/sample - loss: 1.4048 - accuracy: 0.8035\n",
      "Epoch 134/500\n",
      "453/453 [==============================] - 0s 625us/sample - loss: 1.3920 - accuracy: 0.7991\n",
      "Epoch 135/500\n",
      "453/453 [==============================] - 0s 768us/sample - loss: 1.3773 - accuracy: 0.8057\n",
      "Epoch 136/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.3832 - accuracy: 0.7969\n",
      "Epoch 137/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.3724 - accuracy: 0.8013\n",
      "Epoch 138/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.3464 - accuracy: 0.8057\n",
      "Epoch 139/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.3306 - accuracy: 0.8124\n",
      "Epoch 140/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 1.3134 - accuracy: 0.8146\n",
      "Epoch 141/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 1.3037 - accuracy: 0.82340s - loss: 1.3918 - accura\n",
      "Epoch 142/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.3098 - accuracy: 0.8146\n",
      "Epoch 143/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.3228 - accuracy: 0.8035\n",
      "Epoch 144/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 1.3099 - accuracy: 0.8079\n",
      "Epoch 145/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 1.3020 - accuracy: 0.8079\n",
      "Epoch 146/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.2906 - accuracy: 0.79910s - loss: 1.2498 - accura\n",
      "Epoch 147/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.2537 - accuracy: 0.8256\n",
      "Epoch 148/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 1.2429 - accuracy: 0.8322\n",
      "Epoch 149/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.2156 - accuracy: 0.8344\n",
      "Epoch 150/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.1945 - accuracy: 0.8344\n",
      "Epoch 151/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.1785 - accuracy: 0.8433\n",
      "Epoch 152/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.1641 - accuracy: 0.8366\n",
      "Epoch 153/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.1519 - accuracy: 0.8477\n",
      "Epoch 154/500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.1383 - accuracy: 0.8499\n",
      "Epoch 155/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.1257 - accuracy: 0.8565\n",
      "Epoch 156/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 1.1137 - accuracy: 0.8565\n",
      "Epoch 157/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.1021 - accuracy: 0.8653\n",
      "Epoch 158/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 1.0906 - accuracy: 0.8609\n",
      "Epoch 159/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.0781 - accuracy: 0.8698\n",
      "Epoch 160/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 1.0671 - accuracy: 0.8698\n",
      "Epoch 161/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 1.0554 - accuracy: 0.8631\n",
      "Epoch 162/500\n",
      "453/453 [==============================] - 0s 592us/sample - loss: 1.0434 - accuracy: 0.8675\n",
      "Epoch 163/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 1.0340 - accuracy: 0.8720\n",
      "Epoch 164/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 1.0232 - accuracy: 0.8764\n",
      "Epoch 165/500\n",
      "453/453 [==============================] - 0s 619us/sample - loss: 1.0123 - accuracy: 0.8786\n",
      "Epoch 166/500\n",
      "453/453 [==============================] - 0s 641us/sample - loss: 0.9998 - accuracy: 0.8786\n",
      "Epoch 167/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.9897 - accuracy: 0.8808\n",
      "Epoch 168/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.9790 - accuracy: 0.8830\n",
      "Epoch 169/500\n",
      "453/453 [==============================] - 0s 615us/sample - loss: 0.9717 - accuracy: 0.8808\n",
      "Epoch 170/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.9622 - accuracy: 0.8830\n",
      "Epoch 171/500\n",
      "453/453 [==============================] - 0s 648us/sample - loss: 0.9542 - accuracy: 0.8874\n",
      "Epoch 172/500\n",
      "453/453 [==============================] - 0s 633us/sample - loss: 0.9440 - accuracy: 0.8985\n",
      "Epoch 173/500\n",
      "453/453 [==============================] - 0s 622us/sample - loss: 0.9346 - accuracy: 0.8940\n",
      "Epoch 174/500\n",
      "453/453 [==============================] - 0s 625us/sample - loss: 0.9276 - accuracy: 0.8962\n",
      "Epoch 175/500\n",
      "453/453 [==============================] - 0s 609us/sample - loss: 0.9141 - accuracy: 0.9029\n",
      "Epoch 176/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.9278 - accuracy: 0.8918\n",
      "Epoch 177/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.9219 - accuracy: 0.8852\n",
      "Epoch 178/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.9221 - accuracy: 0.8852\n",
      "Epoch 179/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.9410 - accuracy: 0.8830\n",
      "Epoch 180/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.9206 - accuracy: 0.8918\n",
      "Epoch 181/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.9935 - accuracy: 0.8565\n",
      "Epoch 182/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 0.9647 - accuracy: 0.8609\n",
      "Epoch 183/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.9502 - accuracy: 0.8698\n",
      "Epoch 184/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.9128 - accuracy: 0.8918\n",
      "Epoch 185/500\n",
      "453/453 [==============================] - 0s 648us/sample - loss: 0.8997 - accuracy: 0.8852\n",
      "Epoch 186/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.8660 - accuracy: 0.8918\n",
      "Epoch 187/500\n",
      "453/453 [==============================] - 0s 669us/sample - loss: 0.8525 - accuracy: 0.8918\n",
      "Epoch 188/500\n",
      "453/453 [==============================] - 0s 651us/sample - loss: 0.8338 - accuracy: 0.8985\n",
      "Epoch 189/500\n",
      "453/453 [==============================] - 0s 669us/sample - loss: 0.8168 - accuracy: 0.9073\n",
      "Epoch 190/500\n",
      "453/453 [==============================] - 0s 656us/sample - loss: 0.8044 - accuracy: 0.9073\n",
      "Epoch 191/500\n",
      "453/453 [==============================] - 0s 981us/sample - loss: 0.7948 - accuracy: 0.9139\n",
      "Epoch 192/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.7881 - accuracy: 0.90730s - loss: 0.7140 - accura\n",
      "Epoch 193/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.7757 - accuracy: 0.9117\n",
      "Epoch 194/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 0.7688 - accuracy: 0.9073\n",
      "Epoch 195/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.7590 - accuracy: 0.9073\n",
      "Epoch 196/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.7546 - accuracy: 0.9117\n",
      "Epoch 197/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.7436 - accuracy: 0.9183\n",
      "Epoch 198/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.7671 - accuracy: 0.9095\n",
      "Epoch 199/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.7474 - accuracy: 0.9117\n",
      "Epoch 200/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.7452 - accuracy: 0.9117\n",
      "Epoch 201/500\n",
      "453/453 [==============================] - 0s 766us/sample - loss: 0.7283 - accuracy: 0.9249\n",
      "Epoch 202/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 0.7101 - accuracy: 0.9205\n",
      "Epoch 203/500\n",
      "453/453 [==============================] - 1s 1ms/sample - loss: 0.7034 - accuracy: 0.9205\n",
      "Epoch 204/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6942 - accuracy: 0.9205\n",
      "Epoch 205/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6888 - accuracy: 0.9205\n",
      "Epoch 206/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6801 - accuracy: 0.9205\n",
      "Epoch 207/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6731 - accuracy: 0.9227\n",
      "Epoch 208/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6692 - accuracy: 0.9227\n",
      "Epoch 209/500\n",
      "453/453 [==============================] - 0s 992us/sample - loss: 0.6640 - accuracy: 0.9249\n",
      "Epoch 210/500\n",
      "453/453 [==============================] - 0s 925us/sample - loss: 0.6519 - accuracy: 0.9249\n",
      "Epoch 211/500\n",
      "453/453 [==============================] - 0s 545us/sample - loss: 0.6451 - accuracy: 0.9272\n",
      "Epoch 212/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6391 - accuracy: 0.9249\n",
      "Epoch 213/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6320 - accuracy: 0.9272\n",
      "Epoch 214/500\n",
      "453/453 [==============================] - 0s 905us/sample - loss: 0.6261 - accuracy: 0.9249\n",
      "Epoch 215/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6207 - accuracy: 0.9272\n",
      "Epoch 216/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6145 - accuracy: 0.9272\n",
      "Epoch 217/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6087 - accuracy: 0.9272\n",
      "Epoch 218/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.6030 - accuracy: 0.9294\n",
      "Epoch 219/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.5971 - accuracy: 0.9338\n",
      "Epoch 220/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.5909 - accuracy: 0.93380s - loss: 0.5831 - accuracy: 0.\n",
      "Epoch 221/500\n",
      "453/453 [==============================] - 0s 581us/sample - loss: 0.5866 - accuracy: 0.9338\n",
      "Epoch 222/500\n",
      "453/453 [==============================] - 0s 632us/sample - loss: 0.5823 - accuracy: 0.9404\n",
      "Epoch 223/500\n",
      "453/453 [==============================] - 0s 702us/sample - loss: 0.5763 - accuracy: 0.9404 - loss: 0.5324 - accuracy\n",
      "Epoch 224/500\n",
      "453/453 [==============================] - 0s 820us/sample - loss: 0.5691 - accuracy: 0.9360\n",
      "Epoch 225/500\n",
      "453/453 [==============================] - 0s 818us/sample - loss: 0.5648 - accuracy: 0.9404\n",
      "Epoch 226/500\n",
      "453/453 [==============================] - 0s 822us/sample - loss: 0.5589 - accuracy: 0.9360\n",
      "Epoch 227/500\n",
      "453/453 [==============================] - 0s 831us/sample - loss: 0.5534 - accuracy: 0.9382\n",
      "Epoch 228/500\n",
      "453/453 [==============================] - 0s 813us/sample - loss: 0.5480 - accuracy: 0.9404\n",
      "Epoch 229/500\n",
      "453/453 [==============================] - 0s 832us/sample - loss: 0.5441 - accuracy: 0.9404\n",
      "Epoch 230/500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "453/453 [==============================] - 0s 820us/sample - loss: 0.5394 - accuracy: 0.9338\n",
      "Epoch 231/500\n",
      "453/453 [==============================] - 0s 812us/sample - loss: 0.5339 - accuracy: 0.9316\n",
      "Epoch 232/500\n",
      "453/453 [==============================] - 0s 819us/sample - loss: 0.5301 - accuracy: 0.9360\n",
      "Epoch 233/500\n",
      "453/453 [==============================] - 0s 725us/sample - loss: 0.5253 - accuracy: 0.9404\n",
      "Epoch 234/500\n",
      "453/453 [==============================] - 0s 648us/sample - loss: 0.5194 - accuracy: 0.9360\n",
      "Epoch 235/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 0.5141 - accuracy: 0.9404\n",
      "Epoch 236/500\n",
      "453/453 [==============================] - 0s 647us/sample - loss: 0.5101 - accuracy: 0.9426\n",
      "Epoch 237/500\n",
      "453/453 [==============================] - 0s 610us/sample - loss: 0.5055 - accuracy: 0.9426\n",
      "Epoch 238/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.5031 - accuracy: 0.9426\n",
      "Epoch 239/500\n",
      "453/453 [==============================] - 0s 632us/sample - loss: 0.4974 - accuracy: 0.9448\n",
      "Epoch 240/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 0.4958 - accuracy: 0.9448 - loss: 0.4828 - accuracy: \n",
      "Epoch 241/500\n",
      "453/453 [==============================] - 0s 654us/sample - loss: 0.4892 - accuracy: 0.9448\n",
      "Epoch 242/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 0.4844 - accuracy: 0.9448\n",
      "Epoch 243/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 0.4898 - accuracy: 0.9404\n",
      "Epoch 244/500\n",
      "453/453 [==============================] - 0s 612us/sample - loss: 0.4855 - accuracy: 0.9404\n",
      "Epoch 245/500\n",
      "453/453 [==============================] - 0s 632us/sample - loss: 0.4773 - accuracy: 0.9426\n",
      "Epoch 246/500\n",
      "453/453 [==============================] - 0s 645us/sample - loss: 0.4734 - accuracy: 0.9426\n",
      "Epoch 247/500\n",
      "453/453 [==============================] - 0s 651us/sample - loss: 0.4681 - accuracy: 0.9448\n",
      "Epoch 248/500\n",
      "453/453 [==============================] - 0s 620us/sample - loss: 0.4635 - accuracy: 0.9448\n",
      "Epoch 249/500\n",
      "453/453 [==============================] - 0s 645us/sample - loss: 0.4599 - accuracy: 0.9404\n",
      "Epoch 250/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 0.4542 - accuracy: 0.9426 - loss: 0.4937 - accuracy\n",
      "Epoch 251/500\n",
      "453/453 [==============================] - 0s 631us/sample - loss: 0.4517 - accuracy: 0.9404\n",
      "Epoch 252/500\n",
      "453/453 [==============================] - 0s 595us/sample - loss: 0.4473 - accuracy: 0.9382\n",
      "Epoch 253/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.4445 - accuracy: 0.9448\n",
      "Epoch 254/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.4403 - accuracy: 0.9426\n",
      "Epoch 255/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.4366 - accuracy: 0.9382\n",
      "Epoch 256/500\n",
      "453/453 [==============================] - 0s 598us/sample - loss: 0.4321 - accuracy: 0.9426\n",
      "Epoch 257/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.4282 - accuracy: 0.9448\n",
      "Epoch 258/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.4272 - accuracy: 0.9448\n",
      "Epoch 259/500\n",
      "453/453 [==============================] - 0s 647us/sample - loss: 0.4258 - accuracy: 0.9404\n",
      "Epoch 260/500\n",
      "453/453 [==============================] - 0s 612us/sample - loss: 0.4230 - accuracy: 0.9404\n",
      "Epoch 261/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.4222 - accuracy: 0.9448\n",
      "Epoch 262/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.4225 - accuracy: 0.9426\n",
      "Epoch 263/500\n",
      "453/453 [==============================] - 0s 610us/sample - loss: 0.4340 - accuracy: 0.9382\n",
      "Epoch 264/500\n",
      "453/453 [==============================] - 0s 612us/sample - loss: 0.4184 - accuracy: 0.9426\n",
      "Epoch 265/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.4094 - accuracy: 0.9492\n",
      "Epoch 266/500\n",
      "453/453 [==============================] - 0s 627us/sample - loss: 0.4051 - accuracy: 0.9492\n",
      "Epoch 267/500\n",
      "453/453 [==============================] - 0s 611us/sample - loss: 0.4007 - accuracy: 0.9470\n",
      "Epoch 268/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.3953 - accuracy: 0.9492\n",
      "Epoch 269/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.3915 - accuracy: 0.9470\n",
      "Epoch 270/500\n",
      "453/453 [==============================] - 0s 633us/sample - loss: 0.3871 - accuracy: 0.9492\n",
      "Epoch 271/500\n",
      "453/453 [==============================] - 0s 575us/sample - loss: 0.3893 - accuracy: 0.9492\n",
      "Epoch 272/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 0.3815 - accuracy: 0.9492\n",
      "Epoch 273/500\n",
      "453/453 [==============================] - 0s 641us/sample - loss: 0.3787 - accuracy: 0.9492\n",
      "Epoch 274/500\n",
      "453/453 [==============================] - 0s 629us/sample - loss: 0.3988 - accuracy: 0.9382\n",
      "Epoch 275/500\n",
      "453/453 [==============================] - 0s 539us/sample - loss: 0.4155 - accuracy: 0.9404\n",
      "Epoch 276/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.4033 - accuracy: 0.9404\n",
      "Epoch 277/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.3958 - accuracy: 0.9448\n",
      "Epoch 278/500\n",
      "453/453 [==============================] - 0s 641us/sample - loss: 0.3939 - accuracy: 0.9426\n",
      "Epoch 279/500\n",
      "453/453 [==============================] - 0s 590us/sample - loss: 0.3892 - accuracy: 0.9360\n",
      "Epoch 280/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.3773 - accuracy: 0.9404\n",
      "Epoch 281/500\n",
      "453/453 [==============================] - 0s 633us/sample - loss: 0.3937 - accuracy: 0.9316\n",
      "Epoch 282/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.4039 - accuracy: 0.9338\n",
      "Epoch 283/500\n",
      "453/453 [==============================] - 0s 612us/sample - loss: 0.3831 - accuracy: 0.9448\n",
      "Epoch 284/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.3785 - accuracy: 0.9426\n",
      "Epoch 285/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.3802 - accuracy: 0.9404\n",
      "Epoch 286/500\n",
      "453/453 [==============================] - 0s 632us/sample - loss: 0.3547 - accuracy: 0.9448\n",
      "Epoch 287/500\n",
      "453/453 [==============================] - 0s 647us/sample - loss: 0.3525 - accuracy: 0.9492\n",
      "Epoch 288/500\n",
      "453/453 [==============================] - 0s 631us/sample - loss: 0.3504 - accuracy: 0.9470 - loss: 0.3253 - accuracy: \n",
      "Epoch 289/500\n",
      "453/453 [==============================] - 0s 653us/sample - loss: 0.3447 - accuracy: 0.9470\n",
      "Epoch 290/500\n",
      "453/453 [==============================] - 0s 635us/sample - loss: 0.3450 - accuracy: 0.9492\n",
      "Epoch 291/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 0.3412 - accuracy: 0.9514\n",
      "Epoch 292/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 0.3358 - accuracy: 0.9470\n",
      "Epoch 293/500\n",
      "453/453 [==============================] - 0s 637us/sample - loss: 0.3405 - accuracy: 0.9492\n",
      "Epoch 294/500\n",
      "453/453 [==============================] - 0s 610us/sample - loss: 0.3320 - accuracy: 0.9514\n",
      "Epoch 295/500\n",
      "453/453 [==============================] - 0s 635us/sample - loss: 0.3412 - accuracy: 0.9470\n",
      "Epoch 296/500\n",
      "453/453 [==============================] - 0s 626us/sample - loss: 0.3358 - accuracy: 0.9448\n",
      "Epoch 297/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.3321 - accuracy: 0.9426\n",
      "Epoch 298/500\n",
      "453/453 [==============================] - 0s 614us/sample - loss: 0.3260 - accuracy: 0.9448\n",
      "Epoch 299/500\n",
      "453/453 [==============================] - 0s 635us/sample - loss: 0.3173 - accuracy: 0.9470\n",
      "Epoch 300/500\n",
      "453/453 [==============================] - 0s 632us/sample - loss: 0.3141 - accuracy: 0.9514\n",
      "Epoch 301/500\n",
      "453/453 [==============================] - 0s 641us/sample - loss: 0.3114 - accuracy: 0.9514\n",
      "Epoch 302/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 0.3079 - accuracy: 0.9492 - loss: 0.3180 - accuracy:  - ETA: 0s - loss: 0.3045 - accuracy: 0.95\n",
      "Epoch 303/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.3052 - accuracy: 0.9492\n",
      "Epoch 304/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 0.3033 - accuracy: 0.9514\n",
      "Epoch 305/500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "453/453 [==============================] - 0s 630us/sample - loss: 0.3026 - accuracy: 0.9514\n",
      "Epoch 306/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 0.2997 - accuracy: 0.9514\n",
      "Epoch 307/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 0.2970 - accuracy: 0.9514\n",
      "Epoch 308/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.2939 - accuracy: 0.9514\n",
      "Epoch 309/500\n",
      "453/453 [==============================] - 0s 594us/sample - loss: 0.2917 - accuracy: 0.9470\n",
      "Epoch 310/500\n",
      "453/453 [==============================] - 0s 647us/sample - loss: 0.2895 - accuracy: 0.9536\n",
      "Epoch 311/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 0.2871 - accuracy: 0.9536\n",
      "Epoch 312/500\n",
      "453/453 [==============================] - 0s 654us/sample - loss: 0.2864 - accuracy: 0.9514\n",
      "Epoch 313/500\n",
      "453/453 [==============================] - 0s 631us/sample - loss: 0.2843 - accuracy: 0.9536\n",
      "Epoch 314/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.2813 - accuracy: 0.9536\n",
      "Epoch 315/500\n",
      "453/453 [==============================] - 0s 647us/sample - loss: 0.2797 - accuracy: 0.9536\n",
      "Epoch 316/500\n",
      "453/453 [==============================] - 0s 645us/sample - loss: 0.2774 - accuracy: 0.9514\n",
      "Epoch 317/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 0.2811 - accuracy: 0.9492\n",
      "Epoch 318/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.2815 - accuracy: 0.9492\n",
      "Epoch 319/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 0.2757 - accuracy: 0.9536\n",
      "Epoch 320/500\n",
      "453/453 [==============================] - 0s 641us/sample - loss: 0.2718 - accuracy: 0.9514\n",
      "Epoch 321/500\n",
      "453/453 [==============================] - 0s 612us/sample - loss: 0.2713 - accuracy: 0.9536\n",
      "Epoch 322/500\n",
      "453/453 [==============================] - 0s 645us/sample - loss: 0.2691 - accuracy: 0.9536\n",
      "Epoch 323/500\n",
      "453/453 [==============================] - 0s 630us/sample - loss: 0.2662 - accuracy: 0.9536\n",
      "Epoch 324/500\n",
      "453/453 [==============================] - 0s 619us/sample - loss: 0.2642 - accuracy: 0.9536\n",
      "Epoch 325/500\n",
      "453/453 [==============================] - 0s 620us/sample - loss: 0.2619 - accuracy: 0.9536\n",
      "Epoch 326/500\n",
      "453/453 [==============================] - 0s 646us/sample - loss: 0.2605 - accuracy: 0.9536\n",
      "Epoch 327/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.2587 - accuracy: 0.9536\n",
      "Epoch 328/500\n",
      "453/453 [==============================] - 0s 786us/sample - loss: 0.2576 - accuracy: 0.9492\n",
      "Epoch 329/500\n",
      "453/453 [==============================] - 0s 806us/sample - loss: 0.2563 - accuracy: 0.9470\n",
      "Epoch 330/500\n",
      "453/453 [==============================] - 0s 821us/sample - loss: 0.2555 - accuracy: 0.9492\n",
      "Epoch 331/500\n",
      "453/453 [==============================] - 0s 713us/sample - loss: 0.2536 - accuracy: 0.9536\n",
      "Epoch 332/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.2529 - accuracy: 0.9492\n",
      "Epoch 333/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 0.2520 - accuracy: 0.9492\n",
      "Epoch 334/500\n",
      "453/453 [==============================] - 0s 635us/sample - loss: 0.2503 - accuracy: 0.9514\n",
      "Epoch 335/500\n",
      "453/453 [==============================] - 0s 614us/sample - loss: 0.2492 - accuracy: 0.9514\n",
      "Epoch 336/500\n",
      "453/453 [==============================] - 0s 637us/sample - loss: 0.2574 - accuracy: 0.9448\n",
      "Epoch 337/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.2761 - accuracy: 0.9470\n",
      "Epoch 338/500\n",
      "453/453 [==============================] - 0s 649us/sample - loss: 0.2887 - accuracy: 0.9382\n",
      "Epoch 339/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.2736 - accuracy: 0.9470\n",
      "Epoch 340/500\n",
      "453/453 [==============================] - 0s 630us/sample - loss: 0.2636 - accuracy: 0.9448\n",
      "Epoch 341/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.2576 - accuracy: 0.9426\n",
      "Epoch 342/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.2512 - accuracy: 0.9470\n",
      "Epoch 343/500\n",
      "453/453 [==============================] - 0s 592us/sample - loss: 0.2467 - accuracy: 0.9448\n",
      "Epoch 344/500\n",
      "453/453 [==============================] - 0s 647us/sample - loss: 0.2397 - accuracy: 0.9470\n",
      "Epoch 345/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.2386 - accuracy: 0.9448\n",
      "Epoch 346/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 0.2366 - accuracy: 0.9514\n",
      "Epoch 347/500\n",
      "453/453 [==============================] - 0s 698us/sample - loss: 0.2344 - accuracy: 0.9448\n",
      "Epoch 348/500\n",
      "453/453 [==============================] - 0s 986us/sample - loss: 0.2322 - accuracy: 0.9514\n",
      "Epoch 349/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.2306 - accuracy: 0.9514\n",
      "Epoch 350/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.2288 - accuracy: 0.9514\n",
      "Epoch 351/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.2293 - accuracy: 0.9492\n",
      "Epoch 352/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.2269 - accuracy: 0.9536\n",
      "Epoch 353/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.2257 - accuracy: 0.9514\n",
      "Epoch 354/500\n",
      "453/453 [==============================] - 0s 978us/sample - loss: 0.2227 - accuracy: 0.9514 - loss: 0.2279 - accuracy: 0.94\n",
      "Epoch 355/500\n",
      "453/453 [==============================] - 0s 988us/sample - loss: 0.2232 - accuracy: 0.9514\n",
      "Epoch 356/500\n",
      "453/453 [==============================] - 0s 988us/sample - loss: 0.2219 - accuracy: 0.9514\n",
      "Epoch 357/500\n",
      "453/453 [==============================] - 0s 973us/sample - loss: 0.2204 - accuracy: 0.9514\n",
      "Epoch 358/500\n",
      "453/453 [==============================] - 0s 990us/sample - loss: 0.2186 - accuracy: 0.9426\n",
      "Epoch 359/500\n",
      "453/453 [==============================] - 0s 994us/sample - loss: 0.2169 - accuracy: 0.9470\n",
      "Epoch 360/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.2169 - accuracy: 0.9404\n",
      "Epoch 361/500\n",
      "453/453 [==============================] - 0s 859us/sample - loss: 0.2154 - accuracy: 0.9492\n",
      "Epoch 362/500\n",
      "453/453 [==============================] - 0s 627us/sample - loss: 0.2138 - accuracy: 0.9492\n",
      "Epoch 363/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.2132 - accuracy: 0.9448\n",
      "Epoch 364/500\n",
      "453/453 [==============================] - 0s 633us/sample - loss: 0.2119 - accuracy: 0.9514\n",
      "Epoch 365/500\n",
      "453/453 [==============================] - 0s 601us/sample - loss: 0.2110 - accuracy: 0.9514\n",
      "Epoch 366/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.2096 - accuracy: 0.9492\n",
      "Epoch 367/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 0.2087 - accuracy: 0.9536\n",
      "Epoch 368/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.2079 - accuracy: 0.9492\n",
      "Epoch 369/500\n",
      "453/453 [==============================] - 0s 605us/sample - loss: 0.2066 - accuracy: 0.9492\n",
      "Epoch 370/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.2064 - accuracy: 0.9514\n",
      "Epoch 371/500\n",
      "453/453 [==============================] - 0s 645us/sample - loss: 0.2045 - accuracy: 0.9514\n",
      "Epoch 372/500\n",
      "453/453 [==============================] - 0s 656us/sample - loss: 0.2042 - accuracy: 0.9536\n",
      "Epoch 373/500\n",
      "453/453 [==============================] - 0s 635us/sample - loss: 0.2027 - accuracy: 0.9448\n",
      "Epoch 374/500\n",
      "453/453 [==============================] - 0s 658us/sample - loss: 0.2016 - accuracy: 0.9536\n",
      "Epoch 375/500\n",
      "453/453 [==============================] - 0s 829us/sample - loss: 0.2015 - accuracy: 0.9492\n",
      "Epoch 376/500\n",
      "453/453 [==============================] - 0s 677us/sample - loss: 0.1990 - accuracy: 0.9492\n",
      "Epoch 377/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 0.1981 - accuracy: 0.9536\n",
      "Epoch 378/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.1970 - accuracy: 0.9470\n",
      "Epoch 379/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 0.1957 - accuracy: 0.9492\n",
      "Epoch 380/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 0.1952 - accuracy: 0.9492 - loss: 0.2163 - accuracy: \n",
      "Epoch 381/500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "453/453 [==============================] - 0s 647us/sample - loss: 0.1945 - accuracy: 0.9514\n",
      "Epoch 382/500\n",
      "453/453 [==============================] - 0s 631us/sample - loss: 0.1942 - accuracy: 0.9514\n",
      "Epoch 383/500\n",
      "453/453 [==============================] - 0s 649us/sample - loss: 0.1930 - accuracy: 0.9514\n",
      "Epoch 384/500\n",
      "453/453 [==============================] - 0s 589us/sample - loss: 0.1913 - accuracy: 0.9536\n",
      "Epoch 385/500\n",
      "453/453 [==============================] - 0s 635us/sample - loss: 0.1903 - accuracy: 0.9514\n",
      "Epoch 386/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 0.1896 - accuracy: 0.9514 - loss: 0.1916 - accuracy: 0.95\n",
      "Epoch 387/500\n",
      "453/453 [==============================] - 0s 645us/sample - loss: 0.1896 - accuracy: 0.9536\n",
      "Epoch 388/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.1879 - accuracy: 0.9536\n",
      "Epoch 389/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 0.1872 - accuracy: 0.9558\n",
      "Epoch 390/500\n",
      "453/453 [==============================] - 0s 633us/sample - loss: 0.1864 - accuracy: 0.9514\n",
      "Epoch 391/500\n",
      "453/453 [==============================] - 0s 615us/sample - loss: 0.1850 - accuracy: 0.9536\n",
      "Epoch 392/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.1843 - accuracy: 0.9536\n",
      "Epoch 393/500\n",
      "453/453 [==============================] - 0s 632us/sample - loss: 0.1834 - accuracy: 0.9536\n",
      "Epoch 394/500\n",
      "453/453 [==============================] - 0s 649us/sample - loss: 0.1825 - accuracy: 0.9536\n",
      "Epoch 395/500\n",
      "453/453 [==============================] - 0s 667us/sample - loss: 0.1821 - accuracy: 0.9536\n",
      "Epoch 396/500\n",
      "453/453 [==============================] - 0s 821us/sample - loss: 0.1815 - accuracy: 0.9536\n",
      "Epoch 397/500\n",
      "453/453 [==============================] - 0s 793us/sample - loss: 0.1807 - accuracy: 0.9492\n",
      "Epoch 398/500\n",
      "453/453 [==============================] - 0s 701us/sample - loss: 0.1806 - accuracy: 0.9536\n",
      "Epoch 399/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.1836 - accuracy: 0.9536\n",
      "Epoch 400/500\n",
      "453/453 [==============================] - 0s 652us/sample - loss: 0.1812 - accuracy: 0.9514\n",
      "Epoch 401/500\n",
      "453/453 [==============================] - 0s 632us/sample - loss: 0.1810 - accuracy: 0.9514\n",
      "Epoch 402/500\n",
      "453/453 [==============================] - 0s 625us/sample - loss: 0.1794 - accuracy: 0.9536\n",
      "Epoch 403/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 0.1794 - accuracy: 0.9492\n",
      "Epoch 404/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.1767 - accuracy: 0.9536\n",
      "Epoch 405/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 0.1759 - accuracy: 0.9514\n",
      "Epoch 406/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.1740 - accuracy: 0.9514\n",
      "Epoch 407/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.1734 - accuracy: 0.9514\n",
      "Epoch 408/500\n",
      "453/453 [==============================] - 0s 979us/sample - loss: 0.1724 - accuracy: 0.9514 - loss: 0.1642 - accuracy: 0.\n",
      "Epoch 409/500\n",
      "453/453 [==============================] - 0s 973us/sample - loss: 0.1718 - accuracy: 0.9514\n",
      "Epoch 410/500\n",
      "453/453 [==============================] - 0s 993us/sample - loss: 0.1712 - accuracy: 0.9470\n",
      "Epoch 411/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.1702 - accuracy: 0.9470\n",
      "Epoch 412/500\n",
      "453/453 [==============================] - 0s 1ms/sample - loss: 0.1698 - accuracy: 0.9470\n",
      "Epoch 413/500\n",
      "453/453 [==============================] - 0s 802us/sample - loss: 0.1711 - accuracy: 0.9514\n",
      "Epoch 414/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 0.1827 - accuracy: 0.9492\n",
      "Epoch 415/500\n",
      "453/453 [==============================] - 0s 626us/sample - loss: 0.1790 - accuracy: 0.9514\n",
      "Epoch 416/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 0.1818 - accuracy: 0.9536\n",
      "Epoch 417/500\n",
      "453/453 [==============================] - 0s 629us/sample - loss: 0.1727 - accuracy: 0.9470\n",
      "Epoch 418/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.1691 - accuracy: 0.9448\n",
      "Epoch 419/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.1785 - accuracy: 0.9470\n",
      "Epoch 420/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.1818 - accuracy: 0.9492\n",
      "Epoch 421/500\n",
      "453/453 [==============================] - 0s 583us/sample - loss: 0.1809 - accuracy: 0.9492\n",
      "Epoch 422/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 0.1925 - accuracy: 0.9492\n",
      "Epoch 423/500\n",
      "453/453 [==============================] - 0s 629us/sample - loss: 0.1837 - accuracy: 0.9514\n",
      "Epoch 424/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 0.1721 - accuracy: 0.9514\n",
      "Epoch 425/500\n",
      "453/453 [==============================] - 0s 620us/sample - loss: 0.1702 - accuracy: 0.9492\n",
      "Epoch 426/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.1673 - accuracy: 0.9470\n",
      "Epoch 427/500\n",
      "453/453 [==============================] - 0s 636us/sample - loss: 0.1644 - accuracy: 0.9470\n",
      "Epoch 428/500\n",
      "453/453 [==============================] - 0s 641us/sample - loss: 0.1623 - accuracy: 0.9514 - loss: 0.1460 - accuracy\n",
      "Epoch 429/500\n",
      "453/453 [==============================] - 0s 624us/sample - loss: 0.1606 - accuracy: 0.9492\n",
      "Epoch 430/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.1600 - accuracy: 0.9492\n",
      "Epoch 431/500\n",
      "453/453 [==============================] - 0s 641us/sample - loss: 0.1588 - accuracy: 0.9514\n",
      "Epoch 432/500\n",
      "453/453 [==============================] - 0s 624us/sample - loss: 0.1580 - accuracy: 0.9492\n",
      "Epoch 433/500\n",
      "453/453 [==============================] - 0s 635us/sample - loss: 0.1568 - accuracy: 0.9470\n",
      "Epoch 434/500\n",
      "453/453 [==============================] - 0s 631us/sample - loss: 0.1570 - accuracy: 0.9514\n",
      "Epoch 435/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 0.1570 - accuracy: 0.9448\n",
      "Epoch 436/500\n",
      "453/453 [==============================] - 0s 619us/sample - loss: 0.1565 - accuracy: 0.9448\n",
      "Epoch 437/500\n",
      "453/453 [==============================] - 0s 651us/sample - loss: 0.1553 - accuracy: 0.9536\n",
      "Epoch 438/500\n",
      "453/453 [==============================] - 0s 630us/sample - loss: 0.1542 - accuracy: 0.9426\n",
      "Epoch 439/500\n",
      "453/453 [==============================] - 0s 633us/sample - loss: 0.1543 - accuracy: 0.9514\n",
      "Epoch 440/500\n",
      "453/453 [==============================] - 0s 621us/sample - loss: 0.1541 - accuracy: 0.9492\n",
      "Epoch 441/500\n",
      "453/453 [==============================] - 0s 632us/sample - loss: 0.1527 - accuracy: 0.9448\n",
      "Epoch 442/500\n",
      "453/453 [==============================] - 0s 641us/sample - loss: 0.1530 - accuracy: 0.9514\n",
      "Epoch 443/500\n",
      "453/453 [==============================] - 0s 585us/sample - loss: 0.1522 - accuracy: 0.9514\n",
      "Epoch 444/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.1521 - accuracy: 0.9448\n",
      "Epoch 445/500\n",
      "453/453 [==============================] - 0s 647us/sample - loss: 0.1506 - accuracy: 0.9492\n",
      "Epoch 446/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 0.1502 - accuracy: 0.9536\n",
      "Epoch 447/500\n",
      "453/453 [==============================] - 0s 608us/sample - loss: 0.1496 - accuracy: 0.9470 - loss: 0.1531 - accuracy: \n",
      "Epoch 448/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 0.1487 - accuracy: 0.9470\n",
      "Epoch 449/500\n",
      "453/453 [==============================] - 0s 631us/sample - loss: 0.1483 - accuracy: 0.9514\n",
      "Epoch 450/500\n",
      "453/453 [==============================] - 0s 667us/sample - loss: 0.1472 - accuracy: 0.9536\n",
      "Epoch 451/500\n",
      "453/453 [==============================] - 0s 714us/sample - loss: 0.1474 - accuracy: 0.9536\n",
      "Epoch 452/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.1475 - accuracy: 0.9470\n",
      "Epoch 453/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.1464 - accuracy: 0.9514 - loss: 0.1430 - accuracy: 0.95\n",
      "Epoch 454/500\n",
      "453/453 [==============================] - 0s 637us/sample - loss: 0.1457 - accuracy: 0.9492 - loss: 0.1174 - accuracy\n",
      "Epoch 455/500\n",
      "453/453 [==============================] - 0s 585us/sample - loss: 0.1459 - accuracy: 0.9470\n",
      "Epoch 456/500\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "453/453 [==============================] - 0s 638us/sample - loss: 0.1458 - accuracy: 0.9448\n",
      "Epoch 457/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.1444 - accuracy: 0.9514\n",
      "Epoch 458/500\n",
      "453/453 [==============================] - 0s 763us/sample - loss: 0.1441 - accuracy: 0.9492\n",
      "Epoch 459/500\n",
      "453/453 [==============================] - 0s 580us/sample - loss: 0.1433 - accuracy: 0.9470\n",
      "Epoch 460/500\n",
      "453/453 [==============================] - 0s 637us/sample - loss: 0.1443 - accuracy: 0.9492\n",
      "Epoch 461/500\n",
      "453/453 [==============================] - 0s 642us/sample - loss: 0.1427 - accuracy: 0.9514\n",
      "Epoch 462/500\n",
      "453/453 [==============================] - 0s 626us/sample - loss: 0.1428 - accuracy: 0.9492\n",
      "Epoch 463/500\n",
      "453/453 [==============================] - 0s 646us/sample - loss: 0.1420 - accuracy: 0.9514\n",
      "Epoch 464/500\n",
      "453/453 [==============================] - 0s 643us/sample - loss: 0.1418 - accuracy: 0.9492\n",
      "Epoch 465/500\n",
      "453/453 [==============================] - 0s 645us/sample - loss: 0.1420 - accuracy: 0.9470\n",
      "Epoch 466/500\n",
      "453/453 [==============================] - 0s 645us/sample - loss: 0.1410 - accuracy: 0.9492\n",
      "Epoch 467/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 0.1403 - accuracy: 0.9492\n",
      "Epoch 468/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.1398 - accuracy: 0.9514\n",
      "Epoch 469/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.1392 - accuracy: 0.9492\n",
      "Epoch 470/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 0.1394 - accuracy: 0.9514\n",
      "Epoch 471/500\n",
      "453/453 [==============================] - 0s 647us/sample - loss: 0.1389 - accuracy: 0.9492 - loss: 0.1341 - accuracy\n",
      "Epoch 472/500\n",
      "453/453 [==============================] - 0s 639us/sample - loss: 0.1388 - accuracy: 0.9448\n",
      "Epoch 473/500\n",
      "453/453 [==============================] - 0s 641us/sample - loss: 0.1385 - accuracy: 0.9536\n",
      "Epoch 474/500\n",
      "453/453 [==============================] - 0s 616us/sample - loss: 0.1375 - accuracy: 0.9470\n",
      "Epoch 475/500\n",
      "453/453 [==============================] - 0s 634us/sample - loss: 0.1372 - accuracy: 0.9492\n",
      "Epoch 476/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 0.1374 - accuracy: 0.9514\n",
      "Epoch 477/500\n",
      "453/453 [==============================] - 0s 618us/sample - loss: 0.1363 - accuracy: 0.9492\n",
      "Epoch 478/500\n",
      "453/453 [==============================] - 0s 656us/sample - loss: 0.1359 - accuracy: 0.9492\n",
      "Epoch 479/500\n",
      "453/453 [==============================] - 0s 644us/sample - loss: 0.1352 - accuracy: 0.9492\n",
      "Epoch 480/500\n",
      "453/453 [==============================] - 0s 632us/sample - loss: 0.1348 - accuracy: 0.9514\n",
      "Epoch 481/500\n",
      "453/453 [==============================] - 0s 630us/sample - loss: 0.1351 - accuracy: 0.9514\n",
      "Epoch 482/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.1349 - accuracy: 0.9492 - loss: 0.1557 - accuracy:  - ETA: 0s - loss: 0.1326 - accuracy: 0.95\n",
      "Epoch 483/500\n",
      "453/453 [==============================] - 0s 646us/sample - loss: 0.1345 - accuracy: 0.9470\n",
      "Epoch 484/500\n",
      "453/453 [==============================] - 0s 623us/sample - loss: 0.1341 - accuracy: 0.9514\n",
      "Epoch 485/500\n",
      "453/453 [==============================] - 0s 620us/sample - loss: 0.1336 - accuracy: 0.9470\n",
      "Epoch 486/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.1333 - accuracy: 0.9514\n",
      "Epoch 487/500\n",
      "453/453 [==============================] - 0s 954us/sample - loss: 0.1330 - accuracy: 0.9448\n",
      "Epoch 488/500\n",
      "453/453 [==============================] - 0s 971us/sample - loss: 0.1327 - accuracy: 0.9536\n",
      "Epoch 489/500\n",
      "453/453 [==============================] - 0s 992us/sample - loss: 0.1326 - accuracy: 0.9470\n",
      "Epoch 490/500\n",
      "453/453 [==============================] - 0s 997us/sample - loss: 0.1311 - accuracy: 0.9448\n",
      "Epoch 491/500\n",
      "453/453 [==============================] - 0s 626us/sample - loss: 0.1310 - accuracy: 0.9492\n",
      "Epoch 492/500\n",
      "453/453 [==============================] - 0s 637us/sample - loss: 0.1311 - accuracy: 0.9492\n",
      "Epoch 493/500\n",
      "453/453 [==============================] - 0s 650us/sample - loss: 0.1306 - accuracy: 0.9514\n",
      "Epoch 494/500\n",
      "453/453 [==============================] - 0s 646us/sample - loss: 0.1303 - accuracy: 0.9470\n",
      "Epoch 495/500\n",
      "453/453 [==============================] - 0s 649us/sample - loss: 0.1315 - accuracy: 0.9492\n",
      "Epoch 496/500\n",
      "453/453 [==============================] - 0s 641us/sample - loss: 0.1321 - accuracy: 0.9426\n",
      "Epoch 497/500\n",
      "453/453 [==============================] - 0s 638us/sample - loss: 0.1410 - accuracy: 0.9426\n",
      "Epoch 498/500\n",
      "453/453 [==============================] - 0s 628us/sample - loss: 0.1374 - accuracy: 0.9514\n",
      "Epoch 499/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.1497 - accuracy: 0.9492\n",
      "Epoch 500/500\n",
      "453/453 [==============================] - 0s 640us/sample - loss: 0.1525 - accuracy: 0.9492\n"
     ]
    }
   ],
   "source": [
    "model = Sequential()\n",
    "model.add(Embedding(total_words,64,input_length=max_sequence_len-1))\n",
    "model.add(Bidirectional(LSTM(20)))\n",
    "model.add(Dense(total_words,activation='softmax'))\n",
    "model.compile(loss='categorical_crossentropy',optimizer='adam',metrics=['accuracy'])\n",
    "hist = model.fit(xs,ys,epochs=500)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<matplotlib.lines.Line2D at 0x27e4557a9c8>]"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAWoAAAD4CAYAAADFAawfAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAAe40lEQVR4nO3deXSc5X328e9vZjSj1ZK1WrbkXd5tbFCMwTYBYsBAaJaGBJrQNCxOk7TFTXPacNI3adI2bdM2W5MGfCBAlgLhzcISF0LABHAMRgavGONNXuRFsmRJlmRJs9zvHxr7NWGxbGv0PDNzfc6Zo5lnRtJ1C/ni1j3PYs45RETEvwJeBxARkXenohYR8TkVtYiIz6moRUR8TkUtIuJzoVR80fLycjd+/PhUfGkRkYy0bt26I865ird7LiVFPX78eBoaGlLxpUVEMpKZ7Xmn57T0ISLicypqERGfU1GLiPicilpExOdU1CIiPqeiFhHxORW1iIjP+aao+2Jx7vrdThoa27yOIiLiK74p6kQC7vt9I1997DUSCZ0jW0TkBN8UdV44yPIldWxq6mBjU4fXcUREfMM3RQ2wuG7gMPcN+9q9DSIi4iO+Kurq4lwqiyKsV1GLiJzkq6I2M6ZVj2BHc5fXUUREfMNXRQ1QXhimrbvf6xgiIr7hu6IuKwhzpKsPXR1dRGSA/4q6MEJfLEFPf9zrKCIivuC7oi4tCAPQ2qXlDxER8GFRlxcOFPWR7j6Pk4iI+IPvirqsIAJAm2bUIiKAD4u6vGigqFu6NKMWEQEfFnVlUYSAwYH2415HERHxBd8VdU4wwKgRuTSpqEVEAB8WNcDokjzNqEVEknxZ1GNG5nGgvdfrGCIivuDLoq4dmc+B9uMc10EvIiL+LOoLJ5YSSzjW7DridRQREc8NqqjNrNHMNpnZejNrSHWo+RNKycsJsnLToVR/KxER3zuTGfVlzrm5zrn6lKVJioSCXF9fw69ebWL/0Z5UfzsREV/z5dIHwGcunUTAjO+v2ul1FBERTw22qB3wGzNbZ2bL3u4FZrbMzBrMrKGlpeWcg1UX53Hj/Fp+1rCPbYeOnfPXExFJV4Mt6kXOufOBq4HPmdklf/gC59wK51y9c66+oqJiSML99RVTyM8J8t1ntg/J1xMRSUeDKmrnXFPyYzPwS2B+KkOdUJIf5uMLxvG/mw6yp7V7OL6liIjvnLaozazAzIpO3AeuBDanOtgJn1o4nlAgwN3P7x6ubyki4iuDmVFXAS+Y2QZgLfBr59wTqY11yjcfkcuH5o3hZw37aNUZ9UQkC522qJ1zu5xz5yVvM51z/zwcwU5186IJ9MUSPLbhwHB/axERz/l297xTTR1VxNSqIn696aDXUUREhl1aFDXAtXOqadhzlEMdOlmTiGSXtCnqa2ZX4xz872bNqkUku6RNUU+uLGTaqCJ+vVFFLSLZJW2KGuD9yeWPgx26qICIZI+0KuprZlcD6Kx6IpJV0qqoJ1YUMr16BCu194eIZJG0KmqApTNH8creozQf094fIpId0q6or5pVhXPw1GuHvY4iIjIs0q6op1YVMa4snyc2a51aRLJD2hW1mbF01ijW7Gylrbvf6zgiIimXdkUN8IHzxhBLOB1SLiJZIS2Lenp1EXWVhTzyapPXUUREUi4ti9rM+OC8MTTsOcq+Nl38VkQyW1oWNcAfnTcagEd16lMRyXBpW9S1pflcMG4kj65XUYtIZkvbogb4wNzRbDt8jNcPdXodRUQkZdK6qK+dXU0wYPzqVc2qRSRzpXVRlxVGWFxXzqPrm4gnnNdxRERSIq2LGuD6C2o50NHLs9uavY4iIpISaV/UV86soro4l++t2oFzmlWLSOZJ+6LOCQZYvqSOV/e26/wfIpKR0r6oAf74/BomVxby709uIxpPeB1HRGRIZURRh4IB/m7pNHYd6dZ+1SKScTKiqAGWTK+kZmQej21UUYtIZsmYojYzrp1dzeodR+joiXodR0RkyGRMUcPAxW+jccdvXtObiiKSOQZd1GYWNLNXzezxVAY6F3NqipPLHzpPtYhkjjOZUd8ObE1VkKFgZnxg7mhe2N6ii9+KSMYYVFGbWQ1wLXB3auOcuz8+vwYH/HjNHq+jiIgMicHOqL8N/C3wjjspm9kyM2sws4aWlpahyHZWJlYUctWMUdz/+0aO9epNRRFJf6ctajN7P9DsnFv3bq9zzq1wztU75+orKiqGLODZ+Oxlk+jsjfHQy/s8zSEiMhQGM6NeCPyRmTUCDwKXm9lPUprqHM2pKeE940fy4xf3kNBZ9UQkzZ22qJ1zdzjnapxz44EbgGecc59IebJz9IkF49jT2sNz271bhhERGQoZtR/1qa6eVU15YURvKopI2jujonbOPeuce3+qwgylcCjAjfNreWZbs65ULiJpLWNn1AA3zh+LAT99aa/XUUREzlpGF/XokjyumFHFzxr20ReLex1HROSsZHRRw8Cbim3d/bqogIikrYwv6oWTyhlXlq/lDxFJWxlf1IGAceP8sazd3ca2Q8e8jiMicsYyvqgBPlpfSyQU4P41jV5HERE5Y1lR1KUFYT4wdzS/fKVJFxUQkbSTFUUN8MmLx3M8GufhdTr/h4ikl6wp6pmji3nP+JHcv6aRuM7/ISJpJGuKGuCWRRPY13acx3UBXBFJI1lV1FfMGMW0UUV87bHXaO/p9zqOiMigZFVRBwPGf1x/Hq3d/TywVmvVIpIesqqoAWaNKebiSWX8aE0j0fg7XrBGRMQ3sq6oAW5eOIGDHb2s3KSrlYuI/2VlUV8+rZKpVUV866k36OmPeR1HRORdZWVRBwLGl6+bwZ62Hr7xxDav44iIvKusLGqAhZPL+cj5NTz48l6OdmsPEBHxr6wtaoBbF0+kN5rgpy/pcl0i4l9ZXdRTRxVxyZQK7l+zRxcWEBHfyuqiBrht8QRajvXxy1eavI4iIvK2sr6oF00u57zaEr779Ha6+7QHiIj4T9YXtZnxf66dzqHOXr7xxOtexxEReYusL2qA+vGlXH9BLQ++vI/Wrj6v44iIvImKOunWxRPoiyV0bUUR8R0VdVJdVRGXTq3gR2sa6Y1qDxAR8Q8V9SluWzyRI139PLpe56sWEf9QUZ/i4kllTBtVxN0v7CKhq8CIiE+ctqjNLNfM1prZBjPbYmZfHY5gXjAzPnvZZN443MXdL+zyOo6ICDC4GXUfcLlz7jxgLrDUzBakNJWHrptTzeXTKvn+qp10ab9qEfGB0xa1G9CVfJiTvGXsuoCZ8Vfvq6PjeJQ7n93pdRwRkcGtUZtZ0MzWA83AU865l1KaymNza0v40LwxrHhuF7tauk7/CSIiKTSoonbOxZ1zc4EaYL6ZzfrD15jZMjNrMLOGlpaWIY45/O64Zho5QeN7z+zwOoqIZLkz2uvDOdcOrAKWvs1zK5xz9c65+oqKiiGK553Kolw+OG8Mv950kOZjvV7HEZEsNpi9PirMrCR5Pw+4AsiKk2LcsmgCALf9aB1HdGi5iHhkMDPqamCVmW0EXmZgjfrx1Mbyh4kVhXznhrlsPdjJF3++0es4IpKlQqd7gXNuIzBvGLL40tJZ1dz+vm7+/cltbDt0jKmjiryOJCJZRkcmDsIN76klJ2j8aE2j11FEJAupqAehrDDCR+treejlfdpdT0SGnYp6kJYvmUIkFODfdHEBERlmKupBqiiK8On3TuLJLYdpaGzzOo6IZBEV9Rm4dfEEKosifH3lVpzL2KPoRcRnVNRnID8c4m+unMIre9tZuemQ13FEJEuoqM/QRy6oZXr1CP7hsS20dfd7HUdEsoCK+gwFA8Y3P3oeHT1R7vjFRi2BiEjKqajPwvTqEXzhqik8ueUwD6/b73UcEclwKuqzdOuiiSyYWMpXH93C3tYer+OISAZTUZ+lQMD4z4/OJRAwPv2TdRzq0Bn2RCQ1VNTnYExJHt//k/PZ1dLFh/97Nb3RuNeRRCQDqajP0SVTKrjzpgs40NHLg2v3eh1HRDKQinoIXDqlgoWTy/iP37zB7iPdXscRkQyjoh4CZsa/fGgOsUSCy/7jWX7xivYEEZGho6IeImPL8vnhJ99DdXEuX3l0C0d1MIyIDBEV9RC6eHI59988n66+GPP+8Sn+/lebvI4kIhlART3EplQV8amLB661+JMX9/L6oU6PE4lIulNRp8CXrp3OT265kLycICue2+V1HBFJcyrqFAgGjEV15dwwv5ZH1x/gQPtxryOJSBpTUafQLYsm4IAfvrDb6ygiksZU1ClUMzKf6+ZU88DavbT3aC8QETk7KuoU+8ylk+mNJfiXlbrWooicHRV1ik0dVcRtiyfyUMM+Vu844nUcEUlDKuphsHxJHRPKC7jjF5s43q8TN4nImVFRD4PcnCD/8uHZ7G3r4ZtPbfM6joikGRX1MFkwsYw/uXAs97ywm4bGNq/jiEgaUVEPozuunkZtaT6f+59XaDnW53UcEUkTpy1qM6s1s1Vm9pqZbTGz24cjWCYqys3hBx+/gPaeKH//q026MK6IDMpgZtQx4G+cczOABcDnzGxGamNlrhmjR7B8ycCFce/RgTAiMginLWrn3EHn3CvJ+8eArcCYVAfLZJ++ZCJXzxrFP6/cyspNB72OIyI+d0Zr1GY2HpgHvPQ2zy0zswYza2hpaRmieJkpEDC+9bG5nD92JH/90Ho2N3V4HUlEfGzQRW1mhcDPgeXOubecu9M5t8I5V++cq6+oqBjKjBkpNyfIXTddQFlBmE//eJ3eXBSRdzSoojazHAZK+qfOuV+kNlL2KC+McNdN9bR293Hbjxp0FXMReVuD2evDgHuArc65b6Y+UnaZXVPMtz82jw372/n8z9aTSGhPEBF5s8HMqBcCNwGXm9n65O2aFOfKKktnjeJL10xn5aZD/NuTOnmTiLxZ6HQvcM69ANgwZMlqtyyaQGNrN3f9bhdHjvXztQ/MpCBy2v88IpIF1AQ+YWb8w3UzKc0P871VO9jRfIx7PzWf0oKw19FExGM6hNxHQsEAn79yKituquf1Q8f48x+vIxpPeB1LRDymovahJTOq+PqHZrO2sY2P3LmGjuNRryOJiIdU1D71xxfU8F83zmNLUwdf//VWr+OIiIe0Ru1j1503mnV7jnLf7xspyc/hjmumex1JRDygova5L149jaM9/dz13C72tPbwd1dPY0J5gdexRGQYaenD53Jzgvzrh+fwqYXjWb3jCF/65SavI4nIMFNRp4G8cJCvXDeTv3pfHb/f2cpvthzyOpKIDCMVdRr55MXjmTl6BH/3840c6uj1Oo6IDBMVdRoJhwJ898Z59McS/Nm9aznQftzrSCIyDFTUaWZSRSE/+MQFNB09zvV3rmFPa7fXkUQkxVTUaeiSKRU8sGwBPf0xPnbXi+w+orIWyWQq6jQ1a0wx/3PbAvrjCT521xp2NHd5HUlEUkRFncamV4/ggdsWkHCOG1a8yO/e0CXQRDKRijrNTR1VxIPLFlCcF+LW+19WWYtkIBV1BphcWcQvPrOQSRWF3Hzfy/zj46/xyt6jXscSkSGios4Qxfk5/N/PXMxlUyu454XdfPi/f8/dz+/yOpaIDAEVdQYpjIS466Z6Hly2gEWTy/nGk9tYveMIzjkOd+oAGZF0pZMyZZhgwFgwsYwpVUXcuOJFbr7vZeqqCtnc1MmN88fyhSunUFYY8TqmiJwBzagzVGlBmAeWLWDBxDKaO/uYVFHAww37uO6/XuB4f9zreCJyBjSjzmClBWHuv3n+ycfPbmvmz+59me+t2s4XrpyKma5ZLJIONKPOIu+dUsFVM6v4/qqdPLL+gNdxRGSQVNRZxMz4749fwHm1JXzl0S069FwkTaios0wwYHz3hrkEDK6/cw2b9nd4HUlETkNFnYXGlRXw8J9fRCQU4MM/WM2K53aSSDivY4nIO1BRZ6nJlUU89peLuHxaJV9f+Tofv/slnd9axKdU1FmstCDMnZ+4gG98ZA4b97ez9NvP8dgGvcko4jenLWoz+6GZNZvZ5uEIJMPLzPhofS0rb1/MpMpC/vKBV/n8Q+vp7I16HU1EkgYzo74PWJriHOKxcWUFPPzpi1i+pI5HNhzg6m8/z/PbdSY+ET84bVE7554D2oYhi3gsFAywfMkUHv7ziwiHAtx0z1r+6oFXaT727ucJaWo/jnN6M1IkVYZsjdrMlplZg5k1tLRoJpbOzh87kv+9fTHLl9TxxOZDvO8/f8eP1jQSiyfe8trth4+x8F+f4du/3e5BUpHsMGRF7Zxb4Zyrd87VV1RUDNWXFY/k5gRZvmQKTyxfzJyaYr78yBau/e4LrN5x5ORr+mMJfri6EYDvPL2dF3e1epRWJLPZYP5kNbPxwOPOuVmD+aL19fWuoaHhHKOJXzjneGLzIf555Vb2Hz3OrDEjyA+HaDnWd/LoxrGl+YzIC/HYXyzSOUREzoKZrXPO1b/dc9o9T07LzLh6djW//fx7+dulU9nd0k1DYxt90YGz8H3m0kl87rJJbG7q5J9+vdXjtCKZ57RnzzOzB4BLgXIz2w98xTl3T6qDif/k5gT57KWT+fiF44iEAuTmBE8+l0g4Xt3bzr2rd/Ph88cwc3Sxh0lFMstg9vq40TlX7ZzLcc7VqKSlOC/nTSUNEAgYX7hqKqUFEW5Y8SJbDugcIiJDRUsfMmTKCyP88rMXUxQJ8af3rOWF7UdO/0kicloqahlStaX5/OTWCynOy+ET97zEzfe9zJ5WnU5V5FwMaq+PM6W9PqQ3Gufu53fxg2d30hON894pFbx2oJPzx47kax+cSWVRrtcRRXzl3fb6UFFLSh3q6OWnL+3h3tWNdPXFgIGrpf/k1guZW1vibTgRH1FRi+c6eqJsPdRJTtBY/tB62nui3LpoIp9YMFZXRRdBRS0+s6+thy8/splV21qIhAJcNXMUV88axZIZVeQE9baJZKd3K2pdhVyGXW1pPvd+aj47mo9x7+pGVm46yKMbDjC9egR/etE4rp1TzYjcHK9jiviGZtTiuVg8wcrNh/jOb99gZ0v3yVn2B+eNZnFdhWbZkhW09CFpwTnHhv0d/Hzdfh7beID2niilBWGumT2K902v4qKJZW850EYkU6ioJe30xxL87o0WfrW+iae3HqY3miA/HGTJ9CqunFnFe6dUUKTlEckgWqOWtBMOBbhiRhVXzKiiNxrnxV2tPLnlME9uOcSjGw6QEzQunFDGJVPKuWRKBVOrinTWPslYmlFLWoknHK/sPcpTrx3m2W3NvHG4C4CqEREuqavgvVMrOK+mhJqReSpuSSta+pCMdbDjOM+90cJzbxzh+e0tdPYOHFQzpiSPcCjAxZPKeP+c0Zw/roRISOvb4l8qaskKsXiCjU0dbG7q4NltLcQSjhd3ttIfTxAJBbhoUhnzJ5Qya3Qx88aWaI1bfEVFLVmrszfK2l1trN55hFWvN9PY2gNAwGDm6GLqKgsZkZdDJCfAiztbue2Sibx/zui3fJ32nn7CoQD5Yb2tI6mhohZJ6uiJsqmpg7WNbazd3cr+o8fp6IlyLHkeEoC6ykIW1ZVz8aRyRo3IJT8S5KN3riEnGOCWRROoHz+S1TuOsLeth3/64GzCIe3nLedORS1yGrF4gljC8dDL+/jt1sOs3d1GX+zNV10vL4xwpKvvTduKIiGumFnFf15/nt68lHOiohY5Q73ROK/ubWdvWzdbDx7jY++pZXr1CA539vJwwz5KCyIU5oZY8dxONjd1UjUiwuwxxUwoL2BsaT79cUdeTpDFdeXUluZ7PRxJAypqkRSJJxw/XtPIhv0drN/XTtPR4/TH3zwTr6ss5PLplcyrHcnkykJqRubpCEt5Cx3wIpIiwYDxZwsnnHx8vD9OU3sPbd1RivNyeGHHEZ55/TD3PL+bWGLXydeVFYSpLc1nbGk+48ryqS3Np2ZkHj19cfIjQerHlQ5q7XvrwU6eeu0wty2eSF5Y5Z+pNKMWGQbdfTF2NHexs6WLA+3HaWo/zt62Hva29XCgvZd44s3/DnOCRllBhNrSPGpL8ykvjFCSn0NpfpiS/DAj83No7e5n+UPr6Y8lqKss5OZFE7hmVjXF+drtMB1p6UPEx6LxBAfbe9nb1kNeOMjR7n7W7T1Ky7E+9rb1sL+th9bu/re8uQkwviyfz102mbue28WO5i7CwQBzaoqpLsmjujiXsoIwpQVhygrDlBZETj7ODwf15qfPqKhFMsDx/jhHe/pp6+6nvSdKbzTOwsnl5IWDOOfY3NTJI+ub2Hygg4MdvRzs6KX/bcodBs6lMiI3hxG5IQpzQxTlhiiMhCiM5FCcl0NJ/olbmKJIiIJIiIJIkILwwP3CSIjcnMCQl31vNJ616/daoxbJAHnhIHnhPEaX5L3lOTNjdk0xs2uKT25zztHdH6etq5/W7j7auvtp7R4o+qPd/XT2xjjWG6WrL0ZXb4wjx3o41hulszd28vqW7yZgnCzugkiQwkiI/OTjrr4o+eEQJfk5jMwPU5KXQ0lBmMJIkGAggHOO321roTcWZ3JlEbF4gg3723lxVxtXzazi+gtqqRwRoWpELqX5YQKB7J79a0YtIm8RjSdo74nS3tNPV1+M7r44XX0xevpjdPfF6OqLJz+e2BZPvm5gW0EkRG80TntPlKM9/fT0x9/yPQIGFUURDnf2YQZTKovICRnbD3e9aZknFDAqiyKUFoYJBwPkBAOEQwHCyY+5OUFycwJEQkFyc4LkJR+f2J4T/P+3cMgIBZKfHwoQSX6dnGCAUNAIBwOEggFygnbycwLGu/7l0B9LsKmpnae3NtPZG+WfPjj7rH7mmlGLyBnJCQaoKIpQUTQ0Fx7ui8Xp6InS3R8nnnDEE46qERFK8sPsaxs4rP/E/uZHu/vZ3dpNc2cvzcf6ONzZy+HOgb8IovEEfbEEXX0x+mMJ+mMJemNxeqMJeqNxeqNxovGhn3wGbGAPn4DZyY8ntvXFEvT0xwkYzKkpwTk35EtCKmoRSblIKEjliLdfe/7DA4JGFoQZWRA+6+8VT7iTpd0bSxCLDxR6NO6IxhNEk4/74gn6ogOPY4kE0ZgjmkgQjQ0cpdofH9gWd45E4pSPyfvODXwvh2NKVREXTSyjrqrorHO/m0EVtZktBb4DBIG7nXP/mpI0IiLnKBiw5Lp55sxDT7tHvZkFge8DVwMzgBvNbEaqg4mIyIDBnPZrPrDDObfLOdcPPAh8ILWxRETkhMEU9Rhg3ymP9ye3iYjIMBiyE+ma2TIzazCzhpaWlqH6siIiWW8wRd0E1J7yuCa57U2ccyucc/XOufqKioqhyicikvUGU9QvA3VmNsHMwsANwKOpjSUiIiecdv8V51zMzP4CeJKB3fN+6JzbkvJkIiICDHI/aufcSmBlirOIiMjbSMm5PsysBdhzlp9eDhwZwjjpQGPODhpzdjjbMY9zzr3tG3wpKepzYWYN73RikkylMWcHjTk7pGLMus69iIjPqahFRHzOj0W9wusAHtCYs4PGnB2GfMy+W6MWEZE38+OMWkRETqGiFhHxOd8UtZktNbNtZrbDzL7odZ6hYmY/NLNmM9t8yrZSM3vKzLYnP45Mbjcz+27yZ7DRzM73LvnZM7NaM1tlZq+Z2RYzuz25PWPHbWa5ZrbWzDYkx/zV5PYJZvZScmwPJU/DgJlFko93JJ8f7+kAzoGZBc3sVTN7PPk4o8dsZo1mtsnM1ptZQ3JbSn+3fVHUGX5xgvuApX+w7YvA0865OuDp5GMYGH9d8rYM+MEwZRxqMeBvnHMzgAXA55L/PTN53H3A5c6584C5wFIzWwD8G/At59xk4ChwS/L1twBHk9u/lXxdurod2HrK42wY82XOubmn7C+d2t9t55znN+Ai4MlTHt8B3OF1riEc33hg8ymPtwHVyfvVwLbk/buAG9/udel8Ax4BrsiWcQP5wCvAhQwcoRZKbj/5e87AuXMuSt4PJV9nXmc/i7HWJIvpcuBxwLJgzI1A+R9sS+nvti9m1GTfxQmqnHMHk/cPAVXJ+xn3c0j+eTsPeIkMH3dyCWA90Aw8BewE2p1zseRLTh3XyTEnn+8AyoY18ND4NvC3QCL5uIzMH7MDfmNm68xsWXJbSn+3M+fqj2nKOefMLCP3kTSzQuDnwHLnXKeZnXwuE8ftnIsDc82sBPglMM3bRKllZu8Hmp1z68zsUo/jDKdFzrkmM6sEnjKz1099MhW/236ZUQ/q4gQZ5LCZVQMkPzYnt2fMz8HMchgo6Z86536R3Jzx4wZwzrUDqxj4s7/EzE5MiE4d18kxJ58vBlqHN+k5Wwj8kZk1MnAt1cuB75DZY8Y515T82MzA/5Dnk+Lfbb8UdbZdnOBR4JPJ+59kYA33xPY/Tb5TvADoOOXPqbRhA1Pne4CtzrlvnvJUxo7bzCqSM2nMLI+BNfmtDBT2R5Iv+8Mxn/hZfAR4xiUXMdOFc+4O51yNc248A/9mn3HOfZwMHrOZFZhZ0Yn7wJXAZlL9u+31wvwpi+zXAG8wsK73Ja/zDOG4HgAOAlEG1qduYWBd7mlgO/BboDT5WmNg75edwCag3uv8ZznmRQys420E1idv12TyuIE5wKvJMW8GvpzcPhFYC+wAHgYiye25ycc7ks9P9HoM5zj+S4HHM33MybFtSN62nOiqVP9u6xByERGf88vSh4iIvAMVtYiIz6moRUR8TkUtIuJzKmoREZ9TUYuI+JyKWkTE5/4f9z3w8Fqt7N8AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline\n",
    "plt.plot(hist.history['loss'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Laurence went to dublin his pipes bellows chanters a fainted and tea replied and ground ground ground ground terrible hullabaloo ned call ask ned ask ask ned call ask ned arrived mcgilligan so drop so too so catchers daughter hall again again taras catchers daughter old hall hall didnt ned hall hall odaly nice didnt saw away weeks weeks new steps for lanigans ball all mad were all might catchers mad didnt i suppose might ask ned ask ask mcgilligan ask nice didnt milliner ned call hall ask relations ground ground ground groups saw didnt swore ned banished wall wall nonsense academy i catchers\n"
     ]
    }
   ],
   "source": [
    "seed_txt = \"Laurence went to dublin\"\n",
    "next_words = 100\n",
    "for i in range(next_words):\n",
    "    token_list = tokenizer.texts_to_sequences([seed_txt])[0]\n",
    "    token_list = pad_sequences([token_list],maxlen = max_sequence_len-1,padding='pre')\n",
    "    predicted = model.predict_classes(token_list)\n",
    "    output_word = \"\"\n",
    "    for word, index in tokenizer.word_index.items():\n",
    "        if index == predicted :\n",
    "            output_word = word\n",
    "            break\n",
    "    seed_txt = seed_txt + \" \" +output_word\n",
    "print(seed_txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Laurence went to dublin round athy one jeremy lanigan eyes glisten glisten glisten glisten glisten glisten glisten put nice gray me them the a jig jig jig jig able polkas her a your glisten glisten glisten glisten glisten died me them of gray and the wall man all the girls they got a call call ask glisten glisten died me them me a polkas polkas polkas polkas red a call water leg eyes gray me creature up fainted gray gray relations relations relations a cask man and father cask cask glisten glisten glisten glisten died me them of the wall jig cask polkas red\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "'wget' is not recognized as an internal or external command,\n",
      "operable program or batch file.\n"
     ]
    }
   ],
   "source": [
    "!wget --no-check-certificate \\\n",
    "    https://storage.googleapis.com/laurencemoroney-blog.appspot.com/irish-lyrics-eof.txt \\\n",
    "    -O /tmp/irish-lyrics-eof.txt\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}