11. seq2seq model (translate).ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** keras example **\n",
    "\n",
    "download data in http://www.manythings.org/anki/\n",
    "\n",
    "i use fra-eng.zip file\n",
    "\n",
    "문자 단위로 처리하고 출력하는 모델을 만듭니다.\n",
    "\n",
    "진행 과정은 아래와 같이 진행됩니다.\n",
    "\n",
    "1. 문장들을 3차원 배열(encoder_input, decoder_input, decoder_target)으로 변환합니다.\n",
    "    - encoder_input은 (num_pairs, max_english_sentence_length, num_elglish_character) 형태의 3차원 배열로 영어 문장의 one-hot 형식 벡터 데이터를 갖고 있습니다.\n",
    "    \n",
    "    - decoder_input은 (num_pairs, max_french_sentece_length, num_french_character) 형태의 3차원 배열로 불어 문장의 one-hot 형식입니다.\n",
    "    - decoder_target은 decoder_input과 같지만 하나의 time step 만큼 offset 됩니다. decoer_target[:, t, :]는 decoder_input[:, t+1, :]과 같습니다.\n",
    "    \n",
    "2. 기본 LSTM 기반의 seq2seq model을 주어진 encoder_input과 decoder_input로 decoder_target을 예측합니다.\n",
    "3. model이 작동하는지 확인하기 위해 일부 문장을 디코딩합니다.(encdoer_input의 샘플을 decoder_target의 표본으로 변환합니다)\n",
    "\n",
    "\n",
    "문장을 디코딩 하는 학습 단계와 추론 단계는 좀 다릅니다. 같은 내부 계층을 사용하지만 서로 다른 모델을 사용하죠.\n",
    "\n",
    "- return_state : encoder의 출력과 내부 RNN 상태 반환\n",
    "- initial_state : decoder의 초기 상태를 지정\n",
    "- return_sequences : 전체 시퀀스를 반환"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import random\n",
    "from keras.models import Model\n",
    "from keras.layers import Input, LSTM, Dense\n",
    "from keras.callbacks import EarlyStopping, ModelCheckpoint"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [],
   "source": [
    "batch_size = 64\n",
    "epochs = 60\n",
    "latent_dim = 256\n",
    "num_samples = 10000"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_texts, target_texts = [], []\n",
    "input_characters = set()\n",
    "target_characters = set()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('fra.txt', 'r', encoding = 'utf-8') as f:\n",
    "    lines = f.read().split('\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Go.\\tVa !', 'Hi.\\tSalut !', 'Run!\\tCours\\u202f!']"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lines[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이렇게 \\t을 기준으로 번역 상태가 들어가있다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Go. ,    Va !\n",
      "Hi. ,    Salut !\n",
      "Run! ,    Cours !\n",
      "Run! ,    Courez !\n",
      "Wow! ,    Ça alors !\n",
      "Fire! ,    Au feu !\n",
      "Help! ,    À l'aide !\n",
      "Jump. ,    Saute.\n",
      "Stop! ,    Ça suffit !\n",
      "Stop! ,    Stop !\n"
     ]
    }
   ],
   "source": [
    "for cnt, line in enumerate(lines[: min(num_samples, len(lines) - 1)]):\n",
    "    \n",
    "    input_text, target_text = line.split('\\t')\n",
    "    if cnt < 10: print(input_text, \",   \", target_text)\n",
    "    # \\t 문자를 시작문자로, \\n 문자를 종료 문자로 사용한다.\n",
    "    target_text = '\\t' + target_text + '\\n'\n",
    "    input_texts.append(input_text)\n",
    "    target_texts.append(target_text)\n",
    "    for char in input_text:\n",
    "        if char not in input_characters:\n",
    "            input_characters.add(char)\n",
    "    for char in target_text:\n",
    "        if char not in target_characters:\n",
    "            target_characters.add(char)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Go.', 'Hi.', 'Run!', 'Run!', 'Wow!']"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "input_texts[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['\\tVa !\\n',\n",
       " '\\tSalut !\\n',\n",
       " '\\tCours\\u202f!\\n',\n",
       " '\\tCourez\\u202f!\\n',\n",
       " '\\tÇa alors\\u202f!\\n']"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "target_texts[:5]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['E', '3', 'T', '.', 'N', 's', 'V', 't', '$', 'G']\n",
      "['è', '1', '»', 'P', 'e', 'd', ')', 'G', 'à', 'c']\n"
     ]
    }
   ],
   "source": [
    "print(random.sample(input_characters, 10))\n",
    "print(random.sample(target_characters, 10))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_characters = sorted(list(input_characters))\n",
    "target_characters = sorted(list(target_characters))\n",
    "num_encoder_tokens = len(input_characters)\n",
    "num_decoder_tokens = len(target_characters)\n",
    "max_encoder_seq_length = max([len(txt) for txt in input_texts])\n",
    "max_decoder_seq_length = max([len(txt) for txt in target_texts])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[' ', '!', '$', '%', '&']\n",
      "['\\t', '\\n', ' ', '!', '$']\n",
      "Number of samples: 10000\n",
      "Number of unique input tokens: 69\n",
      "Number of unique output tokens: 93\n",
      "Max sequence length for inputs: 16\n",
      "Max sequence length for outputs: 59\n"
     ]
    }
   ],
   "source": [
    "print(input_characters[:5])\n",
    "print(target_characters[:5])\n",
    "print('Number of samples:', len(input_texts))\n",
    "print('Number of unique input tokens:', num_encoder_tokens)\n",
    "print('Number of unique output tokens:', num_decoder_tokens)\n",
    "print('Max sequence length for inputs:', max_encoder_seq_length)\n",
    "print('Max sequence length for outputs:', max_decoder_seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [],
   "source": [
    "input_token_index = dict([(char, i) for i, char in enumerate(input_characters)])\n",
    "target_token_index = dict([(char, i) for i, char in enumerate(target_characters)])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "   ,  0\n",
      "!  ,  1\n",
      "$  ,  2\n",
      "%  ,  3\n",
      "&  ,  4\n",
      "'  ,  5\n",
      ",  ,  6\n",
      "-  ,  7\n",
      ".  ,  8\n",
      "0  ,  9\n"
     ]
    }
   ],
   "source": [
    "for key, value in input_token_index.items():\n",
    "    if value < 10 : print( key , \" , \", value)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_input_data = np.zeros((len(input_texts), max_encoder_seq_length, num_encoder_tokens),dtype='float32')\n",
    "decoder_input_data = np.zeros((len(input_texts), max_decoder_seq_length, num_decoder_tokens),dtype='float32')\n",
    "decoder_target_data = np.zeros((len(input_texts), max_decoder_seq_length, num_decoder_tokens),dtype='float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_input_data = np.zeros((len(input_texts), max_encoder_seq_length, num_encoder_tokens),dtype='float32')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "(10000, 16, 69)\n",
      "(10000, 59, 93)\n",
      "(10000, 59, 93)\n"
     ]
    }
   ],
   "source": [
    "print(np.shape(encoder_input_data))\n",
    "print(np.shape(decoder_input_data))\n",
    "print(np.shape(decoder_target_data))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이제 one-hot 형식의 데이터로 만들어준다"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0 0   //    char :     G  //  char index :  25\n",
      "0 1   //    char :     o  //  char index :  57\n",
      "0 2   //    char :     .  //  char index :  8\n",
      "[[0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " ...\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]\n",
      " [0. 0. 0. ... 0. 0. 0.]]\n",
      "[0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n",
      "[0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.\n",
      " 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]\n"
     ]
    }
   ],
   "source": [
    "for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):\n",
    "    for t, char in enumerate(input_text):\n",
    "        print(i, t, \"  //    char :    \", char, \" //  char index : \", input_token_index[char])\n",
    "        test_input_data[i, t, input_token_index[char]] = 1.\n",
    "    print(test_input_data[0])\n",
    "    print(test_input_data[0][1])\n",
    "    print(test_input_data[0][2])\n",
    "    break"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "이런식으로 one-hot을 만들어준다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i, (input_text, target_text) in enumerate(zip(input_texts, target_texts)):\n",
    "    for t, char in enumerate(input_text):\n",
    "        encoder_input_data[i, t, input_token_index[char]] = 1.\n",
    "    for t, char in enumerate(target_text):\n",
    "        decoder_input_data[i, t, target_token_index[char]] = 1.\n",
    "        \n",
    "        if t > 0:\n",
    "            decoder_target_data[i, t-1, target_token_index[char]] = 1."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "모델을 만들어준다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_inputs = Input(shape=(None, num_encoder_tokens))\n",
    "encoder = LSTM(latent_dim, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_inputs)\n",
    "# 출력 상태 벡터만 가져온다.\n",
    "encoder_states = [state_h, state_c]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_inputs = Input(shape=(None, num_decoder_tokens))\n",
    "\n",
    "decoder_lstm = LSTM(latent_dim, return_sequences=True, return_state=True)\n",
    "decoder_outputs, _, _ = decoder_lstm(decoder_inputs, initial_state=encoder_states)\n",
    "decoder_dense = Dense(num_decoder_tokens, activation='softmax')\n",
    "decoder_outputs = decoder_dense(decoder_outputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_5 (InputLayer)            (None, None, 69)     0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_6 (InputLayer)            (None, None, 93)     0                                            \n",
      "__________________________________________________________________________________________________\n",
      "lstm_5 (LSTM)                   [(None, 256), (None, 333824      input_5[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "lstm_6 (LSTM)                   [(None, None, 256),  358400      input_6[0][0]                    \n",
      "                                                                 lstm_5[0][1]                     \n",
      "                                                                 lstm_5[0][2]                     \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, None, 93)     23901       lstm_6[0][0]                     \n",
      "==================================================================================================\n",
      "Total params: 716,125\n",
      "Trainable params: 716,125\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "model = Model([encoder_inputs, decoder_inputs], decoder_outputs)\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "model.compile(loss = 'categorical_crossentropy', optimizer = 'adam')\n",
    "#checkpoint = ModelCheckpoint(file_path, monitor = 'val_loss', verbose = 1, save_best_only=True)\n",
    "early_stop = EarlyStopping(monitor = 'val_loss', patience = 3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 8000 samples, validate on 2000 samples\n",
      "Epoch 1/60\n",
      "8000/8000 [==============================] - 12s 2ms/step - loss: 0.9644 - val_loss: 0.9829\n",
      "Epoch 2/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.7654 - val_loss: 0.7964\n",
      "Epoch 3/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.6376 - val_loss: 0.7094\n",
      "Epoch 4/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.5791 - val_loss: 0.6614\n",
      "Epoch 5/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.5442 - val_loss: 0.6277\n",
      "Epoch 6/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.5187 - val_loss: 0.6041\n",
      "Epoch 7/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.4954 - val_loss: 0.5912\n",
      "Epoch 8/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.4775 - val_loss: 0.5650\n",
      "Epoch 9/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.4611 - val_loss: 0.5534\n",
      "Epoch 10/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.4465 - val_loss: 0.5418\n",
      "Epoch 11/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.4325 - val_loss: 0.5314\n",
      "Epoch 12/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.4208 - val_loss: 0.5193\n",
      "Epoch 13/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.4117 - val_loss: 0.5090\n",
      "Epoch 14/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3979 - val_loss: 0.4999\n",
      "Epoch 15/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3870 - val_loss: 0.4940\n",
      "Epoch 16/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3778 - val_loss: 0.4823\n",
      "Epoch 17/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3688 - val_loss: 0.4770\n",
      "Epoch 18/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3598 - val_loss: 0.4708\n",
      "Epoch 19/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3508 - val_loss: 0.4675\n",
      "Epoch 20/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3424 - val_loss: 0.4603\n",
      "Epoch 21/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3346 - val_loss: 0.4601\n",
      "Epoch 22/60\n",
      "8000/8000 [==============================] - 12s 1ms/step - loss: 0.3273 - val_loss: 0.4609\n",
      "Epoch 23/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3189 - val_loss: 0.4512\n",
      "Epoch 24/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3111 - val_loss: 0.4490\n",
      "Epoch 25/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.3039 - val_loss: 0.4476\n",
      "Epoch 26/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.2975 - val_loss: 0.4448\n",
      "Epoch 27/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.2909 - val_loss: 0.4453\n",
      "Epoch 28/60\n",
      "8000/8000 [==============================] - 12s 1ms/step - loss: 0.2841 - val_loss: 0.4404\n",
      "Epoch 29/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.2767 - val_loss: 0.4417\n",
      "Epoch 30/60\n",
      "8000/8000 [==============================] - 12s 1ms/step - loss: 0.2718 - val_loss: 0.4444\n",
      "Epoch 31/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.2647 - val_loss: 0.4396\n",
      "Epoch 32/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.2596 - val_loss: 0.4451\n",
      "Epoch 33/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.2526 - val_loss: 0.4451\n",
      "Epoch 34/60\n",
      "8000/8000 [==============================] - 11s 1ms/step - loss: 0.2471 - val_loss: 0.4421\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<keras.callbacks.History at 0x23616633208>"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "model.fit([encoder_input_data, decoder_input_data], decoder_target_data, batch_size=batch_size, epochs=epochs, validation_split=0.2, callbacks = [early_stop])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "추론 모델 생성\n",
    "\n",
    "- 입력문장을 encode하고 초기 상태의 decoder의 상태를 가지고 옵니다.\n",
    "- 초기 상태 decoder의 한 단계와 \"시퀀스 시작\" 토큰을 목표로 실행합니다. 출력은 다음 목표 문자입니다.\n",
    "- 예측된 목표 문자를 붙이고 반복합니다."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "_________________________________________________________________\n",
      "Layer (type)                 Output Shape              Param #   \n",
      "=================================================================\n",
      "input_5 (InputLayer)         (None, None, 69)          0         \n",
      "_________________________________________________________________\n",
      "lstm_5 (LSTM)                [(None, 256), (None, 256) 333824    \n",
      "=================================================================\n",
      "Total params: 333,824\n",
      "Trainable params: 333,824\n",
      "Non-trainable params: 0\n",
      "_________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "encoder_model = Model(encoder_inputs, encoder_states)\n",
    "encoder_model.summary()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "__________________________________________________________________________________________________\n",
      "Layer (type)                    Output Shape         Param #     Connected to                     \n",
      "==================================================================================================\n",
      "input_6 (InputLayer)            (None, None, 93)     0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_7 (InputLayer)            (None, 256)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "input_8 (InputLayer)            (None, 256)          0                                            \n",
      "__________________________________________________________________________________________________\n",
      "lstm_6 (LSTM)                   [(None, None, 256),  358400      input_6[0][0]                    \n",
      "                                                                 input_7[0][0]                    \n",
      "                                                                 input_8[0][0]                    \n",
      "__________________________________________________________________________________________________\n",
      "dense_3 (Dense)                 (None, None, 93)     23901       lstm_6[1][0]                     \n",
      "==================================================================================================\n",
      "Total params: 382,301\n",
      "Trainable params: 382,301\n",
      "Non-trainable params: 0\n",
      "__________________________________________________________________________________________________\n"
     ]
    }
   ],
   "source": [
    "decoder_state_input_h = Input(shape = (latent_dim, ))\n",
    "decoder_state_input_c = Input(shape = (latent_dim, ))\n",
    "decoder_states_inputs = [decoder_state_input_h, decoder_state_input_c]\n",
    "decoder_outputs, state_h, state_c = decoder_lstm(decoder_inputs, initial_state = decoder_states_inputs)\n",
    "decoder_states = [state_h, state_c]\n",
    "decoder_outputs = decoder_dense(decoder_outputs)\n",
    "decoder_model = Model([decoder_inputs] + decoder_states_inputs, [decoder_outputs] + decoder_states)\n",
    "decoder_model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "위에서와 다르게 이번에는 idx : char 구조로 변환\n",
    "\n",
    "즉, 이것은 숫자 -> 문자 변환용"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [],
   "source": [
    "reverse_input_char_index = dict((i, char) for char, i in input_token_index.items())\n",
    "reverse_target_char_index = dict((i, char) for char, i in target_token_index.items())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0  ,   \n",
      "1  ,  !\n",
      "2  ,  $\n",
      "3  ,  %\n",
      "4  ,  &\n",
      "5  ,  '\n",
      "6  ,  ,\n",
      "7  ,  -\n",
      "8  ,  .\n",
      "9  ,  0\n"
     ]
    }
   ],
   "source": [
    "for key, value in reverse_input_char_index.items():\n",
    "    if key < 10 : print( key , \" , \", value)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "추론할 때 사용하는 함수 구현"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {},
   "outputs": [],
   "source": [
    "def decode_sequence(input_seq):\n",
    "    \n",
    "    #아까 만들어졌던 one-hot 값이 들어온다.\n",
    "    #print(\"input_seq : \", input_seq , \"\\n\")\n",
    "    \n",
    "    #입력 문장을 인코딩\n",
    "    states_value = encoder_model.predict(input_seq)\n",
    "    \n",
    "    #디코더의 입력으로 쓸 단일 문자\n",
    "    target_seq = np.zeros((1, 1, num_decoder_tokens))\n",
    "    # (1, 1, 93) 모양을 가진다.\n",
    "    #print(\"np.shape(target_seq) : \", np.shape(target_seq), \"\\n\")\n",
    "    \n",
    "    #처음 입력은 시작 문자였던 \\t 로 설정한다.\n",
    "    target_seq[0, 0, target_token_index['\\t']] = 1.\n",
    "    \n",
    "    stop_condition = False\n",
    "    decoded_sentence = ''\n",
    "    while not stop_condition:\n",
    "        #이전의 출력, 상태를 디코더에 넣어 새로운 출력, 상태를 얻음\n",
    "        # 이전 문자와 상태 데이터를 가지고 다음 문장과 상태를 얻는 것\n",
    "        \n",
    "        output_tokens, h, c = decoder_model.predict([target_seq] + states_value)\n",
    "        \n",
    "        #output으로 나온 벡터행렬\n",
    "        #print(\"output_token : \", output_tokens, \"\\n\")\n",
    "        \n",
    "        #h도 마찬가지 벡터 행렬\n",
    "        #print(\"h : \", h, \"\\n\")\n",
    "        \n",
    "        # 사전을 이용해 one-hot을 실제 문자로 변환\n",
    "        sampled_token_index = np.argmax(output_tokens[0, -1, :])\n",
    "        \n",
    "        #예를들어 42 이런식으로 인덱스가 나온다.\n",
    "        #print(\"sampled_token_index : \", sampled_token_index, \"\\n\")\n",
    "        sampled_char = reverse_target_char_index[sampled_token_index]\n",
    "        \n",
    "        # 인덱스가 이제 실제 값으로 바뀐다. 42 -> V\n",
    "        #print(\"sampled_char : \", sampled_char, \"\\n\")\n",
    "        \n",
    "        decoded_sentence += sampled_char\n",
    "        \n",
    "        #종료 문자나 문장 길이가 초과되면 종료\n",
    "        if (sampled_char == '\\n' or len(decoded_sentence) > max_decoder_seq_length):\n",
    "            stop_condition = True\n",
    "            \n",
    "        # 디코더의 다음 입력으로 쓸 데이터\n",
    "        target_seq = np.zeros((1, 1, num_decoder_tokens))\n",
    "        target_seq[0, 0, sampled_token_index] = 1.\n",
    "        \n",
    "        #아까 위에서 나온 그 인덱스 값(42)가 이제 다음에 들어가야 하니까 저렇게 둔다.\n",
    "        #print(\"target_seq[0, 0, sampled_token_index] : \", target_seq[0, 0, sampled_token_index], \"\\n\")\n",
    "        #상태값 변경\n",
    "        states_value = [h, c]\n",
    "    return decoded_sentence"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "-\n",
      "Input sentence: Go.\n",
      "Decoded sentence: Va te chercher !\n",
      "\n",
      "-\n",
      "Input sentence: Hi.\n",
      "Decoded sentence: Attends une bien.\n",
      "\n",
      "-\n",
      "Input sentence: Run!\n",
      "Decoded sentence: Prends le commences !\n",
      "\n",
      "-\n",
      "Input sentence: Run!\n",
      "Decoded sentence: Prends le commences !\n",
      "\n",
      "-\n",
      "Input sentence: Wow!\n",
      "Decoded sentence: Comme c'est chaut !\n",
      "\n",
      "-\n",
      "Input sentence: Fire!\n",
      "Decoded sentence: Assieds-toi !\n",
      "\n",
      "-\n",
      "Input sentence: Help!\n",
      "Decoded sentence: Aidez-vous !\n",
      "\n",
      "-\n",
      "Input sentence: Jump.\n",
      "Decoded sentence: Pas un peux maintenant.\n",
      "\n",
      "-\n",
      "Input sentence: Stop!\n",
      "Decoded sentence: Arrête de te plaîter !\n",
      "\n",
      "-\n",
      "Input sentence: Stop!\n",
      "Decoded sentence: Arrête de te plaîter !\n",
      "\n"
     ]
    }
   ],
   "source": [
    "for seq_index in range(10):\n",
    "    #아까 만들어졌던 one-hot 값이 들어간다\n",
    "    input_seq = encoder_input_data[seq_index: seq_index + 1]\n",
    "    decoded_sentence = decode_sequence(input_seq)\n",
    "    print('-')\n",
    "    print('Input sentence:', input_texts[seq_index])\n",
    "    print('Decoded sentence:', decoded_sentence)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}