06-自然语言处理/26-Attention使用示例.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 目录\n",
    "1. [EncoderDecoder Network](#EncoderDecoder-Network)\n",
    "2. [Attention机制](#Attention机制)\n",
    "3. [西班牙语-英语翻译示例](#西班牙语-英语翻译示例)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [EncoderDecoder Network](#目录)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encoder-Decoder Network，如下图所示\n",
    "- 英文句子输入到 encoder 中，decoder 输出法文句子。\n",
    "- 英文句子是逆序提供给 encoder 的：“I drink milk”--->“milk drink I”，保证句首最后提供给 encoder，因为 decoder 是最先翻译句首\n",
    "- 目标法文句子也是 decoder 的输入，但后移一步；句首为 `<sos>` (start of sequence)标志，句尾为 `<eos>` 标志。\n",
    "- decoder 每一步先输出词汇表中所有单词的 score，然后 softmax 层转化为概率，概率最高的单词即为结果；sparse_categorical_crossentropy 损失函数\n",
    "- 推理阶段，没有目标句子，decoder 每一步的输入是其上一步的输出，初始为 `<sos>` 标志"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"../images/neural_machine_translation.jpg\" width=\"80%\" alt=\"简单机器翻译模型\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- encoder 的输入序列长度不定，decoder 的输入序列也如此：\n",
    "    - 输入序列的长度差别不是很大，可使用 masking\n",
    "    - 否则，将序列分组进 buckets，如一个 bucket 用于1-6个单词的句子、另一 bucket 用于7-12单词的句子；然后使用 padding，如“I drink milk”--->“`<pad> <pad> <pad> milk drink I`”；tf.data.experimental.bucket_by_sequence_length() 函数\n",
    "- 忽略输出序列 `<eos>` 标志后的内容\n",
    "- 当输出的词汇表太大时，计算其中所有单词的概率太耗费计算资源；可以仅仅选择正确单词即随机采样的错误单词，tf.nn.sampled_softmax 函数；但仅仅在训练时，如此操作，推理阶段应是正常的 softmax 函数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import tensorflow as tf\n",
    "from tensorflow import keras\n",
    "# import tensorflow_addons as tfa"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "decoder_inputs = keras.layers.Input(shape=[None], dtype=np.int32)\n",
    "sequence_lengths = keras.layers.Input(shape=[], dtype=np.int32)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab_size, embed_size = 10000, 200"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "embeddings = keras.layers.Embedding(vocab_size, embed_size)\n",
    "encoder_embeddings = embeddings(encoder_inputs)\n",
    "decoder_embeddings = embeddings(decoder_inputs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "encoder = keras.layers.LSTM(512, return_state=True)\n",
    "encoder_outputs, state_h, state_c = encoder(encoder_embeddings)\n",
    "encoder_state = [state_h, state_c]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sampler = tfa.seq2seq.sampler.TrainingSampler()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "decoder_cell = keras.layers.LSTMCell(512)\n",
    "output_layer = keras.layers.Dense(vocab_size)\n",
    "decoder = tf1.seq2seq.basic_decoder.BasicDecoder(decoder_cell, sampler, output_layer = output_layer)\n",
    "final_outputs, final_state, final_sequence_lengths = decoder(decoder_embeddings, \n",
    "                                                             initial_state=encoder_state,\n",
    "                                                            sequence_lengths=sequence_lengths)\n",
    "Y_proba = tf.nn.softmax(final_outputs.rnn_output)\n",
    "\n",
    "model = keras.Model(inputs=[encoder_inputs, decoder_inputs, sequence_lengths],\n",
    "                   outputs=[Y_proba])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- 双向 RNN\n",
    "- Beam Search"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [Attention机制](#目录)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](../images/attention_mechanism.jpg)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Attention 计算流程\n",
    "1. 给定 Encoder 的隐藏状态：$h_1, h_2, ... h_N \\in R^{d_1}$\n",
    "2. 在 Decoder 的时间 t，其隐藏状态为 $s_t \\in R^{d_2}$\n",
    "3. 求该时间步的 attention scores:\n",
    "$e^t = [s_t^Th_1, s_t^Th_2, ..., s_t^Th_N]$\n",
    "4. 求得 attention 分布：$\\alpha^t=softmax(e^t)$\n",
    "5. attention 输出为：$a_t = \\sum_{i=1}^{N}\\alpha^t_ih_i$\n",
    "6. 连接 $s_t$ 和 $a_t$：$[a_t;s_t]$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`tf.keras.layers.Attention`：\n",
    "- 输入：\n",
    "    - `query, shape=[batch_size, Tq, dim]`;\n",
    "    - `value, shape=[batch_size, Tv, dim]`;\n",
    "    - `key, shape=[batch_size, Tv, dim]`\n",
    "- 计算过程：\n",
    "    - `scores = tf.matmul(query, key, transpose_b=True), shape=[batch_size, Tq, Tv]`\n",
    "    - `distribution = tf.nn.softmax(scores)`\n",
    "    - `return tf.matmul(distribution, value), shape=[batch_size, Tq, dim]`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Attention+CNN"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Variable-length int sequences.\n",
    "query_input = tf.keras.Input(shape=(None,), dtype='int32')\n",
    "value_input = tf.keras.Input(shape=(None,), dtype='int32')\n",
    "\n",
    "# Embedding lookup.\n",
    "token_embedding = tf.keras.layers.Embedding(max_tokens, dimension)\n",
    "# Query embeddings of shape [batch_size, Tq, dimension].\n",
    "query_embeddings = token_embedding(query_input)\n",
    "# Value embeddings of shape [batch_size, Tv, dimension].\n",
    "value_embeddings = token_embedding(query_input)\n",
    "\n",
    "# CNN layer.\n",
    "cnn_layer = tf.keras.layers.Conv1D(\n",
    "    filters=100,\n",
    "    kernel_size=4,\n",
    "    # Use 'same' padding so outputs have the same shape as inputs.\n",
    "    padding='same')\n",
    "# Query encoding of shape [batch_size, Tq, filters].\n",
    "query_seq_encoding = cnn_layer(query_embeddings)\n",
    "# Value encoding of shape [batch_size, Tv, filters].\n",
    "value_seq_encoding = cnn_layer(value_embeddings)\n",
    "\n",
    "# Query-value attention of shape [batch_size, Tq, filters].\n",
    "query_value_attention_seq = tf.keras.layers.Attention()(\n",
    "    [query_seq_encoding, value_seq_encoding])\n",
    "\n",
    "# Reduce over the sequence axis to produce encodings of shape\n",
    "# [batch_size, filters].\n",
    "query_encoding = tf.keras.layers.GlobalAveragePooling1D()(\n",
    "    query_seq_encoding)\n",
    "query_value_attention = tf.keras.layers.GlobalAveragePooling1D()(\n",
    "    query_value_attention_seq)\n",
    "\n",
    "# Concatenate query and document encodings to produce a DNN input layer.\n",
    "input_layer = tf.keras.layers.Concatenate()(\n",
    "    [query_encoding, query_value_attention])\n",
    "\n",
    "# Add DNN layers, and create Model.\n",
    "# ...\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# [西班牙语-英语翻译示例](#目录)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow as tf\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.ticker as ticker\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "import unicodedata\n",
    "import re\n",
    "import numpy as np\n",
    "import os\n",
    "import io\n",
    "import time"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1、数据集预处理\n",
    "1. 每句话添加 start 及 end 标志\n",
    "2. 删除特殊字符\n",
    "3. word2id 及 id2word\n",
    "4. 每句话 pad 为最大长度"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "path_to_zip = tf.keras.utils.get_file(\n",
    "    'spa-eng.zip', origin='http://storage.googleapis.com/download.tensorflow.org/data/spa-eng.zip',\n",
    "    extract=True)\n",
    "\n",
    "path_to_file = os.path.dirname(path_to_zip)+\"/spa-eng/spa.txt\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Converts the unicode file to ascii\n",
    "def unicode_to_ascii(s):\n",
    "    return ''.join(c for c in unicodedata.normalize('NFD', s)\n",
    "        if unicodedata.category(c) != 'Mn')\n",
    "\n",
    "\n",
    "def preprocess_sentence(w):\n",
    "    w = unicode_to_ascii(w.lower().strip())\n",
    "\n",
    "    # creating a space between a word and the punctuation following it\n",
    "    # eg: \"he is a boy.\" => \"he is a boy .\"\n",
    "    # Reference:- https://stackoverflow.com/questions/3645931/python-padding-punctuation-with-white-spaces-keeping-punctuation\n",
    "    w = re.sub(r\"([?.!,¿])\", r\" \\1 \", w)\n",
    "    w = re.sub(r'[\" \"]+', \" \", w)\n",
    "\n",
    "    # replacing everything with space except (a-z, A-Z, \".\", \"?\", \"!\", \",\")\n",
    "    w = re.sub(r\"[^a-zA-Z?.!,¿]+\", \" \", w)\n",
    "\n",
    "    w = w.rstrip().strip()\n",
    "\n",
    "    # adding a start and an end token to the sentence\n",
    "    # so that the model know when to start and stop predicting.\n",
    "    w = '<start> ' + w + ' <end>'\n",
    "    return w"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<start> may i borrow this book ? <end>\n",
      "b'<start> \\xc2\\xbf puedo tomar prestado este libro ? <end>'\n"
     ]
    }
   ],
   "source": [
    "en_sentence = u\"May I borrow this book?\"\n",
    "sp_sentence = u\"¿Puedo tomar prestado este libro?\"\n",
    "print(preprocess_sentence(en_sentence))\n",
    "print(preprocess_sentence(sp_sentence).encode('utf-8'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# 1. Remove the accents\n",
    "# 2. Clean the sentences\n",
    "# 3. Return word pairs in the format: [ENGLISH, SPANISH]\n",
    "def create_dataset(path, num_examples):\n",
    "    lines = io.open(path, encoding='UTF-8').read().strip().split('\\n')\n",
    "\n",
    "    word_pairs = [[preprocess_sentence(w) for w in l.split('\\t')]  for l in lines[:num_examples]]\n",
    "\n",
    "    return zip(*word_pairs)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<start> if you want to sound like a native speaker , you must be willing to practice saying the same sentence over and over in the same way that banjo players practice the same phrase over and over until they can play it correctly and at the desired tempo . <end>\n",
      "<start> si quieres sonar como un hablante nativo , debes estar dispuesto a practicar diciendo la misma frase una y otra vez de la misma manera en que un musico de banjo practica el mismo fraseo una y otra vez hasta que lo puedan tocar correctamente y en el tiempo esperado . <end>\n"
     ]
    }
   ],
   "source": [
    "en, sp = create_dataset(path_to_file, None)\n",
    "print(en[-1])\n",
    "print(sp[-1])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(118964, 118964)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(en),len(sp)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "def max_length(tensor):\n",
    "    return max(len(t) for t in tensor)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def tokenize(lang):\n",
    "    lang_tokenizer = tf.keras.preprocessing.text.Tokenizer(\n",
    "      filters='')\n",
    "    lang_tokenizer.fit_on_texts(lang)\n",
    "\n",
    "    tensor = lang_tokenizer.texts_to_sequences(lang)\n",
    "\n",
    "    tensor = tf.keras.preprocessing.sequence.pad_sequences(tensor,\n",
    "                                                         padding='post')\n",
    "\n",
    "    return tensor, lang_tokenizer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def load_dataset(path, num_examples=None):\n",
    "    # creating cleaned input, output pairs\n",
    "    targ_lang, inp_lang = create_dataset(path, num_examples)\n",
    "\n",
    "    input_tensor, inp_lang_tokenizer = tokenize(inp_lang)\n",
    "    target_tensor, targ_lang_tokenizer = tokenize(targ_lang)\n",
    "\n",
    "    return input_tensor, target_tensor, inp_lang_tokenizer, targ_lang_tokenizer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 选择数据集中的 30000 句作为示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Try experimenting with the size of that dataset\n",
    "num_examples = 30000\n",
    "input_tensor, target_tensor, inp_lang, targ_lang = load_dataset(path_to_file, num_examples)\n",
    "\n",
    "# Calculate max_length of the target tensors\n",
    "max_length_targ, max_length_inp = max_length(target_tensor), max_length(input_tensor)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "24000 24000 6000 6000\n"
     ]
    }
   ],
   "source": [
    "# Creating training and validation sets using an 80-20 split\n",
    "input_tensor_train, input_tensor_val, target_tensor_train, target_tensor_val = train_test_split(input_tensor, target_tensor, test_size=0.2)\n",
    "\n",
    "# Show length\n",
    "print(len(input_tensor_train), len(target_tensor_train), len(input_tensor_val), len(target_tensor_val))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "def convert(lang, tensor):\n",
    "    for t in tensor:\n",
    "        if t!=0:\n",
    "            print (\"%d ----> %s\" % (t, lang.index_word[t]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input Language; index to word mapping\n",
      "1 ----> <start>\n",
      "6 ----> ¿\n",
      "60 ----> donde\n",
      "88 ----> hay\n",
      "15 ----> un\n",
      "1166 ----> hospital\n",
      "5 ----> ?\n",
      "2 ----> <end>\n",
      "\n",
      "Target Language; index to word mapping\n",
      "1 ----> <start>\n",
      "71 ----> where\n",
      "8 ----> is\n",
      "9 ----> a\n",
      "1007 ----> hospital\n",
      "7 ----> ?\n",
      "2 ----> <end>\n"
     ]
    }
   ],
   "source": [
    "print (\"Input Language; index to word mapping\")\n",
    "convert(inp_lang, input_tensor_train[0])\n",
    "print ()\n",
    "print (\"Target Language; index to word mapping\")\n",
    "convert(targ_lang, target_tensor_train[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### tf.data 对象"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "BUFFER_SIZE = len(input_tensor_train)\n",
    "BATCH_SIZE = 64\n",
    "steps_per_epoch = len(input_tensor_train)//BATCH_SIZE\n",
    "embedding_dim = 256\n",
    "units = 1024\n",
    "vocab_inp_size = len(inp_lang.word_index)+1\n",
    "vocab_tar_size = len(targ_lang.word_index)+1\n",
    "\n",
    "dataset = tf.data.Dataset.from_tensor_slices((input_tensor_train, target_tensor_train)).shuffle(BUFFER_SIZE)\n",
    "dataset = dataset.batch(BATCH_SIZE, drop_remainder=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(TensorShape([64, 16]), TensorShape([64, 11]))"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example_input_batch, example_target_batch = next(iter(dataset))\n",
    "example_input_batch.shape, example_target_batch.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2、创建 encoder-decoder 模型\n",
    "约定：\n",
    "- FC = Fully connected (dense) layer\n",
    "- EO = Encoder output\n",
    "- H = hidden state\n",
    "- X = input to the decoder\n",
    "       \n",
    "伪代码：\n",
    "- `score = FC(tanh(FC(EO) + FC(H)))`\n",
    "- `attention weights = softmax(score, axis = 1)`\n",
    "- `context vector = sum(attention weights * EO, axis = 1)`\n",
    "- `merged vector = concat(embedding output, context vector)`\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](../images/attention_mechanism.jpg)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Encoder(tf.keras.Model):\n",
    "    def __init__(self, vocab_size, embedding_dim, enc_units, batch_sz):\n",
    "        super(Encoder, self).__init__()\n",
    "        self.batch_sz = batch_sz\n",
    "        self.enc_units = enc_units\n",
    "        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
    "        self.gru = tf.keras.layers.GRU(self.enc_units, return_sequences=True,\n",
    "                                       return_state=True, \n",
    "                                       recurrent_initializer='glorot_uniform')\n",
    "    def call(self, x, hidden):\n",
    "        x = self.embedding(x)\n",
    "        output, state = self.gru(x, initial_state = hidden)\n",
    "        return output, state\n",
    "    \n",
    "    def initialize_hidden_state(self):\n",
    "        return tf.zeros((self.batch_sz, self.enc_units))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Encoder output shape: (batch size, sequence length, units) (64, 16, 1024)\n",
      "Encoder Hidden state shape: (batch size, units) (64, 1024)\n"
     ]
    }
   ],
   "source": [
    "encoder = Encoder(vocab_inp_size, embedding_dim, units, BATCH_SIZE)\n",
    "\n",
    "# sample input\n",
    "sample_hidden = encoder.initialize_hidden_state()\n",
    "sample_output, sample_hidden = encoder(example_input_batch, sample_hidden)\n",
    "print ('Encoder output shape: (batch size, sequence length, units) {}'.format(sample_output.shape))\n",
    "print ('Encoder Hidden state shape: (batch size, units) {}'.format(sample_hidden.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "class BahdanauAttention(tf.keras.layers.Layer):\n",
    "    def __init__(self, units):\n",
    "        super(BahdanauAttention, self).__init__()\n",
    "        self.W1 = tf.keras.layers.Dense(units)\n",
    "        self.W2 = tf.keras.layers.Dense(units)\n",
    "        self.V = tf.keras.layers.Dense(1)\n",
    "        \n",
    "    def call(self, query, values):\n",
    "        # hidden shape == (batch_size, hidden size)\n",
    "        # hidden_with_time_axis shape == (batch_size, 1, hidden size)\n",
    "        # we are doing this to perform addition to calculate the score\n",
    "        hidden_with_time_axis = tf.expand_dims(query, 1)\n",
    "        \n",
    "         # score shape == (batch_size, max_length, 1)\n",
    "        # we get 1 at the last axis because we are applying score to self.V\n",
    "        # the shape of the tensor before applying self.V is (batch_size, max_length, units)\n",
    "        score = self.V(tf.nn.tanh(\n",
    "            self.W1(values) + self.W2(hidden_with_time_axis)))\n",
    "        \n",
    "        # attention_weights shape == (batch_size, max_length, 1)\n",
    "        attention_weights = tf.nn.softmax(score, axis=1)\n",
    "\n",
    "         # context_vector shape after sum == (batch_size, hidden_size)\n",
    "        context_vector = attention_weights * values\n",
    "        context_vector = tf.reduce_sum(context_vector, axis=1)\n",
    "\n",
    "        return context_vector, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Attention result shape: (batch size, units) (64, 1024)\n",
      "Attention weights shape: (batch_size, sequence_length, 1) (64, 16, 1)\n"
     ]
    }
   ],
   "source": [
    "attention_layer = BahdanauAttention(10)\n",
    "attention_result, attention_weights = attention_layer(sample_hidden, sample_output)\n",
    "\n",
    "print(\"Attention result shape: (batch size, units) {}\".format(attention_result.shape))\n",
    "print(\"Attention weights shape: (batch_size, sequence_length, 1) {}\".format(attention_weights.shape))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Decoder(tf.keras.Model):\n",
    "    def __init__(self, vocab_size, embedding_dim, dec_units, batch_sz):\n",
    "        super(Decoder, self).__init__()\n",
    "        self.batch_sz = batch_sz\n",
    "        self.dec_units = dec_units\n",
    "        self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
    "        self.gru = tf.keras.layers.GRU(self.dec_units,\n",
    "                                       return_sequences=True,\n",
    "                                       return_state=True,\n",
    "                                       recurrent_initializer='glorot_uniform')\n",
    "        self.fc = tf.keras.layers.Dense(vocab_size)\n",
    "\n",
    "        # used for attention\n",
    "        self.attention = BahdanauAttention(self.dec_units)\n",
    "        \n",
    "    def call(self, x, hidden, enc_output):\n",
    "        # enc_output shape == (batch_size, max_length, hidden_size)\n",
    "        context_vector, attention_weights = self.attention(hidden, enc_output)\n",
    "\n",
    "        # x shape after passing through embedding == (batch_size, 1, embedding_dim)\n",
    "        x = self.embedding(x)\n",
    "\n",
    "        # x shape after concatenation == (batch_size, 1, embedding_dim + hidden_size)\n",
    "        x = tf.concat([tf.expand_dims(context_vector, 1), x], axis=-1)\n",
    "\n",
    "        # passing the concatenated vector to the GRU\n",
    "        output, state = self.gru(x)\n",
    "\n",
    "        # output shape == (batch_size * 1, hidden_size)\n",
    "        output = tf.reshape(output, (-1, output.shape[2]))\n",
    "\n",
    "        # output shape == (batch_size, vocab)\n",
    "        x = self.fc(output)\n",
    "\n",
    "        return x, state, attention_weights"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Decoder output shape: (batch_size, vocab size) (64, 4935)\n"
     ]
    }
   ],
   "source": [
    "decoder = Decoder(vocab_tar_size, embedding_dim, units, BATCH_SIZE)\n",
    "\n",
    "sample_decoder_output, _, _ = decoder(tf.random.uniform((64, 1)),\n",
    "                                      sample_hidden, sample_output)\n",
    "\n",
    "print ('Decoder output shape: (batch_size, vocab size) {}'.format(sample_decoder_output.shape))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3、训练模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "optimizer = tf.keras.optimizers.Adam()\n",
    "loss_object = tf.keras.losses.SparseCategoricalCrossentropy(\n",
    "    from_logits=True, reduction='none')\n",
    "\n",
    "def loss_function(real, pred):\n",
    "    mask = tf.math.logical_not(tf.math.equal(real, 0))\n",
    "    loss_ = loss_object(real, pred)\n",
    "\n",
    "    mask = tf.cast(mask, dtype=loss_.dtype)\n",
    "    loss_ *= mask\n",
    "\n",
    "    return tf.reduce_mean(loss_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "checkpoint_dir = './training_checkpoints'\n",
    "checkpoint_prefix = os.path.join(checkpoint_dir, \"ckpt\")\n",
    "checkpoint = tf.train.Checkpoint(optimizer=optimizer,\n",
    "                                 encoder=encoder,\n",
    "                                 decoder=decoder)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "@tf.function\n",
    "def train_step(inp, targ, enc_hidden):\n",
    "    loss = 0\n",
    "\n",
    "    with tf.GradientTape() as tape:\n",
    "        enc_output, enc_hidden = encoder(inp, enc_hidden)\n",
    "\n",
    "        dec_hidden = enc_hidden\n",
    "\n",
    "        dec_input = tf.expand_dims([targ_lang.word_index['<start>']] * BATCH_SIZE, 1)\n",
    "\n",
    "        # Teacher forcing - feeding the target as the next input\n",
    "        for t in range(1, targ.shape[1]):\n",
    "            # passing enc_output to the decoder\n",
    "            predictions, dec_hidden, _ = decoder(dec_input, dec_hidden, enc_output)\n",
    "\n",
    "            loss += loss_function(targ[:, t], predictions)\n",
    "\n",
    "            # using teacher forcing\n",
    "            dec_input = tf.expand_dims(targ[:, t], 1)\n",
    "\n",
    "    batch_loss = (loss / int(targ.shape[1]))\n",
    "\n",
    "    variables = encoder.trainable_variables + decoder.trainable_variables\n",
    "\n",
    "    gradients = tape.gradient(loss, variables)\n",
    "\n",
    "    optimizer.apply_gradients(zip(gradients, variables))\n",
    "\n",
    "    return batch_loss"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Epoch 1 Loss 2.0338\n",
      "Time taken for 1 epoch 1002.6916968822479 sec\n",
      "\n",
      "Epoch 2 Loss 1.3765\n",
      "Time taken for 1 epoch 900.9487726688385 sec\n",
      "\n",
      "Epoch 3 Loss 0.9524\n",
      "Time taken for 1 epoch 869.3331334590912 sec\n",
      "\n",
      "Epoch 4 Loss 0.6402\n",
      "Time taken for 1 epoch 971.220751285553 sec\n",
      "\n",
      "Epoch 5 Loss 0.4374\n",
      "Time taken for 1 epoch 939.8397359848022 sec\n",
      "\n",
      "Epoch 6 Loss 0.3069\n",
      "Time taken for 1 epoch 794.041389465332 sec\n",
      "\n",
      "Epoch 7 Loss 0.2187\n",
      "Time taken for 1 epoch 743.7652299404144 sec\n",
      "\n",
      "Epoch 8 Loss 0.1648\n",
      "Time taken for 1 epoch 748.7127416133881 sec\n",
      "\n",
      "Epoch 9 Loss 0.1285\n",
      "Time taken for 1 epoch 745.3105812072754 sec\n",
      "\n",
      "Epoch 10 Loss 0.1058\n",
      "Time taken for 1 epoch 746.6278686523438 sec\n",
      "\n"
     ]
    }
   ],
   "source": [
    "EPOCHS = 10\n",
    "\n",
    "for epoch in range(EPOCHS):\n",
    "    start = time.time()\n",
    "\n",
    "    enc_hidden = encoder.initialize_hidden_state()\n",
    "    total_loss = 0\n",
    "\n",
    "    for (batch, (inp, targ)) in enumerate(dataset.take(steps_per_epoch)):\n",
    "        batch_loss = train_step(inp, targ, enc_hidden)\n",
    "        total_loss += batch_loss\n",
    "\n",
    "    if batch % 100 == 0:\n",
    "        print('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1,\n",
    "                                                     batch,\n",
    "                                                     batch_loss.numpy()))\n",
    "    # saving (checkpoint) the model every 2 epochs\n",
    "    if (epoch + 1) % 2 == 0:\n",
    "        checkpoint.save(file_prefix = checkpoint_prefix)\n",
    "\n",
    "    print('Epoch {} Loss {:.4f}'.format(epoch + 1,\n",
    "                                      total_loss / steps_per_epoch))\n",
    "    print('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4、评估"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "def evaluate(sentence):\n",
    "    attention_plot = np.zeros((max_length_targ, max_length_inp))\n",
    "\n",
    "    sentence = preprocess_sentence(sentence)\n",
    "\n",
    "    inputs = [inp_lang.word_index[i] for i in sentence.split(' ')]\n",
    "    inputs = tf.keras.preprocessing.sequence.pad_sequences([inputs],\n",
    "                                                           maxlen=max_length_inp,\n",
    "                                                           padding='post')\n",
    "    inputs = tf.convert_to_tensor(inputs)\n",
    "\n",
    "    result = ''\n",
    "\n",
    "    hidden = [tf.zeros((1, units))]\n",
    "    enc_out, enc_hidden = encoder(inputs, hidden)\n",
    "\n",
    "    dec_hidden = enc_hidden\n",
    "    dec_input = tf.expand_dims([targ_lang.word_index['<start>']], 0)\n",
    "\n",
    "    for t in range(max_length_targ):\n",
    "        predictions, dec_hidden, attention_weights = decoder(dec_input,\n",
    "                                                             dec_hidden,\n",
    "                                                             enc_out)\n",
    "\n",
    "        # storing the attention weights to plot later on\n",
    "        attention_weights = tf.reshape(attention_weights, (-1, ))\n",
    "        attention_plot[t] = attention_weights.numpy()\n",
    "\n",
    "        predicted_id = tf.argmax(predictions[0]).numpy()\n",
    "\n",
    "        result += targ_lang.index_word[predicted_id] + ' '\n",
    "\n",
    "        if targ_lang.index_word[predicted_id] == '<end>':\n",
    "            return result, sentence, attention_plot\n",
    "\n",
    "        # the predicted ID is fed back into the model\n",
    "        dec_input = tf.expand_dims([predicted_id], 0)\n",
    "\n",
    "    return result, sentence, attention_plot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [],
   "source": [
    "# function for plotting the attention weights\n",
    "def plot_attention(attention, sentence, predicted_sentence):\n",
    "    fig = plt.figure(figsize=(10,10))\n",
    "    ax = fig.add_subplot(1, 1, 1)\n",
    "    ax.matshow(attention, cmap='viridis')\n",
    "\n",
    "    fontdict = {'fontsize': 14}\n",
    "\n",
    "    ax.set_xticklabels([''] + sentence, fontdict=fontdict, rotation=90)\n",
    "    ax.set_yticklabels([''] + predicted_sentence, fontdict=fontdict)\n",
    "\n",
    "    ax.xaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "    ax.yaxis.set_major_locator(ticker.MultipleLocator(1))\n",
    "\n",
    "    plt.show()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "def translate(sentence):\n",
    "    result, sentence, attention_plot = evaluate(sentence)\n",
    "\n",
    "    print('Input: %s' % (sentence))\n",
    "    print('Predicted translation: {}'.format(result))\n",
    "\n",
    "    attention_plot = attention_plot[:len(result.split(' ')), :len(sentence.split(' '))]\n",
    "    plot_attention(attention_plot, sentence.split(' '), result.split(' '))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<tensorflow.python.training.tracking.util.CheckpointLoadStatus at 0x2452fc18088>"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# restoring the latest checkpoint in checkpoint_dir\n",
    "checkpoint.restore(tf.train.latest_checkpoint(checkpoint_dir))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: <start> hace mucho frio aqui . <end>\n",
      "Predicted translation: it s very cold here . <end> \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAiIAAAJwCAYAAAC08grWAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3deZzlB1nn+++TdBYSNgEFXBAUkH1JWpRlBAbHOOB+3TAgiJegwhUUN2TUyNyAICowuBAXGDZH4MIg6LDIYlRADKiAAUIMhJ0QiZAA2Z/7x++0qS6qQxI69Zzuer9fr369qn7n1Kmnfun0+dRvre4OAMCEQ6YHAAB2LiECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRImugqm5VVa+vqjtOzwIA20mIrIeHJLlPkocNzwEA26rc9G5WVVWSDyR5bZLvTPKV3X3p6FAAsE1sEZl33yTXSfLTSS5Jcv/ZcQBg+wiReT+a5CXd/bkkf5plNw0A7Ah2zQyqqqOTfCzJA7r7b6rqLknenGX3zLmz0wHANc8WkVn/V5JzuvtvkqS7/ynJ+5L88OhUABzwquroqvrRqrre9CxXRIjMenCS529a9vzYPQPAl+4Hkzw7y3vN2rJrZkhVfU2S9ye5bXe/b8Pyr85yFs3tuvv0ofFYA1V1pyQ/l+R2STrJaUme2t3vHB0MOCBU1RuTfEWSz3X37uFx9kmIwBqqqu9K8tIkf5Pkb1eL77X6833d/Yqp2YD1V1U3T3J6krsleUuSY7r7tMmZ9kWIDKqqmyX5UG/xH6GqbtbdHxwYizVQVe9I8rLu/rVNy5+Q5Lu7+84zkwEHgqr6lST36e77VdVLk7yvu39xeq6tOEZk1vuTfPnmhVV1w9Vj7Fy3TvK8LZY/L8k3bPMswIHnR3P5vyHPT3L86gKaa0eIzKos+/43u3aSC7Z5FtbL2UmO3WL5sUk+sc2zAAeQqrpHkpsmefFq0SuTHJXkW8eGugK7pgfYiarqGasPO8mTqupzGx4+NMs+vX/a9sFYJ3+Y5FlVdcskb8ryd+VeWQ5e/c3JwYC195AkL+/uzyZJd19UVS9K8tAstxNZK44RGVBVb1h9eO8sFzC7aMPDF2U5a+apG8+mYWdZbUJ9TJLHJvnK1eKPZomQZ2x1XBFAVR2R5ONJHtjdr9qw/F5JXp3kxt19/tR8WxEiQ1ZvNC9K8rDuPm96HtZXVV0nSfw9Ab6YqrpRlnuWPb+7L9v02IOS/FV3f3xkuH0QIkOq6tAsx4HceV1PqQKAa5pjRIZ096VVdVaSw6dnYf1U1Q2SnJTkflkuSLTXgeXdfd2JuQD2NyEy678n+Y2qelB3nzM9DGvlj5PcNcnJWY4NsekS2Keqen+u5L8T3f111/A4V4ldM4Oq6p1JbpHksCQfTvLZjY93950m5mJeVX0myX/p7r+fngVYf1X12A2fXjvJzyZ5a5YTIpLk7lnOyPyt7n7CNo93hWwRmfWS6QFYW2cnWasj24H11d2/tefjqnpOkid39xM3PqeqHpfk9ts82hdliwisoar6oSx3znzIup1qB6y31RbVY7r7jE3Lb5nk7et2jJktIqyNqvqpJI/MsrvqDt19ZlX9UpIzu/tFs9Nd81a76jb+ZnCLJGevDmq+eONz7bYDrsBnk9wnyRmblt8nyec2P3maEBlUVYcneXySBya5WZZjRf5Ddx86MdeEqnpMkl9I8uQkv7HhoY8keVSWa64c7OyqA/aH30nyu1W1O8udd5Pkm7NccfXEqaH2xa6ZQVX15CQ/lORJWf7i/LckN0/yw0l+pbufNTfd9qqq9yR5bHf/RVWdl+X6KmdW1e2TnNLdNxweEUZV1TFJ/qm7L1t9vE/d/fZtGos1VVU/mOTRSW67WvTuJE9fx63LQmTQ6nSrn+zuV63efO/S3f9aVT+Z5H7d/f3DI26bqvp8ktt091mbQuTWWf7xPWp4xG1VVfdOku7+6y2Wd3efMjIYY6rqsiQ36e6zVx93lhtnbtY7aWsqBz67ZmbdOMmeq6qen+T6q49flWUXxU5yZpJjkpy1afn9c/k62kl+J8lWp9hdN8um1a3uzMvB7RZJPrnhY/iiqur6+cILIn5qaJwtCZFZH8xyQ7MPZjmo6Lgkb8tyvvfnB+ea8NQkz6yqo7L8lnf3qnpwluNGHjY62YxvSPLPWyx/5+oxdpjuPmurj2GzqvraJH+Q5L7Z+9jDyrIlba22mAmRWS/LcgnvtyR5epI/raqHJ/mq7LBbvXf3s6tqV5InJjkqyfOyHKj60939Z6PDzfh8lkh9/6blX52979bMDuQYEb6IZ2fZwv6wHABXZnaMyBqpqm9Kcs8kp3f3K6fnmbK6e+Qh3X329CxTquoFWc6k+q7uPne17AZJ/neSj3T3AyfnY9Y+jhH5j3/MHSOys1XV+Um+ubvfNT3LlSFEBlXVtyR5U3dfsmn5riT32EkHJK7Ojjm0u9+xafmdklyy0+5QXFU3TXJKlhve7Vknd8pyxdV7d/dHp2Zj3mrT+0aHZbk30eOTPK67/8/2T8W6WF2T6KHd/bbpWa4MITKoqi5NctPNv/lX1Q2TnL2Tfqupqr9L8rvd/cJNy384yaO6+14zk81ZHS9zfJK7ZPnN9+1JXtjda3dBou1QVf85ye2y/OZ/Wne/YXiktVNV35bk17r7ntOzMGf1/8ovJfmpzVdXXUdCZNBq8+qNu/uTm5bfOsmp63YZ3mvS6pTdu25xSeKvz3JJ4uvNTMa0qvqqLMdTHZtlf3eyHD9zapLvtXXoclV1qyynux89PQtzVv+eHpHloNQLk+y11X3d3lscrDqgqv589WEneX5VXbjh4UOT3CHJm7Z9sFmXJtkqNr4sW18r4aBWVd93RY9390u3a5Y18Iwsfz9u2d3vT5Kq+rokz189tmOut7PH6nihvRYluWmWU7vfu+0DsW4eNT3AVWGLyICqevbqw4dkuXT5xlN1L0rygSR/2N3nbPNoY6rq5VnebH6guy9dLduV5MVJDuvu75icb7uttpZtpZOddTDi6gZe99l8Jsjq8tWv24lbyzYcrLrX4iQfSvJD3f2WL/wqWE+2iAzo7h9Lkqr6QJKndvdnZydaC7+Q5G+TnFFVf7tadq8k107yLWNTDenuvS5AtIqyu2Y5rfvxI0Otn33F2k5w302fX5blYmdnbD74nZ2pqm6c5MFJvj7LLUPOqap7Jvnoni2L68IWkUFVdUiSdPdlq89vkuQ7shyIt9N2zew5U+RR2fvgzN9zDMDlquoeSX6/u+88Pct2qaqXJfnyJA/s7g+tlt0syQuSfLK7r3A3Fuw0VXVsktdluQ7R7bPcPuPMqjoxya27+0cm59tMiAyqqv+T5FXd/fSqunaS9yQ5OstWgB/v7ueODsjaqarbJXlrd197epbtUlVfk+TlSe6Yyy/O9FVZTmv+7u7+8OB4I1an/l8pO+kyACyq6g1Zbhb6a5vu3XX3JP+ruzef/j3KrplZx2bZJZEk35fkM1nuIXF8kp9LsuNCpKq+MsuFvA7fuHyn/WO6xZUz9xyM+ItJ/nH7J5qz2gpyTFX9lyS3ybIuTuvuv5qdbNQbc/kxInsO5t78+Z5lO+Z4Iv7DsUl+fIvlH8tyj7O1IkRmXSfJv68+/rYkL+vui6vq9Ul+d26s7bcKkBdmOR5kzxUjN26u22n/mJ6are+u+pbszHvvpLtfm+S103Osie/Icn+mk5K8ebXs7kl+OcsvNw5W3dk+n+WMw81uk+WiiGtFiMz6YJJ7VtUrstzw7gdWy2+QZKddtOppWc6auV2Sf0jy7VnK/QlJfmZwrimb7656WZbjIS6YGGa7VdXPZjk+6ILVx/vU3b+9TWOtk/+e5NGrONvjzKo6O8lTuvuuQ3OxHl6e5Neqas97SlfVzbPc1f3/mxpqXxwjMqiqHpHkmUnOT3JWkmO6+7Kq+ukk39Pd/3l0wG1UVZ9I8oDuPnV1uubu7j69qh6Q5Yjvbx4ecdutDl6+R5bLvG++jffvjQy1Tarq/Vn+Dvzb6uN96e7+uu2aa11U1eez/Hvx7k3Lb5fkbd19rZnJWAdVdd0kf5nlthBHJ/l4ll/s3pTkv67bmZpCZNjq6OabJXltd5+/WvaAJP/e3X83Otw2WsXHnbr7A6vTmh/U3X9bVbdI8i/dfdTshNurqh6U5I+y7Jo5N3vvpuru/sqRwVgLVXVqkjOS/Fh3f3617FpZ7rp6y+7ePTkf62F1qfdjsvwi8/Z1Pa7KrpkhVXW9LG+8f5Nk842J/j3JjrrJW5Yzhm6T5WJu/5TkJ6rqQ0kemeQjg3NNOSnJU5I8YSdfF6KqDstyfZkf7W5XDL3cTyZ5ZZKPVNWemyLeMcvuzQeMTcW4je8t3f36JK/f8Ng9sxzofe7YgFuwRWRIVV0nyxHMx23c8lFVd0ny90m+aoddWfX4LFdQfc7qjJFXJblRlvskPKS7XzQ64DarqnOTHNvdZ07PMm113MO9uvv06VnWyYabIt42qzOJstwUca02u7O9DsT3FiEyqKpekOT87n7EhmVPzXLBme+am2ze6h/Z2yT54Lr9T7MdquqZSd7b3f9jepZpVfWbSdLdPz89yzpZXW33btn6dPcdd+o/lzvQ3luEyKCqOi7Jn2a5A+/FqyutfjjLbe930k3NkiRV9UNJ7petD85cu/95rklVdXiS/53l3kPvTHLxxse7+wkTc02oqt/L8pv/+7PsxtzrN/7u/umJuSZV1W2SvCLL2VWVZZfMrix/Ty5ct7ursr0OtPcWx4jMem2W03S/M8lLs7wJH57lH5gdZfVb72OSvCGXXz1zJ3tEllOYz0lyy2w6WDXLac0HrdWVQ9+0Oj7mtlku958km8+Q2al/T56WJcrukuWMiLtkuXv17yf5b4NzsR4OqPcWW0SGVdWTk3xDd39PVT03yXnd/cjpubbb6vTdR3b3S6ZnWQer4yKe1N2/Mz3LhKq6NMlNu/vsqjozyTd2979Nz7Uuqurfkty7u99VVZ9Ocrfufm9V3TvJ/+juOw2PyLAD6b3FFpF5z03yttX9NL43S7nuRIdkOVuGxaFJ/nx6iEHnZtntcHaSm2fTrjpSufyih5/Mcu+d92bZ/H7LqaFYKwfMe4stImugqv4hyQVJbtTdt52eZ0JVnZTk4u4+cXqWdbA6sOwzO+lYkI2q6llJHpLl6P+bZXmDvXSr5+7QC5qdkuR3uvtlVfXCJDdM8sQkD89y6qYtIhww7y22iKyH52XZ5/v46UG2U1U9Y8OnhyQ5fnVjs3fkCw/O3GkHJB6V5P9eHXS2E9fHT2TZInSrJL+d5UJd541OtF5OynLFzGQ5JuSVWY6vOifJD04NtW6q6t1JbtXdO/W97oB4b9mp/3HWzfOz3KDo2dODbLM7bvp8z66Z22xavhM32902l99ld8etj1421f5FklTVnZP8VncLkZXufvWGj89McruqukGSc9tm7o1+N8vWop3qgHhvsWsGABjjADAAYIwQAQDGCJE1UVUnTM+wTqyPvVkfe7M+9mZ97M362Nu6rw8hsj7W+i/KAOtjb9bH3qyPvVkfe7M+9rbW60OIAABjdvxZM4fXEX3kf5yOP+fiXJjDcsT0GGvD+tib9bE362Nv1sfe1mV91K5Dp0dIklx02QU5/JAjp8fIZy4555zu/vLNy3f8dUSOzNH5plrbK98C6+yQ9XijWRt92fQEa+XQ63/Z9Ahr5dXnnHzWVsvtmgEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxhwUIVJVz6mqV07PAQBcNbumB9hPHp2kkqSq3pjkXd39qNGJAIAv6qAIke7+9PQMAMBVd1CESFU9J8mNkpyT5N5J7l1Vj1w9fIvu/sDQaADAFTgoQmSDRye5dZL3JPnl1bJPzo0DAFyRgypEuvvTVXVRks9198f39byqOiHJCUlyZI7arvEAgE0OirNmrqruPrm7d3f37sNyxPQ4ALBj7cgQAQDWw8EYIhclOXR6CADgizsYQ+QDSe5WVTevqhtV1cH4MwLAQeFgfJN+apatIqdlOWPmZrPjAAD7clCcNdPdD93w8elJ7j43DQBwZR2MW0QAgAOEEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxggRAGCMEAEAxuyaHmDcUddK3eGO01Osjes/7SPTI6yVj/3GLadHWCtH/817p0dYK33RxdMjrJXLLrhweoS1cum/fWp6hAOCLSIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMESIAwBghAgCMOehCpKq+pareUlXnV9Wnq+rvq+oO03MBAF9o1/QA+1NV7Ury8iR/nOT4JIclOSbJpZNzAQBbO6hCJMl1k1w/ySu6+19Xy96z+UlVdUKSE5LkyMOvt33TAQB7Oah2zXT3p5I8J8mrq+ovqupnq+prtnjeyd29u7t3H7br6G2fEwBYHFQhkiTd/WNJvinJKUm+K8npVXXc7FQAwFYOuhBJku7+5+5+cnffJ8kbkzxkdiIAYCsHVYhU1S2q6jeq6h5V9bVVdd8kd0py2vRsAMAXOtgOVv1cklsneXGSGyX5RJIXJHny5FAAwNYOqhDp7k8k+b7pOQCAK+eg2jUDABxYhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMEaIAABjhAgAMGbX9ADT6qKLcsiZH5keY2185qHXnx5hrVzrWf5ubPSp6952eoS1coNXnT49wnq54MLpCTgA2SICAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIw54EOkqg6fngEAuHq2NUSq6hFV9Ymq2rVp+Qur6uWrj7+zqt5WVRdU1fur6qSNsVFVH6iqE6vqT6rq35O8oKpeX1XP3PSa162qz1XV923LDwcAXGXbvUXkRUmun+Rb9yyoqqOTfHeS51fVcUlekOSZSW6f5GFJvj/JEze9zs8meU+S3Ul+OckfJvmRqjpiw3MemOT8JK+4Rn4SAOBLtq0h0t3nJvnLJMdvWPy9SS7JEgyPT/Kb3f3s7v7X7n5Dkl9M8hNVVRu+5q+7+yndfUZ3vy/JS5NctnqtPR6W5LndffHmOarqhKo6tapOveiyC/brzwgAXHkTx4g8P8n3VNVRq8+PT/KS7r4gybFJHl9V5+/5k+SFSY5OcpMNr3Hqxhfs7guTPC9LfKSqbpfkbkn+ZKsBuvvk7t7d3bsPP+TI/fijAQBXxa4v/pT97pVZtoB8d1W9Lstumm9bPXZIkl9P8uItvu6TGz7+7BaP/1GSd1TVzZL8eJI3d/dp+21qAGC/2/YQ6e4Lq+olWbaE3CjJx5P89erhtye5TXefcTVe91+q6u+TPDzJg7Ls5gEA1tjEFpFk2T3zV0lukeSF3X3ZavkTkryyqs7KcmDrJUnukORu3f0LV+J1/zDJHyS5OMmf7fepAYD9auo6Iqck+UiS22WJkiRJd786yQOS3DfJW1d/finJB6/k6/5ZkouSvKi7z9ufAwMA+9/IFpHu7iQ338djr0nymiv42i2/buX6Sa6V5I+/hPEAgG0ytWtmv6qqw5LcNMlJSf6xu/9ueCQA4Eo44C/xvnLPJGcl+aYsB6sCAAeAg2KLSHe/MUl9secBAOvlYNkiAgAcgIQIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY4QIADBGiAAAY3ZNDzCtL7k0l/7bp6bHWB/WxV4OfeStpkdYK7/7l8+YHmGt/PL7fnx6hLVyyDs+Nz3CWrnsgkunRzgg2CICAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIwRIgDAGCECAIw5IEOkqk6sqnd9kec8s6reuE0jAQBXwwEZIgDAwUGIAABjxkKkFo+tqvdV1YVV9eGqetLqsTtW1V9V1eer6lNV9Zyqut4VvNahVfXUqjp39edpSQ7dth8GALhaJreIPDHJryR5UpLbJ/mBJB+qqqOSvCrJ+UnuluR7k9wjyZ9cwWs9NsnDkzwiyd2zRMjx19jkAMB+sWvim1bVtZP8TJLHdPeewDgjyZur6uFJrp3kwd193ur5JyR5Q1XdsrvP2OIlH5PkKd39otXzH53kuCv4/ickOSFJjsxR++mnAgCuqqktIrdLckSS123x2G2TvGNPhKy8Kcllq6/by2qXzU2TvHnPsu6+LMnf7+ubd/fJ3b27u3cfliOu3k8AAHzJpkKkvshjvY/H9rUcADgATYXIaUkuTHK/fTx256q6zoZl98gy67s3P7m7P53kY0m+ec+yqqosx5cAAGts5BiR7j6vqp6e5ElVdWGSU5LcMMmxSf5nkl9P8tyq+tUkX5bkWUleuo/jQ5Lk6UkeV1WnJ3lnkp/KsrvmY9fsTwIAfClGQmTlcUnOzXLmzFcn+USS53b356rquCRPS/LWJBckeXmSR1/Ba/1Wkpsk+aPV589L8oIsx5sAAGtqLERWB5T+xurP5sfema132+x5/MQkJ274/JIsZ+H8zP6eEwC45riyKgAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGN2TQ8A6+zSd79veoS18iu7v316hLXy6nc+b3qEtXLc9zx4eoT1cupp0xOsl0u3XmyLCAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwRogAAGOECAAwZttCpKreWFXP3K7vBwCsP1tEAIAxB3SIVNVh0zMAAFffdofIIVX1xKo6p6rOrqqnVtUhSVJVh1fVk6vqw1X12ar6h6o6bs8XVtV9qqqr6v5V9daquijJcavHvrOq3lZVF1TV+6vqpKo6fJt/NgDgKtq1zd/v+CRPT3KPJHdJ8sIkb0vyp0meneTrk/xIkg8nuX+SV1TVN3b3P294jScneWySM5Kct4qVFyR5dJJTktwsyR8kOSLJz201RFWdkOSEJDkyR+3fnxAAuNK2O0RO6+5fXX18elU9PMn9quqtSR6Y5Obd/cHV48+sqm9N8ogkP7XhNU7s7tfs+aSqHp/kN7v72atF/1pVv5jk+VX1893dm4fo7pOTnJwk160bfMHjAMD22O4Qecemzz+a5CuSHJOkkpxWVRsfPyLJ6zd9zambPj82yd1W8bHHIUmuleQmST72Jc4MAFxDtjtELt70eWeJhkNWH3/jFs/5/KbPP7vp80OS/HqSF2/x/T559cYEALbDdofIvvxjli0iN+nuN1zFr317ktt09xn7fywA4Jq0FiHS3adX1QuSPKeqHpslLm6Q5D5Jzuzul17Blz8hySur6qwkL0pySZI7JLlbd//CNTs5APClWKfriPxYljNnnpLkPUlemeRbkpx1RV/U3a9O8oAk903y1tWfX0rywSv6OgBg3rZtEenu+2yx7KEbPr44yYmrP1t9/Ruz7L7Z6rHXJHnNVo8BAOtrnbaIAAA7jBABAMYIEQBgjBABAMYIEQBgjBABAMYIEQBgjBABAMYIEQBgjBABAMYIEQBgjBABAMYIEQBgjBABAMYIEQBgjBABAMYIEQBgjBABAMYIEQBgjBABAMYIEQBgjBABAMYIEQBgjBABAMYIEQBgjBABAMbsmh4AOIDc6AbTE6yVez76EdMjrJVv/P23TY+wVk7/tutNj7Beztl6sS0iAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMCYXdMDTKiqE5KckCRH5qjhaQBg59qRW0S6++Tu3t3duw/LEdPjAMCOtSNDBABYD0IEABgjRACAMQdtiFTVo6rqPdNzAAD7dtCGSJIbJfmG6SEAgH07aEOku0/s7pqeAwDYt4M2RACA9SdEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGAGwVXYAAAajSURBVCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxQgQAGCNEAIAxu6YHAA4cl57+r9MjrJXrfOBD0yOslZv+6qenR1grbzz+btMjrJenb73YFhEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYMwBEyJV9XNV9YHpOQCA/eeACREA4OCzX0Kkqq5bVdffH691Fb7nl1fVkdv5PQGA/etqh0hVHVpVx1XVC5N8PMmdV8uvV1UnV9XZVXVeVf11Ve3e8HUPrarzq+p+VfWuqvpsVb2hqm6x6fV/oao+vnruc5Nce9MI90/y8dX3uufV/TkAgDlXOUSq6vZV9ZQkH0zyZ0k+m+Tbk5xSVZXkL5J8VZLvSHLXJKckeX1V3XTDyxyR5HFJHpbk7kmun+QPNnyPH0zy/yb5tSTHJHlvkp/dNMoLkvxIkuskeW1VnVFVv7o5aPbxM5xQVadW1akX58KrugoAgP3kSoVIVd2wqn66qk5N8o9JbpPkMUlu3N0P7+5TuruT3DfJXZJ8f3e/tbvP6O5fSXJmkgdveMldSR65es47kjw1yX2ras88j0nyP7v7Wd19eneflOStG2fq7ku6+y+7+4FJbpzkiavv/77VVpiHVdXmrSh7vvbk7t7d3bsPyxFXZhUAANeAK7tF5P9J8vQkFya5VXd/V3e/uLs3b044NslRST652qVyflWdn+QOSb5+w/Mu7O73bvj8o0kOy7JlJElum+TNm1578+f/obvP6+4/6e77JvnGJF+R5I+TfP+V/PkAgAG7ruTzTk5ycZIfTfIvVfWyJM9L8rruvnTD8w5J8okk/2mL1/jMho8v2fRYb/j6q6yqjkjygCxbXe6f5F+ybFV5+dV5PQBge1ypN/7u/mh3n9Td35DkW5Ocn+R/JflwVf1WVd119dS3Z9lNctlqt8zGP2dfhbneneSbNy3b6/Na3KuqnpXlYNlnJjkjybHdfUx3P727z70K3xMA2GZXeQtEd7+lu38yyU2z7LK5dZK3VtV/SvJXSf4uycur6r9W1S2q6u5V9eurx6+spyd5SFU9vKpuVVWPS/JNm57zoCSvSXLdJA9M8jXd/fPd/a6r+jMBADOu7K6ZL7A6PuQlSV5SVV+R5NLu7qq6f5YzXv4wy7Ean8gSJ8+9Cq/9Z1X1dUlOynLMyZ8n+e0kD93wtNcluUl3f+YLXwEAOBBc7RDZaONul+4+L8mjV3+2eu5zkjxn07I3JqlNy56U5EmbvvzEDY9/9OpPDACsA5d4BwDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYIwQAQDGCBEAYMyu6QGAA0j39ARrpS+8cHqEtfL6Ox49PcJauXHeND3CWnnnPpbbIgIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjBEiAMAYIQIAjNk1PcCEqjohyQlJcmSOGp4GAHauHblFpLtP7u7d3b37sBwxPQ4A7Fg7MkQAgPUgRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMUIEABgjRACAMdXd0zOMqqpPJjlreo4kN0pyzvQQa8T62Jv1sTfrY2/Wx96sj72ty/r42u7+8s0Ld3yIrIuqOrW7d0/PsS6sj71ZH3uzPvZmfezN+tjbuq8Pu2YAgDFCBAAYI0TWx8nTA6wZ62Nv1sferI+9WR97sz72ttbrwzEiAMAYW0QAgDFCBAAYI0QAgDFCBAAYI0QAgDH/P8mAWzV99C4SAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "translate(u'hace mucho frio aqui.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: <start> esta es mi vida . <end>\n",
      "Predicted translation: this is my life . <end> \n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAmYAAAJwCAYAAAAjo60MAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nO3de5SlB1nn+9+TC4khCRzkKkLAAURuMqHlOoNBXKIozMhhmFEC4XKIy9ERD6OeYXkYGRQZMOrgQR2CyF0H5IyDgqggcECuKwJyUwEhXIQAQSQJgSQkz/lj75ai0h27Kp1+n135fNaq1bvevWv3U+/q7vr2e63uDgAAyztq6QEAAFgRZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIYQZgAAQwizgarqtlX1uqq689KzAABHjjCb6YwkpyV5zMJzAABHULmJ+SxVVUnOTfKaJA9K8k3dffmiQwEAR4QtZvPcL8lJSX4iyVeTPHDZcQCAI0WYzfPIJC/v7ouT/G5WuzUBgGsBuzIHqarrJvl0ku/v7jdV1V2TvDWr3ZlfWHY6AOCaZovZLP97kvO7+01J0t3vTvKhJP9u0akAYINU1XWr6pFVdb2lZ9kpYTbLI5K8eNuyF8fuTADYiYcleV5WP1c3il2ZQ1TVLZJ8NMm3dfeHtiz/5qzO0rxDd39wofEAYGNU1RuS3DjJxd29b+FxdkSYAQB7RlXdKskHk9w9yduSnNrdH1hypp2wK3OQqrrl+jpmB3zuSM8DABvoEUnetD5O+4+yYYcDCbNZPprkRtsXVtU3rp8DAK7aI5O8aP34xUkefrCNHhMJs1kqyYH2LZ+Y5CtHeBYA2ChVde8kN0vye+tFr0xyQpLvXmyoHTpm6QFIqurX1g87ydOq6uItTx+d1X7ydx/xwQBgs5yR5BXd/aUk6e5Lq+plSR6V1a0OxxNmM9x5/Wsl+bYkl2557tIk70xy1pEeCgA2RVUdl9VlMn5o21MvTvInVXVid1905CfbGWdlDrHe//2yJI/p7guXngcANklV3TCr+0u/uLuv2Pbc6Ule293nLTLcDgizIarq6KyOI/v2TTqtFwA4fBz8P0R3X57kY0mus/QsAMAybDEbpKrOyGrf+Ondff7S8wDAdFX10Rz4igZX0t3fcg2Pc7U5+H+Wn0py6yR/V1WfTPKlrU92910WmQoA5nrWlscnJnlCknckeet62b2yurrBLx/huXZFmM3y8qUHAIBN0t3/GFxV9fwkT+/uX9z6mqp6YpI7HuHRdsWuTABgT6iqC7K6N+aHty2/TZJ3dvfJy0x26Bz8DwDsFV9KctoBlp+W5OIDLB/HrsxBquo6SX42qxMAbpnk2K3Pd/fRS8wFABviV5P8elXtS/K29bJ7ZnVHgCcvNdROCLNZfj7Jv03ytKz+cP10klsl+XdJnrTcWAAwX3c/o6rOTfL4rO4CkCR/leSM7n7ZYoPtgGPMBlmf8vuj3f3HVXVhkrt2999W1Y8muX93P3ThEUeqqkfna1sZv+46cJtwajTsdVV1gyTfmwP/HX3KIkPBULaYzXKTJPuv+n9RkuuvH/9xkqcvMtFwVfXTSZ6Y5NlJ7pvkN5LcZv3Y/UVhYVV1zySvSnJJkhsl+bskN1t/fm4SYcY1oqqun23H0nf33y80ziFz8P8sH0/yTevHH07ygPXjeyX58iITzfe4JGd29xOTXJbkWd394KyuV3PKopMBSfJLSV6S5OZZ3Xbuu7LacnZO/IeTw6yqTqmqV1fVV5J8Psnn1h/nr38dzxazWX4/yf2zOmDxmUl+t6oel9U/aL+05GCDfXNWFxJMVvG6/1To310vf9wSQwH/6C5JHtvdXVWXJzmuuz9SVf9Xkt/JKtrgcHleVnubHpPkUznEOwJMIswGWW/12f/45VX1iST3SfLB7n7lcpONdl6SG2a1tfFjWW1dfHdWuzM37i8k7EGXbnn8may2ZP9VVodrfNMBvwJ27+5J7tnd71t6kN0SZoNU1X2TvKW7v5ok3f32JG+vqmOq6r7d/cZlJxzpdUkenOSdSZ6b5Fer6mFJTk2yEWfgwB73ziTfkeSDSd6Q5Beq6iZJTk/yngXnYm/6aJLjlh7i6nBW5iDrzfw36+7Pblv+jUk+6zpmV1ZVRyU5an/MVtW/zXorY5Jnd/dlS84H13br60md1N2vr6obJXlhvvZ39NHd/d5FB2RPqarvSvKfkvz77Vf/3xTCbJCquiLJTbr7c9uW3y7JOZtwK4kjrapumeQTve0PclVVklt098eXmQyAI219qanjkhyd1Zm/X936/Cb8HLUrc4Cq+oP1w07y4qq6ZMvTRye5U5K3HPHBNsNHszr1/rPblt9g/ZytjADXHj++9ABXlzCb4fPrXyvJF/L1l8a4NMmfJ3nOkR5qQ1QOfJD/iVmdmg8cYeuLZR/S7hgXgeZw6u4XLD3D1SXMBujuRyfJ+jYSZ3X3l5adaL6q+rX1w07ytKraenPao7M6M+fdR3wwIEmeteXxiUmekNXla966XnavrP6O/vIRnotrgfXJJY9I8s+SPKm7z6+q+yT5VHd/dNnp/mmOMRtkfSB7uvuK9ec3TfIDST7Q3XZlblFVr18//M6s/rHfekr+pVldUfys7v7QER4N2KKqnp/VJX9+cdvyJya5Y3efvshg7ElVdbckf5bVoSx3THL79XXznpzkdt39w0vOdyiE2SBV9eokf9zdz6yqE5P8dZLrZvU/zsd29wsXHXCgqnpeksd39wVLzwJcWVVdkOTU7WfIVdVtkrxzEw7GZnOs/9P+xu7+ufWJAN++DrN7Jfkf3T3+jjB2Zc5ytyQ/s378kCQXJLl1kocn+amsTjNni/27gferqm/I6lT8D3X3x5aZavNYbwdXVQ9J8ofdfdn68UF19/88QmNtki8lOS2r28xtdVqSi7e/GK6muyV57AGWfzqr+1GPJ8xmOSnJP6wff0+S31//MHhdkl9fbqy51rtJ3tHdv1FV18nqOJY7Jrm0qn6wu1+96IBDWW878vIkN83qzN+XX8XrOs4CPpBfTfLr6+uZvW297J5Jzkjy5KWGYs/6cpL/7QDLb58rn70/kpuYz/LxJPepqutmdQPz16yX3yD+Z3kwD8jX/rF/cFZxe9Os/sF/8jIjbQTr7RB191H7L/q8fnywD1F2AN39jKwOxL5zkl9Zf9w5yRnd7SbmHG6vSPJzVbX/6v9dVbdK8vQk/+9SQ+2EY8wGqaofyepspouyuu/jqd19RVX9RJJ/3d3fteiAA1XVV5Lcprs/WVW/leSL3f0f138R39vdJy064FDW2+6tT8q5d5Ib5+v/c9vd/ZvLTAUkSVWdnOSPktwlq2O0z8tqF+ZbknzfJlz1wK7MQbr72VV1TpJbJnnN/rMzk/xtkictN9lo5yW5U1V9OqutQGeul5+YxO2YDs5624WqOj3Jb+Vr1xzc+j/bTiLMYEHrE8H+xfrWTKdm9Z+nd3b3a5ed7NAJsyGq6npJ7tLdb0ryF9ue/ockHzjyU22E307y0iSfSnJ5VqdJJ8k9sjqrlQOz3nbnqUmekeQp++/PypWtz8T8lvX1oy7MVVxs1lmZHC5bf4529+uSvG7Lc/fJ6tJTX1hswEMkzOa4Ismrq+oB3f3m/Qur6q5Z/eG6+WKTDdbdT6mq9yU5JcnLunv/9cy+mtUxBRyA9bZrJyd5vij7J/2HJBeuH2/8LXLYGHvi56iD/4fo7guzOmjxkdueOj3Jn3T3+Ud+qo3x5STfneQ1VXWL9bLrZHWsHgdnve3cS5J8/9JDTNfdL+ju/ff8/ddZRdrvrpd/3ceCY7LH7JWfo8Jslhcm+TdVdWzyj3cC+OEkz19yqMmq6uFJXpbkg1ld8+3Y9VNH5WvXhGMb623XnpDk+6rqf1XVz1fVf976sfRwQ305q3/bPlNVz6mq+y49EHvaxv8cFWazvCary2I8aP35/bPagvGHi000388keVx3/59Z7Ybb721J7rrMSBvBetudH0nyvVmdlfmDSf7Nlo+HLjjXWOtb4Nw4q92bN0/y2qr6WFU9raruuOx07EEb/3NUmA2yPgvzJfnaZthHJHlpdztL7uBum6/dGHmri7I6HogDs95250lJ/mN337i779Tdd97ycZelh5uquy/u7hd39wOzirNfyuoH518uOxl7zV74Oerg/3lemOQv1sf8/GBWtc/BfSrJ7bK67ttW983qMiMcmPW2O0cn+YOlh9hUVXV8ku/K6hItt0vyiWUnYo/a6J+jtpgN093vT/LeJL+T5JPd/Y6FR5ru7CS/tj4VOkluUVVnZHVJA9eUOjjrbXeel9W9azlEVXVUVX1PVb0gyWey+vP16STf3d23XnY69qJN/zlqi9lML0ry35L87NKDTNfdz1hfu+Y1SY5P8voklyQ5q7vdX/QgrLddOyHJ/1FVD0jynmy7GG93/8QiU832qSTXS/LqJI9O8sotl2dhF6rqr5Lctrv9DD+4jf056pZMA1XVDbI6UPbZ3X3e0vNsgqo6IckdstoK/IHudsmHQ2C97UxVvf4qnm63Tbuyqjozq2vl/cPSs+wVVfXjSb6xu//L0rNMtck/R4UZAMAQjjEDABhCmAEADCHMBlsfm8EOWW87Z53tjvW2O9bbzllnu7OJ602YzbZxf6CGsN52zjrbHettd6y3nbPOdmfj1pswAwAY4lp/VuZ16rg+PtddeowDuiyX5Ngct/QYG8d62znrbHest92x3nZu8jqro+Zu47m0v5Lr1PFLj3FAF1zx+fO7+0bbl1/rL053fK6be9RG3a0BAMY46sSTlh5hI/3pBc/bfku8JHZlAgCMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGGJkmFXVaVXVVXXDq/MaAIBNMiLMquoNVfWsHX7ZW5LcLMnnr4GRAACOuGOWHmC3uvvSJOctPQcAwOGy+Bazqnp+ku9M8mPrXZOd5Fbrp7+9qt5eVRdX1TlVdeqWr/u6XZlVdb2qelFVfbaqvlJVH6mqnzzS3w8AwG4tHmZJHp/krUmel9WuyZsl+cT6uacl+U9JTs1ql+VLqqoO8j6/kOTOSX4gye2TPCbJ311zYwMAHF6L78rs7i9W1aVJLu7u85Kkqm6/fvpJ3f369bKnJPnzJDdP8skDvNUpSd7V3e9Yf37uwX7PqjozyZlJcnxOOBzfBgDA1TZhi9lVec+Wx59a/3rjg7z2N5M8rKr+sqrOqqrvPNibdvfZ3b2vu/cdm+MO16wAAFfL9DC7bMvjXv96wJm7+9VZbTU7K8kNk7yqqp53zY4HAHD4TAmzS5McfXXfpLvP7+4Xdfejkjw2yRlVZZMYALARFj/GbO3cJHevqlsluSi7CMb1MWjvTPL+rL6vhyT5SHdfctimBAC4Bk3ZYnZWVlvNPpDkc0luuYv3uCTJU5P8ZZI3JzkpyYMO14AAANe06u5/+lV72Ml1g75H3X/pMQBgIx110klLj7CR/vSC5/1Fd+/bvnzKFjMAgGs9YQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGOWXqApdV1js0xN/3mpcfYOF+8x82XHmHjnPZ/v2XpETbSux50ytIjbKT+4gVLj7BxrvjyV5YeYSNdcdFFS4+wp9hiBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwxEaHWVU9v6peufQcAACHwzFLD3A1PT5JLT0EAMDhsNFh1t1fXHoGAIDDZc/syqyq+1bV26rqoqr6YlW9varutPSMAACHaqO3mO1XVcckeUWS5yZ5eJJjk5ya5PIl5wIA2Ik9EWZJTk5y/SR/2N1/u1721wd7cVWdmeTMJDn+6JOu+ekAAA7BRu/K3K+7/z7J85P8SVW9qqqeUFW3uIrXn93d+7p733WO/oYjNicAwFXZE2GWJN396CT3SPLGJA9O8sGqesCyUwEAHLo9E2ZJ0t1/2d1P7+7TkrwhyRnLTgQAcOj2RJhV1a2r6r9W1b2r6pSqul+SuyT5wNKzAQAcqr1y8P/FSW6X5PeS3DDJZ5K8JMnTlxwKAGAnNjrMuvtRWz59yFJzAAAcDntiVyYAwF4gzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMMQxSw+wtL70snz17z699Bgb57ov/+TSI2ycd739lKVH2EiPeO2blx5hIz3rZx+29Agb53p/fu7SI2ykr37ms0uPsKfYYgYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDjAuzqnpDVf1mVf1yVf19VX2uqh5fVcdV1a9X1T9U1cer6hHr17+uqp617T1OrqqLq+ohy3wXAAA7Ny7M1h6e5MIk90jyX5P8tyT/K8kHk+xL8oIkv1VV35TkOUl+uKqO2/L1P5TkoiR/eCSHBgC4OqaG2fu7+8nd/aEkv5Lk/CSXdfczu/vDSZ6SpJLcO8n/THJFkh/c8vWPSfLC7r7sQG9eVWdW1TlVdc5lueQa/UYAAA7V1DB7z/4H3d1JPpvkvVuWXZbkC0lu3N2XJHlRVjGWqrpDkrsn+e2DvXl3n93d+7p737E57mAvAwA4oo5ZeoCD2L6lqw+ybH9Y/laS91TVLZM8Nslbu/sD1+yIAACH19QtZjvS3e9P8vYkj0tyeq5iaxkAwFRTt5jtxnOS/Pestqy9dOFZAAB2bE9sMVt7aZJLk7ysuy9cehgAgJ0at8Wsu087wLI7HWDZTbctun6Sb0jy3GtmMgCAa9a4MNupqjo2yc2SPDXJu7r7zQuPBACwK3thV+Z9knwsq4vRPm7hWQAAdm3jt5h19xuyutgsAMBG2wtbzAAA9gRhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADHHM0gOMcMXlS0/AtcBXP/HJpUfYSC+8512XHmEj/fn7n730CBvnO3/kzKVH2Ejf8OrPLz3CZrriwIttMQMAGEKYAQAMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMMTLMqur5VfXK7Y/Xnx9VVc+uqs9XVVfVaYsNCgBwGB2z9ACH4PFJasvnD0zy6CSnJflIkr9fYCYAgMNufJh19xe3LbpNkk9391uWmAcA4JoyclfmVtt3ayb51SS3XO/GPHe9vKrqZ6rqb6vqy1X13qo6fbmpAQB2bvwWs20en+RjSR6T5DuSXL5e/gtJHprkx5L8TZJ7JXlOVX2hu1+1xKAAADu1UWHW3V+sqguTXN7d5yVJVV03yROSfE93v2n90o9W1d2zCrUrhVlVnZnkzCQ5PicckdkBAP4pGxVmB3GHJMcn+eOq6i3Lj01y7oG+oLvPTnJ2kpxcN+gDvQYA4EjbC2G2/zi5ByX5+LbnLjvCswAA7NpeCLMPJLkkySnd/bqlhwEA2K2ND7PuvrCqzkpyVlVVkjcmOTHJPZNcsd5tCQAw3saH2dqTknwmyU8l+c0kFyR5d5JnLDkUAMBOjAyz7n7UgR6vPz8ryVnblnWS/2f9AQCwkcZfYBYA4NpCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYIhjlh4A4Kr0l7+89Agb6Q6/8e+XHmHjnP7UP1t6hI30xj+9/tIjbKbLDrzYFjMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAxxzNIDLKGqzkxyZpIcnxMWngYAYOVaucWsu8/u7n3dve/YHLf0OAAASa6lYQYAMJEwAwAYYs+GWVX9eFX99dJzAAAcqj0bZklumORblx4CAOBQ7dkw6+4nd3ctPQcAwKHas2EGALBphBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIYQZgAAQwgzAIAhhBkAwBDCDABgCGEGADCEMAMAGEKYAQAMIcwAAIY4ZukBAK7KFV/5ytIjbKRTfuXdS4+wcZ57+3svPcJGOvGl/o7uyr868GJbzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDbEyYVdVPVdW5S88BAHBN2ZgwAwDY6w5LmFXVyVV1/cPxXjv4PW9UVccfyd8TAOCatOswq6qjq+oBVfU7Sc5L8u3r5derqrOr6rNVdWFV/X9VtW/L1z2qqi6qqvtX1fuq6ktV9fqquvW29/+Zqjpv/doXJjlx2wgPTHLe+ve6z26/DwCAKXYcZlV1x6p6RpKPJ3lpki8l+d4kb6yqSvKqJDdP8gNJ/nmSNyZ5XVXdbMvbHJfkiUkek+ReSa6f5L9v+T0eluQXkvxcklOT/E2SJ2wb5SVJfjjJSUleU1Ufrqr/vD3wAAA2xSGFWVV9Y1X9RFWdk+RdSW6f5CeT3KS7H9fdb+zuTnK/JHdN8tDufkd3f7i7n5TkI0keseUtj0nyY+vXvCfJWUnuV1X75/nJJC/o7md39we7+6lJ3rF1pu7+anf/UXf/UJKbJPnF9e//ofVWusdU1fatbPu/nzOr6pyqOueyXHIoqwAA4Bp3qFvM/kOSZya5JMltu/vB3f173b29au6W5IQkn1vvgryoqi5Kcqck/2zL6y7p7r/Z8vmnkhyb1ZazJPm2JG/d9t7bP/9H3X1hd/92d98vyXckuXGS5yZ56EFef3Z37+vufcfmuKv4tgEAjpxjDvF1Zye5LMkjk7y/qn4/yYuS/Fl3X77ldUcl+UySf3mA97hgy+Ovbnuut3z9jlXVcUm+P6utcg9M8v6strq9YjfvBwCwhEMKoe7+VHc/tbu/Ncl3J7koyf9I8smq+uWq+ufrl74zq92KV6x3Y279+OwO5vqrJPfctuzrPq+Vf1FVz87q5INnJflwkrt196nd/czu/sIOfk8AgEXteAtVd7+tu380yc2y2sV5uyTvqKp/meS1Sd6c5BVV9X1VdeuquldV/Zf184fqmUnOqKrHVdVtq+qJSe6x7TWnJ/nTJCcn+aEkt+jun+7u9+30ewIAmOBQd2Veyfr4spcneXlV3TjJ5d3dVfXArM6ofE5Wx3p9JqtYe+EO3vulVfUtSZ6a1TFrf5DkV5I8asvL/izJTbv7giu/AwDA5qnVyZTXXifXDfoedf+lxwA4rI464YSlR9g4Hzz7W5ceYSOdeOJXlh5hI73vX/38X3T3vu3L3ZIJAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhhBmAABDCDMAgCGEGQDAEMIMAGAIYQYAMIQwAwAYQpgBAAwhzAAAhoJnSxwAAAJbSURBVBBmAABDCDMAgCGOWXoAAA6/Ky6+eOkRNs5tTn/X0iNwLfK+gyy3xQwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMcs/QAS6iqM5OcmSTH54SFpwEAWLlWbjHr7rO7e1937zs2xy09DgBAkmtpmAEATCTMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQwgwAYAhhBgAwhDADABhCmAEADCHMAACGEGYAAEMIMwCAIYQZAMAQ1d1Lz7Coqvpcko8tPcdB3DDJ+UsPsYGst52zznbHetsd623nrLPdmbzeTunuG21feK0Ps8mq6pzu3rf0HJvGets562x3rLfdsd52zjrbnU1cb3ZlAgAMIcwAAIYQZrOdvfQAG8p62znrbHest92x3nbOOtudjVtvjjEDABjCFjMAgCGEGQDAEMIMAGAIYQYAMIQwAwAY4v8H92G76rDlnTEAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 720x720 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "translate(u'esta es mi vida.')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.4"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}