deep_learning/RNN.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "  ## Recurrent Neural Networks \n",
    "  \n",
    "  [Wonderful Paper on the status of RNNs in 2017](https://arxiv.org/abs/1801.01078?utm_campaign=Revue%20newsletter&utm_medium=Newsletter&utm_source=The%20Wild%20Week%20in%20AI)\n",
    "  \n",
    "  The recurrent neural network is used for time series data and text. Its purpose was derrived from the need to model memory in networks, since a feed-forward net treats each sample individually. \n",
    "  \n",
    "A RNN refers toa network of artificial neurons with recurrent connections among them. The recurrent connections learn the dependencies among input sequential or time-series data. The ability to learn sequential dependencies has allowed RNNs to gain popularity in applications such as speech recognition, speech synthesis, machine vision, and video description generation.  \n",
    "  \n",
    "A Simple RNN just consists of updating a state vector iteratively white training, as so: \n",
    "  \n",
    "  ```python \n",
    "state = 0\n",
    "for input_ in input_sequence: \n",
    "    state = activation(dot(W, input_) + dot(U, state) + b)\n",
    "```\n",
    "\n",
    "But this naive model quickly loses memory via the vanishing gradient problem. This problem refers to the exponential shrinking of gradient magnitudes as they are propagated back through time. This phenomena causes memory of the network to ignore long term dependencies and hardly learn the correlation between temporally distant events. There are two reasons for that:\n",
    "\n",
    "1. Most nonlinear functions (sigmoid)have a gradient which is  close to zero almost everywhere\n",
    "\n",
    "2. The magnitude of gradient is multiplied over and over by the recurrent matrix as it is back-propagated\n",
    "\n",
    "Likewise,  gradients in training RNNs on long sequences may explode as the weights become larger and the norm of the gradient during training largely increases. One of the main challenges is training RNNs is learning long-term dependencies in data. It occurs generally due to\n",
    "the large number of parameters that need to be optimized during training in RNN over long periods of time."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------------------------------------------------\n",
    "\n",
    "### LSTM \n",
    "\n",
    "One solution is to extend the memory manually. Imagine a conveyor belt running parallel to the sequences you are processing. Information can job onto the conveyor belt at anytime, and jump back on the network at a later timestep, intact. This is the core idea of an LSTM. \n",
    "\n",
    "Under the hood, this approach changes the structure of hidden units from “sigmoid” or “tanh” to memory cells, in which their inputs and outputs are controlled by gates. These gates control flow of information to hidden neurons and preserve extracted features from previous timesteps\n",
    "\n",
    "Below is a Keras LSTM. For multi-layer LSTM's you need to set the `return_sequences` parameter to True for all intemediate layers:\n",
    "\n",
    "```python \n",
    "model = Sequential()\n",
    "model.add(layers.LSTM(32, \n",
    "                    return_sequences=True, #first or middle layers \n",
    "                    input_shape = input_shape))\n",
    "model.add(layers.LSTM(32, \n",
    "                    return_sequences=False, #last layer  \n",
    "                    ))\n",
    "model.add(Dense(1))\n",
    "```\n",
    "\n",
    "The LSTM has a higher computational cost the a simple RNN. It suffers from high complexity in the hidden layer. Foridentical size of hidden layers, a typical LSTM has about four times more  parameters than a simple RNN.\n",
    "\n",
    "\n",
    "\n",
    "#### GRU - Gated Recurrent Unit\n",
    "A computationally less expensive version of the LSTM is the GRU, or gated recurrent unit. These work in the same principle as an LSTM but are more streamlined (but have less representational power). They are more robust to vanishing gradients compared to RNN and have lower memory requirements. \n",
    "\n",
    "A basic  LSTM implementation can be found below. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Downloading data from https://s3.amazonaws.com/text-datasets/imdb.npz\n",
      "17465344/17464789 [==============================] - 15s 1us/step\n"
     ]
    }
   ],
   "source": [
    "from keras.datasets import imdb\n",
    "from keras.preprocessing import sequence\n",
    "\n",
    "max_features = 10000\n",
    "maxlen = 500\n",
    "batch_size = 32\n",
    "\n",
    "(input_train, y_train), (input_test, y_test) = imdb.load_data(num_words = max_features)\n",
    "input_train = sequence.pad_sequences(input_train, maxlen=maxlen)\n",
    "input_test = sequence.pad_sequences(input_test, maxlen=maxlen)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Train on 20000 samples, validate on 5000 samples\n",
      "Epoch 1/10\n",
      "20000/20000 [==============================] - 72s 4ms/step - loss: 0.5656 - acc: 0.7348 - val_loss: 0.3950 - val_acc: 0.8334\n",
      "Epoch 2/10\n",
      "20000/20000 [==============================] - 68s 3ms/step - loss: 0.3224 - acc: 0.8743 - val_loss: 0.3278 - val_acc: 0.8708\n",
      "Epoch 3/10\n",
      "20000/20000 [==============================] - 67s 3ms/step - loss: 0.2549 - acc: 0.9041 - val_loss: 0.3497 - val_acc: 0.8536\n",
      "Epoch 4/10\n",
      "20000/20000 [==============================] - 67s 3ms/step - loss: 0.2064 - acc: 0.9255 - val_loss: 0.7681 - val_acc: 0.7588\n",
      "Epoch 5/10\n",
      "20000/20000 [==============================] - 67s 3ms/step - loss: 0.1864 - acc: 0.9325 - val_loss: 0.3042 - val_acc: 0.8890\n",
      "Epoch 6/10\n",
      "20000/20000 [==============================] - 67s 3ms/step - loss: 0.1651 - acc: 0.9399 - val_loss: 0.3102 - val_acc: 0.8844\n",
      "Epoch 7/10\n",
      "20000/20000 [==============================] - 72s 4ms/step - loss: 0.1500 - acc: 0.9480 - val_loss: 0.3992 - val_acc: 0.8800\n",
      "Epoch 8/10\n",
      "20000/20000 [==============================] - 77s 4ms/step - loss: 0.1394 - acc: 0.9511 - val_loss: 0.4669 - val_acc: 0.8568\n",
      "Epoch 9/10\n",
      "20000/20000 [==============================] - 77s 4ms/step - loss: 0.1282 - acc: 0.9555 - val_loss: 0.4886 - val_acc: 0.8136\n",
      "Epoch 10/10\n",
      "20000/20000 [==============================] - 73s 4ms/step - loss: 0.1173 - acc: 0.9583 - val_loss: 0.3473 - val_acc: 0.8686\n"
     ]
    }
   ],
   "source": [
    "from keras.layers import Dense, Embedding, SimpleRNN, LSTM\n",
    "from keras.models import Sequential \n",
    "\n",
    "model = Sequential()\n",
    "model.add(Embedding(max_features, 32))\n",
    "model.add(LSTM(32))\n",
    "model.add(Dense(1, activation='sigmoid'))\n",
    "\n",
    "model.compile(optimizer='rmsprop', loss='binary_crossentropy', metrics =['acc'])\n",
    "history = model.fit(input_train, y_train, \n",
    "                   epochs=10, \n",
    "                   batch_size =128, \n",
    "                   validation_split = 0.2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------------------------------------------------------------\n",
    "\n",
    "### LSTM Regularization \n",
    "\n",
    "#### Dropout \n",
    "\n",
    "Dropout is a common method to prevent overfitting, but applying into the RNNs is not trivial. Yarin Gal discoverred in 2015 that the optimial way is to apply the same dropout mask to each timestep, rather than individually. You can implement this automatially in Keras. There are two options: `dropout`, which applies the dropout rate for the input units, and `recurrent_dropout`, which specifies the dropout rate for the recurrent units. \n",
    "\n",
    "```python \n",
    "model.add(layers.GRU(32, \n",
    "                     dropout = 0.2, \n",
    "                     recurrent_dropout = 0.2)\n",
    "```\n",
    "\n",
    "### Bidirectional LSTM \n",
    "\n",
    "A bidirectional LSTM is the swiss army knife of NLP. It takes advantage of the fact that you can process temporal data both forwards and backwards to extra more meaning that one direction alone. Sometimes, this techique can catch patters not seen by unidirectional RNNs alone. \n",
    "\n",
    "In implementation, the input vector is copied, reversed, then both are fed into separate LSTM layers. These are then merged (add, concatenate) and spit out. In keras, you can make one as so: \n",
    "\n",
    "```python \n",
    "model = Sequential()\n",
    "model.add(layers.Embedding(max_features, 32))\n",
    "model.add(layers.Bidirectional(layers.LSTM(32))\n",
    "model.add(layers.Dense(1))\n",
    "\n",
    "```\n",
    "\n",
    "------------------------------------------------------------------------\n",
    "\n",
    "## Convolutions in Sequence Processing \n",
    "\n",
    "Like in convnets, convolutions are good for processing sequences. This is because of their ability to represent location data from small patches. A 1D convnet for sequences is less computationally expensive than an RNN, and is good alternative for simple tasks. You can use Convultions when **global ordering is not meaninful.** Otherwise RNNs win out. \n",
    "\n",
    "You can use larger convoltion window sizes - up to 7 or 9. The architecture is similar to image nets - Conv1d --> MaxPooling --> Flatten. Here is an example: \n",
    "\n",
    "```python \n",
    "model = Sequential()\n",
    "model.add(layers.Embeddig(max_feautres, 128, input_lenth=max_len))\n",
    "model.add(layers.Conv1D(32, 7, activation='relu'))\n",
    "model.add(layers.MaxPooling1D(5))\n",
    "model.add(layers.Conv1D(32, 7, activation='relu'))\n",
    "model.add(layers.GlobalMaxPooling1D(5))\n",
    "model.add(layers.Dense(1))\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}