rnn_forward.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Recurrent neural networks made simple"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/rnn_unrolled.png\">\n",
    "https://www.i2tutorials.com/deep-learning-interview-questions-and-answers/what-is-the-difference-between-bidirectional-rnn-and-rnn/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = np.array([\n",
    "    [1, 0, 0],\n",
    "    [0, 1, 1],\n",
    "    [0, 0, 1],\n",
    "    [0, 1, 1],\n",
    "    [1, 2, -1],\n",
    "    [1, 2, 3]\n",
    "])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "INPUT_DIM = 3\n",
    "HIDDEN_STATE_DIM = 7\n",
    "OUTPUT_DIM = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.41836844,  0.21889675,  0.05880767, -0.47092287, -0.18481434,\n",
       "        -0.22895088, -0.27092535],\n",
       "       [ 0.38420912, -0.15487446, -0.39751735, -0.41546494,  0.29946489,\n",
       "        -0.20955256, -0.0605188 ],\n",
       "       [-0.38133651,  0.38155813, -0.25601109, -0.46584831,  0.06213929,\n",
       "        -0.34078352,  0.07841503]])"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# W is the matrix that converts x into hidden state\n",
    "W_xh = np.random.random([INPUT_DIM, HIDDEN_STATE_DIM]) - 0.5\n",
    "W_xh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(7,)"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "xs[0].dot(W_xh).shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "W_hh = np.random.random([HIDDEN_STATE_DIM, HIDDEN_STATE_DIM]) - 0.5\n",
    "bias_h = np.random.random([1, HIDDEN_STATE_DIM]) - 0.5\n",
    "W_hy = np.random.random([HIDDEN_STATE_DIM, OUTPUT_DIM]) - 0.5\n",
    "bias_y = np.random.random([1, OUTPUT_DIM]) - 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.2987199 , 0.77339875, 0.23063691, 0.08510083, 0.74399523,\n",
       "        0.13945208, 0.22740259]])"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden_state = np.zeros([1, 7])\n",
    "nonlinearity = lambda x : np.exp(x) / (1 + np.exp(x))    \n",
    "for i in range(len(xs)):\n",
    "    hidden_state = nonlinearity(xs[i].dot(W_xh) + hidden_state.dot(W_hh) + bias_h)\n",
    "hidden_state"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.43536816, 0.56463184]])"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def softmax(x: np.array):\n",
    "    x_to_e = np.exp(x)\n",
    "    return x_to_e / np.sum(x_to_e)\n",
    "\n",
    "y = softmax(hidden_state.dot(W_hy) + bias_y)\n",
    "y"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([ 1.00038906+0.j        ,  0.08300437+0.4734688j ,\n",
       "        0.08300437-0.4734688j , -0.23752738+0.54182069j,\n",
       "       -0.23752738-0.54182069j, -0.31505461+0.j        ,\n",
       "        0.12142273+0.j        ])"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "np.linalg.eig(W_hh)[0]  # eigenvalues help understanding vanishing gradients and longer dependencies"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pytanie: Jak znaleźć dobre parametry?\n",
    "(wartości elementów macierzy W_xh, W_hh, W_hy)?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Funkcja kosztu i optymalizacja \n",
    "Powyżej dobrze zdefiniowaliśmy model. Pozostało znaleźć jego parametry. Rozważmy formalizm uogólnionych modelów liniowych. Niech $ \\mathbb{E}[y|x] = f_{W_{xh}, W_{hh}, W_{hy}}(x)$, gdzie f(x) jest składową systematyczną (czyli średnią pod warunkiem $x$). W zależności od założonego modelu na $y|x$ może nam dać to różną funkcję celu. Przykładowo jeżeli założymy, $y|x \\sim \\mathcal{N}(\\mu(x), \\sigma^2)$ maksymalizacja wiarygodności daje nam minimalizację błędu średniokwadratowego. Z kolei dla rozkładu Bernouliego maksymalizująca wiarygodności jest równoważna minimalizacji kross-entropii. \n",
    "\n",
    "https://towardsdatascience.com/why-using-mean-squared-error-mse-cost-function-for-binary-classification-is-a-bad-idea-933089e90df7"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0.0003448275862068965, 0.0003333333333333333)"
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n = 3000\n",
    "p = 100\n",
    "\n",
    "1 / (n - p), 1 / n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "TODO: Dopisać więcej - jak się różniczkuje sieć neuronową?\n",
    "\n",
    "Metody optymalizacji mogą być różne. Mamy rzeczywistą, różniczkowalną funkcję straty zależną od wieu parametrów: możemy użyć gradient descent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# LSTM"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/lstm_equations.png\">\n",
    "Stanford NLP Course (http://web.stanford.edu/class/cs224n/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "O $c$ myślimy jak o wewnętrznej pamięci, to co tak naprawdę nas interesuje to stan ukryty ($h$)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "xs = np.array([\n",
    "    [1, 0, 0],\n",
    "    [0, 1, 1],\n",
    "    [0, 0, 1],\n",
    "    [0, 1, 1],\n",
    "    [1, 2, -1],\n",
    "    [1, 2, 3]\n",
    "])\n",
    "INPUT_DIM = 3\n",
    "HIDDEN_STATE_DIM = 7\n",
    "MEMORY_DIM = HIDDEN_STATE_DIM  # has to be equal because we calculate h based on c (look at the last equation in the picture)\n",
    "OUTPUT_DIM = 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [],
   "source": [
    "def sigmoid(x: np.array):\n",
    "    return 1 / (1 + np.exp(-x))\n",
    "\n",
    "def tanh(x: np.array):\n",
    "    return np.tanh(x)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.04742587, 0.26894142, 0.5       , 0.73105858, 0.95257413])"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sigmoid(np.array([-3, -1, 0, 1, 3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-0.99505475, -0.76159416,  0.        ,  0.76159416,  0.99505475])"
      ]
     },
     "execution_count": 66,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tanh(np.array([-3, -1, 0, 1, 3]))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Forget gate matrices\n",
    "W_xh_f = np.random.random([INPUT_DIM, MEMORY_DIM]) - 0.5\n",
    "W_hh_f = np.random.random([HIDDEN_STATE_DIM, MEMORY_DIM]) - 0.5\n",
    "bias_h_f = np.random.random([1, MEMORY_DIM]) - 0.5\n",
    "\n",
    "# Input gate matrices\n",
    "W_xh_i = np.random.random([INPUT_DIM, MEMORY_DIM]) - 0.5\n",
    "W_hh_i = np.random.random([HIDDEN_STATE_DIM, MEMORY_DIM]) - 0.5\n",
    "bias_h_i = np.random.random([1, MEMORY_DIM]) - 0.5\n",
    "\n",
    "# Output gate matrices\n",
    "W_xh_o = np.random.random([INPUT_DIM, HIDDEN_STATE_DIM]) - 0.5\n",
    "W_hh_o = np.random.random([HIDDEN_STATE_DIM, HIDDEN_STATE_DIM]) - 0.5\n",
    "bias_h_o = np.random.random([1, HIDDEN_STATE_DIM]) - 0.5\n",
    "\n",
    "# Temprorary memory cell matrices (to be written)\n",
    "W_xc = np.random.random([INPUT_DIM, MEMORY_DIM]) - 0.5\n",
    "W_hc = np.random.random([HIDDEN_STATE_DIM, MEMORY_DIM]) - 0.5\n",
    "bias_c = np.random.random([1, MEMORY_DIM]) - 0.5\n",
    "\n",
    "# Final output matrices\n",
    "W_hy = np.random.random([HIDDEN_STATE_DIM, OUTPUT_DIM]) - 0.5\n",
    "bias_y = np.random.random([1, OUTPUT_DIM]) - 0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "def compute_memory_and_hidden_state(x_cur, h_prev, c_prev):\n",
    "    forget_gate = sigmoid(h_prev.dot(W_hh_f) + x_cur.dot(W_xh_f) + bias_h_f)\n",
    "    input_gate = sigmoid(h_prev.dot(W_hh_i) + x_cur.dot(W_xh_i) + bias_h_i)\n",
    "    output_gate = sigmoid(h_prev.dot(W_hh_o) + x_cur.dot(W_xh_o) + bias_h_o)\n",
    "    temporary_c = tanh(h_prev.dot(W_hc) + x_cur.dot(W_xc) + bias_c)\n",
    "    # Computing actual state cell c and hidden state h\n",
    "    c_cur = forget_gate * c_prev + input_gate * temporary_c  # `*` means elemnt-wise multiplication\n",
    "    h_cur = output_gate * tanh(c_cur)  # is it sensible to use c both for  memory state calculation later as well as output calculation?\n",
    "    # Also: what's the purpose of this tanh here, we were through nonlinearities before. Maybe it would be ok to drop it.\n",
    "    return c_cur, h_cur"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.47887667, 0.52112333]])"
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "hidden_state = np.zeros([1, HIDDEN_STATE_DIM])\n",
    "memory_state = np.zeros([1, MEMORY_DIM])\n",
    "nonlinearity = lambda x : np.exp(x) / (1 + np.exp(x))    \n",
    "for i in range(len(xs)):\n",
    "    memory_state, hidden_state = compute_memory_and_hidden_state(xs[i], hidden_state, memory_state)\n",
    "y = softmax(hidden_state.dot(W_hy) + bias_y)\n",
    "y"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.9"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}