1. A primer on transformer/1.02. Understanding Encoder of transformer.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding the encoder of transformer \n",
    "\n",
    "The transformer consists of a stack of $N$ number of encoders. The output of one encoder is sent as an input to the encoder above it.  As shown in the following figure, we have a stack of $N$ number of encoders. Each encoder sends its output to the encoder above it. The final encoder returns the representation of the given source sentence as an output. We feed the source sentence as an input to the encoder and get the representation of the source sentence as an output: \n",
    "\n",
    "![title](images/2.png)\n",
    "\n",
    "\n",
    "Note that in the transformer paper \"Attention is all you need\", the authors used $N=6$, meaning that they stacked up 6 encoders one above the another. However, we can try out different values of $N$. For simplicity and better understanding,  let's keep $N=2$: \n",
    "\n",
    "![title](images/3.png)\n",
    "\n",
    "\n",
    "\n",
    "Okay, the question is how exactly does the encoder work? How it is generating the representation for the given source sentence (input sentence)? To understand this, let tap into the encoder and see its components. The following figure shows the components of the encoder:\n",
    "\n",
    "![title](images/4.png)\n",
    "\n",
    "From the preceding figure, we can understand that all the encoder blocks are identical. We can also observe that each encoder block consists of two sublayers:\n",
    "\n",
    "- Multi-head attention \n",
    "- Feedforward network\n",
    "\n",
    "Now, let's get into details and learn how exactly these two sublayers works. To understand how multi-head attention works, first we need to understand the self-attention mechanism. So, in the next section, we will learn how the self-attention mechanism works."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}