index.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "intensive-border",
   "metadata": {},
   "source": [
    "<h1 class=\"intro_title\" style=\"text-align:center; font-size: 45px;\">Python course 2021</h1>\n",
    "<h2 class=\"intro_subtitle\" style=\"text-align:center; font-size: 30px;\">Introduction to machine learning with Keras</h2>\n",
    "\n",
    "<img class=\"intro_logo\" style=\"width:400px\" src=\"https://static.poul.org/assets/logo/logo_text_g.svg\" alt=\"POuL logo\"/>\n",
    "\n",
    "<p class=\"intro_author\" style=\"text-align: center; font-size: 18px;\">Roberto Bochet &lt;avrdudo@poul.org&gt;</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "charming-scholarship",
   "metadata": {},
   "source": [
    "# What is machine learning?\n",
    "##### (as basic as possible)\n",
    "<small style=\"font-size: 0.5em;\">Engineers, mathematicians and scientists have mercy of me!</small>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba00996a",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import math\n",
    "np.random.seed(42) # makes results reproducible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "excellent-response",
   "metadata": {},
   "outputs": [],
   "source": [
    "f = lambda x: np.real(-0.5j*(math.e**(1j*x) - math.e**(-1j*x)) + \\\n",
    "              0.15*(math.e**(10j*x) + math.e**(-10j*x))) + \\\n",
    "              1e-4*np.e**x + np.random.normal(0,0.1,len(x))\n",
    "x = np.random.uniform(-5,5, 200)\n",
    "y = f(x) \n",
    "plt.title(\"What is this?\")\n",
    "plt.plot(x, y, \"o\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ahead-catholic",
   "metadata": {},
   "outputs": [],
   "source": [
    "import scipy as sp\n",
    "from scipy import signal\n",
    "x_fin = np.arange(-5, 5, 0.01)\n",
    "f_tri = lambda x: 2*np.abs(sp.signal.sawtooth(x - np.pi/2)) - 1\n",
    "plt.title(\"Is it a triangle wave?\")\n",
    "plt.plot(x, y.real, \"o\")\n",
    "plt.plot(x_fin, f_tri(x_fin), \"g\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "alleged-universal",
   "metadata": {},
   "outputs": [],
   "source": [
    "f_sin = lambda x: np.sin(x)\n",
    "plt.title(\"...or is it a sine?\")\n",
    "plt.plot(x, y.real, \"o\")\n",
    "plt.plot(x_fin, f_sin(x_fin), \"r\");"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "atlantic-circumstances",
   "metadata": {},
   "outputs": [],
   "source": [
    "f_poly = lambda x: x - x**3/6 + x**5/120 - x**7/5040 \\\n",
    "                    + x**9/362880 - x**11/39916800\n",
    "plt.title(\"...maybe a polynomial?\")\n",
    "plt.plot(x, y, \"o\")\n",
    "plt.plot(x_fin, f_poly(x_fin), \"m\");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "representative-delta",
   "metadata": {},
   "source": [
    "#### What are `triangle wave`, `sine` and `polynomial`?\n",
    "##### (Recap)\n",
    "\n",
    "We had some data in the form of tuple `(x,y)`\n",
    "\n",
    "We notice that there is a kind of relation between `x` and `y`, they are not random (mostly)\n",
    "\n",
    "So, we asked ourselves what value `y` assumes given a generic value of `x` (not present in the orginal dataset)\n",
    "\n",
    "We answered with some mathematical functions which seem to approximate the data quite well"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "therapeutic-exclusion",
   "metadata": {},
   "source": [
    "#### So, from which among the suggested functions was the dataset generated?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compressed-schedule",
   "metadata": {},
   "source": [
    "### Short answer: From none of them\n",
    "\n",
    "In a real scenario it is unrealistic to completely identify the \"real process\" behind a dataset\n",
    "\n",
    "A **mathematical system can provide nothing more than an approximation of a real system**  \n",
    "and this is true for all the real system"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "biblical-pipeline",
   "metadata": {},
   "source": [
    "### The mathematical functions  \n",
    "### we considered are called **mathematical models**\n",
    "an alternative to mathematical models could be **physical models**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0445f649",
   "metadata": {},
   "source": [
    "So, a reasonable question we should answer could be  \n",
    "**\"Which mathematical model better approximates the behaviour of our real system?\"**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2ec58a00",
   "metadata": {},
   "source": [
    "# Machine learning\n",
    "\n",
    ">is the study of **computer algorithms** that  \n",
    "    improve **automatically** through **experience**  \n",
    "    and by the use of data.  \n",
    ">\n",
    ">    &#91;...&#93;  \n",
    ">\n",
    ">Machine learning algorithms build a model based  \n",
    "    on sample data, &#91;...&#93; in order to make **prediction**  \n",
    "    or **decisions** without being  \n",
    "    **explicitly programmed to do so**.  \n",
    ">\n",
    ">[from wikipedia](https://en.wikipedia.org/wiki/Machine_learning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2a69c49",
   "metadata": {},
   "source": [
    "## ML branches\n",
    "\n",
    "ML splits itself in three macro areas\n",
    "\n",
    "*(incredibly simplified summary)*"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d869ca34",
   "metadata": {},
   "source": [
    "### Reinforced learning\n",
    "The model is trained like you would do with a pet:  \n",
    "when it does a good job it gets rewarded,  \n",
    "if it makes a mistake it is punished.\n",
    "\n",
    "The model should try to maximize the reward and avoid the punishments,  \n",
    "consequentially it would learn to do a good job without making mistakes.\n",
    "\n",
    "Some applications:  \n",
    "[songs suggestion](https://medium.com/analytics-vidhya/emotion-based-music-recommendation-system-using-a-deep-reinforcement-learning-approach-6d23a24d3044),\n",
    "[autonomous drive](https://towardsdatascience.com/do-you-want-to-train-a-simplified-self-driving-car-with-reinforcement-learning-be1263622e9e)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "53f1d1f8",
   "metadata": {},
   "source": [
    "### Supervised learning\n",
    "To the model are provided the input data and the result we would expect from it.\n",
    "\n",
    "The model should learn and generalize the relation between input and output,  \n",
    "so that given an input never seen before it can provide a reasonable output.\n",
    "\n",
    "Some applications:  \n",
    "[text translation](https://towardsdatascience.com/language-translation-with-rnns-d84d43b40571),\n",
    "[image classification](https://developers.google.com/machine-learning/practica/image-classification)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b4e68f01",
   "metadata": {},
   "source": [
    "### Unsupervised learning\n",
    "To the model are given only the input data, without providing the output,  \n",
    "the model itself will identify scheme and recurrences in the data.\n",
    "\n",
    "Some applications:  \n",
    "[paints style transfer](https://github.com/jcjohnson/neural-style),\n",
    "[words embedding](https://nlp.stanford.edu/projects/glove/)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2988f050",
   "metadata": {},
   "source": [
    "It could happen that they are used together for complex problems.\n",
    "\n",
    "#### However today we will talk only about **supervised learning**!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7c25a52",
   "metadata": {},
   "source": [
    "# Feed-forward Neural Network\n",
    "is a really simple model inspired by the functioning of the brain\n",
    "\n",
    "## Let us see how to compose it"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4539a85",
   "metadata": {},
   "source": [
    "### Neuron\n",
    "is the basic unit that composes the FNN\n",
    "![a neuron](./images/neuron.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17c472f4",
   "metadata": {},
   "source": [
    "Let us start with a really simple model, a linear combination of the input\n",
    "\n",
    "$ x = w_0 + w_1 u_1 + w_2 u_2 + \\dots + w_k u_k $\n",
    "\n",
    "*where $w_0$ is a parameter called bias, $u_i$ is the i-th input and $w_i$ is an arbitrary multiplication factor*\n",
    "\n",
    "So, the input data are linearly combined to get the value $x$ \n",
    "\n",
    "> The function (with $x=0$) could be seen as an equation  \n",
    "    defining a k-dimension [hyperplane](https://en.wikipedia.org/wiki/Hyperplane) of parameters $w_0, w_1, \\dots, w_k$"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbe6b512",
   "metadata": {},
   "source": [
    "Then we can transform the value $x$ to get an output exploiting an arbitrary function\n",
    "\n",
    "$y = g(x)$\n",
    "\n",
    "Where $g(\\cdot)$ is called [**activation function**](https://en.wikipedia.org/wiki/Activation_function) (and it is generally a non-linear one)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33e5f075",
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow.keras as kr\n",
    "x_act = np.arange(-6,6,0.01)\n",
    "fig, axs = plt.subplots(2,2)\n",
    "fig.suptitle(\"Some exmples of activatcion functions\")\n",
    "axs[0,0].set_title(\"Linear\")\n",
    "axs[0,0].plot(x_act, kr.activations.linear(x_act))\n",
    "axs[0,1].set_title(\"tanh\")\n",
    "axs[0,1].plot(x_act, kr.activations.tanh(x_act))\n",
    "axs[1,0].set_title(\"Sigmoid\")\n",
    "axs[1,0].plot(x_act, kr.activations.sigmoid(x_act))\n",
    "axs[1,1].set_title(\"ReLU\")\n",
    "axs[1,1].plot(x_act, kr.activations.relu(x_act));"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5159d840",
   "metadata": {},
   "source": [
    "$ x = w_0 + w_1 u_1 + w_2 u_2 + \\dots + w_k u_k $  \n",
    "$ y = g(x) $\n",
    "\n",
    "This couple of equations define entirely the concept of **Neuron** for the **FNN**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1997468",
   "metadata": {},
   "source": [
    "A single **neuron** defines the whole model of the simplest possible **FNN** at the base of the [**Perceptron**](https://en.wikipedia.org/wiki/Perceptron), a supervised algorithm invented in 1958 by [*Frank Rosenblatt*](https://en.wikipedia.org/wiki/Frank_Rosenblatt).\n",
    "\n",
    "It composed a binary classifier:  \n",
    "given an input it decided if this was part of a first class or a second one  \n",
    "(you are in or you are out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35292418",
   "metadata": {},
   "source": [
    "### Layer\n",
    "Anyway a lone Neuron is rather useless, so they are assembled in a structure called layer.\n",
    "An arbitrary number of neurons can be arranged side by side, in order to create a layer with $m$ output, where $m$ is the number of Neurons in the layer.\n",
    "\n",
    "This kind of layer is called **Dense layer** or **Fully-connected layer**\n",
    "\n",
    "![a single layer FNN](./images/layer.svg)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "911b7297",
   "metadata": {},
   "source": [
    "Layers, in turn, can be stacked in order to improve the complexity of the final model.\n",
    "\n",
    "The single **Neuron** of a **Dense layer** has as inputs all the outputs of the previous layer  \n",
    "(from here the name **Fully-connected layer**) \n",
    "\n",
    "![a multi layers FNN](./images/multi_layers.svg)\n",
    "\n",
    "n.b. the propagation of signals go only from the input to the output, **there are no loops**!  \n",
    "From here the name **Feed-forward Neural Network**!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bc9078b",
   "metadata": {},
   "source": [
    "A **FNN** model is defined by the **number of inputs**, the **number of layers**, the **number of neurons for layer** (each layer can have a different number of them) and the **neurons' activation functions**. These parameters are called [**Hyperparameters**](https://en.wikipedia.org/wiki/Hyperparameter_(machine_learning)).\n",
    "\n",
    "And by the values **$w_{i,j}$** which are called **weights**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3260712",
   "metadata": {},
   "source": [
    "Once chosen the **Hyperparameters** we created a working **FNN** model,  \n",
    "but still a dummy one!  \n",
    "##### A Hardware without a Software!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "47f2bbd7",
   "metadata": {},
   "source": [
    "## How to \"train\" a FNN?\n",
    "\n",
    "Up to now we only saw a **mathematical model**,  \n",
    "based on arranging an arbitrary number of **Neurons**,  \n",
    "but we still have no clue on how to use the data to **teach** a **behavior** to our model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2099f47e",
   "metadata": {},
   "source": [
    "The **FNN** training is treated as an optimization problem (as always...).\n",
    "\n",
    "So, after defining the **Hyperparameters** we should ask ourselves which values  \n",
    "of the **weights** are the best possible ones in order for  \n",
    "the **FNN** to simulate the behavior of our \"real system\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f3be156e",
   "metadata": {},
   "source": [
    "#### [Optimization problem](https://en.wikipedia.org/wiki/Optimization_problem) (in a nutshell)\n",
    "\n",
    "Given a **mathematical model** (one or more parametric equations) and  \n",
    "a [**loss function (or cost function)**](https://en.wikipedia.org/wiki/Loss_function)  \n",
    "(to benchmark the model given a specific set of parameters)  \n",
    "we want find the parameters set that **minimizes** (or **maximizes**) it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c4941d7",
   "metadata": {},
   "source": [
    "We already have a **mathematical model** (the **FNN** model)\n",
    "#### so we only need a **loss function**!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c06a9ba2",
   "metadata": {},
   "source": [
    "### Loss function\n",
    "In the literature several loss functions are proposed on the basis  \n",
    "of the utility context for the **FNN**.\n",
    "\n",
    "**Mean Squared Error**, **Binary Crossentropy**, **Categorical Crossentropy**  \n",
    "are some of the most used."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f7ac2fa9",
   "metadata": {},
   "source": [
    "### Training\n",
    "\n",
    "Unluckily (or not) **FNN** models overall are non-linear models,  \n",
    "so the choice of an algorithm to find the optimal **weights** is not a trivial decision.\n",
    "\n",
    "We could talk about them for hours, but (luckily for you) we have little time.  \n",
    "You just need to know that this choice too is important for the model result,  \n",
    "and all the algorithms to train the **FNN** are based on a magic thing called [**Backpropagation**](https://en.wikipedia.org/wiki/Backpropagation)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72989043",
   "metadata": {},
   "source": [
    "We can include in the **FNN** **Hyperparameters** also the **loss function**  \n",
    "and the **optimization algorithm**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "99e39a3c",
   "metadata": {},
   "source": [
    "Now we should have a general idea of how **FNN**s work.\n",
    "\n",
    "...of course, several concepts were omitted in the previous content as  \n",
    "we could talk for hours about **FNN**...  \n",
    "\n",
    "but now we have introduced all the arguments required to create  \n",
    "our first model based on **FNN**"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee6dfda5",
   "metadata": {},
   "source": [
    "# Keras\n",
    "\n",
    "<img alt=\"keras logo\" src=\"./images/keras_logo.svg\" style=\"width:15%\"/>\n",
    "\n",
    "is an **open source** library thought to simplify the prototyping of [**deep neural networks**](https://en.wikipedia.org/wiki/Deep_learning).\n",
    "\n",
    "It provides an high level interface to back-end [**TensorFlow**](https://it.wikipedia.org/wiki/TensorFlow)  \n",
    "(previously some other back-ends were supported 😕)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa603a1d",
   "metadata": {},
   "source": [
    "**n.b.**  \n",
    "a develop directly on **TensorFlow** with **Python** is possible,  \n",
    "but **Keras** provides an high abstraction level drastically speed up and simplify the prototyping with **deep neural networks**.\n",
    "\n",
    "**Keras** was considered so important that now it is shipped also  \n",
    "in the **TensorFlow** libraries (`tensorflow.keras`) "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b0647fb",
   "metadata": {},
   "source": [
    "#### Let us start with Keras!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "513dfd7f",
   "metadata": {},
   "source": [
    "# Our first FNN\n",
    "\n",
    "As first experiment we resume the first dataset we saw at the start of this talk  \n",
    "and let us try to build and train a FNN on it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "97a4ec92",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.title(\"Our dataset\")\n",
    "plt.plot(x, y, \"o\");"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d958f459",
   "metadata": {},
   "source": [
    "In particular we are looking for a model that characterizes the relation between `x` and `y` "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1935fb0d",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "Let us start by defining a simple **FNN** model,  \n",
    "also called \"choose the **Hyperparameters**\"."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bc8f879e",
   "metadata": {},
   "source": [
    "In this simple model we will use a 3-layers **FNN** with a single input (`x`).\n",
    "\n",
    "The firsts two layers will have some **Neurons** with a [**Sigmoid**](https://en.wikipedia.org/wiki/Sigmoid_function) as **activation function**.\n",
    "\n",
    "The last one (output layer) instead will have only one **Neuron** with  \n",
    "a **Linear** activation function,  \n",
    "because we need a single continuous value (`y`) as output."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "approximate-petroleum",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the keras module\n",
    "import tensorflow.keras as kr\n",
    "from tensorflow.random import set_seed\n",
    "set_seed(42) # makes the results reproducible\n",
    "\n",
    "# create a new empty model\n",
    "model = kr.Sequential()\n",
    "\n",
    "# set the input size\n",
    "model.add(kr.layers.InputLayer(input_shape=(1,)))\n",
    "# add the three Dense layers to the model\n",
    "model.add(kr.layers.Dense(30, activation=kr.activations.sigmoid))\n",
    "model.add(kr.layers.Dense(10, activation=kr.activations.sigmoid))\n",
    "model.add(kr.layers.Dense(1, activation=kr.activations.linear))\n",
    "\n",
    "# print a summary of the model\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "237556d8",
   "metadata": {},
   "source": [
    "## Optimization\n",
    "**Hyperparameters** that still have to be chosen are  \n",
    "the **optimization algorithm** and **loss function**.\n",
    "\n",
    "For the first one we will use the [**Adam**](https://keras.io/api/optimizers/adam/) algorithm  \n",
    "and since we want to do a [nonlinear regression](https://en.wikipedia.org/wiki/Nonlinear_regression)  \n",
    "we can use the [**Mean Squared Error**](https://en.wikipedia.org/wiki/Mean_squared_error) as loss function \n",
    "\n",
    "We will use this choices to configure the FNN for the training "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "smooth-bottom",
   "metadata": {},
   "outputs": [],
   "source": [
    "# complete the FNN setup with optimizer algorithm and loss function \n",
    "model.compile(\n",
    "    optimizer=kr.optimizers.Adam(learning_rate=1e-3),\n",
    "    loss=kr.losses.mean_squared_error\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75149ff4",
   "metadata": {},
   "source": [
    "##### The **FNN** is ready to be trained!!!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85c35ed7",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "We will use the method `fit` to start the model training.  \n",
    "We have to give to it the inputs data, the expected output data,  \n",
    "and the times the training algorithm will have to iterate on the given dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7afdbe05",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "history = model.fit(x, y, epochs=750)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0f5a73da",
   "metadata": {},
   "source": [
    "#### The training is over!\n",
    "Let us check how it went!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0af7e76",
   "metadata": {},
   "outputs": [],
   "source": [
    "plt.ylabel('loss')\n",
    "plt.xlabel('epoch')\n",
    "plt.plot(history.history[\"loss\"]);"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "de6d1bd2",
   "metadata": {},
   "source": [
    "From our trained model we are now able to get  \n",
    "an estimation on the relation between `x` and `y` values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "75263b56",
   "metadata": {},
   "outputs": [],
   "source": [
    "x_pred = np.arange(-5, 5, 0.01)\n",
    "y_pred = model.predict(x_pred)\n",
    "\n",
    "plt.plot(x, y, \"o\")\n",
    "plt.plot(x_pred, y_pred, \"r\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0eb19e78",
   "metadata": {},
   "source": [
    "We got a model(extremely limited) of our real system that  \n",
    "we can reuse in order to do predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29a47f38",
   "metadata": {},
   "source": [
    "# The \"Hello world\" of ML\n",
    "\n",
    "So, we saw a first(useless) model realized on garbage data,  \n",
    "now it is the time to realize our first model on a real challenge\n",
    "\n",
    "In the machine learning amateur world there is a famous site  \n",
    "to challenge you with several ML problems.  \n",
    "It is [Kaggle](https://www.kaggle.com) and it proposes a first challenge to begin with this world"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31755bf2",
   "metadata": {},
   "source": [
    "Before starting we need to authenticate to **Kaggle**,&nbsp;  \n",
    "in order to do this we need to require a token API."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1c962be7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# insert here your authentication data\n",
    "%env KAGGLE_USERNAME={YOUR_USERNAME}\n",
    "%env KAGGLE_KEY={YOUR_KEY}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1f5309f3",
   "metadata": {},
   "source": [
    "# The Titanic challenge\n",
    "\n",
    "We will be given two **lists of Titanic passengers** with some informations,  \n",
    "the first one (**training dataset**) includes also an indication **if the passenger survived**,&nbsp;  \n",
    "the second one (**test dataset**) does not include the passenger survived informations.\n",
    "\n",
    "Using the training dataset we have to build a model in order to **predict** which passengers in the test dataset survived.\n",
    "\n",
    "## 🥳 a lot of positive vibes"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08a570df",
   "metadata": {},
   "source": [
    "## Dataset\n",
    "First, let us retrieve the dataset, to do this we will exploit the [**Kaggle API**](https://github.com/Kaggle/kaggle-api)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4af9be4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "# this is a special cell,\n",
    "# the above code is pass to bash (directly to the os)\n",
    "kaggle competitions download -c titanic # download the compressed datasets\n",
    "unzip -o titanic.zip -d titanic # unzip the datasets in `titanic` folder"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5cedb09",
   "metadata": {},
   "source": [
    "Now we have to load the dataset in Python, we can choose several ways to do it.\n",
    "\n",
    "A powerful library thought to work with complex datasets could be [**Pandas**](https://pandas.pydata.org/)&nbsp;  \n",
    "I think you  will be fine with it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d70618e6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "ds = pd.read_csv(\"./titanic/train.csv\") # load the training dataset\n",
    "ds # this syntax force jupyter to render the dataset as a table"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5f9b67d",
   "metadata": {},
   "source": [
    "This dataset is really heterogeneous,  \n",
    "this contains data on the shape of `string`, `integer`, `float` and also `enum`.\n",
    "\n",
    "So, we have to pre-process the data before using them to train our model  \n",
    "##### The data pre-processing is not an optional step in ML\n",
    "##### It could be the critical point!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c5704606",
   "metadata": {},
   "outputs": [],
   "source": [
    "# drop useless(maybe) columns\n",
    "ds.drop(columns=[\"Embarked\", \"Cabin\", \"SibSp\",\n",
    "                 \"Parch\", \"Ticket\", \"PassengerId\"], inplace=True)\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b899c1ca",
   "metadata": {},
   "source": [
    "Well, now that we have less data to work with, let us look to the `Age` data.\n",
    "\n",
    "The age data is **not available for all** the dataset entries, so we have to solve this issue.\n",
    "\n",
    "We have two options (IMO), we can set it to an **arbitrary or random value** for the missing ones, or another way could be to **calculate the average age** from the other passengers and use it as an **estimation** for the missing ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "951f685c",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute average age from only not NaN values\n",
    "age_average = ds[\"Age\"].mean(skipna=True)\n",
    "# preserve only two decimals\n",
    "age_average = np.around(age_average, decimals=2)\n",
    "print(f\"The estimated average age is {age_average}\")\n",
    "\n",
    "# override NaN ages with the average age\n",
    "ds.loc[ds[\"Age\"].isna(), \"Age\"] = age_average \n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed07ba5d",
   "metadata": {},
   "source": [
    "`Sex` and `Name` are `string` we cannot use them directly!\n",
    "\n",
    "The first will be converted to `integer`.\n",
    "\n",
    "From the `Name` we can identify the title (e.g. `Mrs.`, `Miss.`, ecc)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b3aa7da",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# enumerate sex\n",
    "ds.loc[ds[\"Sex\"] == \"female\", \"Sex\"] = 0\n",
    "ds.loc[ds[\"Sex\"] == \"male\", \"Sex\"] = 1\n",
    "\n",
    "# enumerate title\n",
    "ds.loc[:,\"Title\"] = 0 # set default value for title to 0\n",
    "ds.loc[ds[\"Name\"].str.contains(\"Rev.\"), \"Title\"] = 1\n",
    "ds.loc[ds[\"Name\"].str.contains(\"Miss.\"), \"Title\"] = 2\n",
    "ds.loc[ds[\"Name\"].str.contains(\"Mr.\"), \"Title\"] = 3\n",
    "ds.loc[ds[\"Name\"].str.contains(\"Mrs.\"), \"Title\"] = 4\n",
    "ds.loc[ds[\"Name\"].str.contains(\"Master.\"), \"Title\"] = 5\n",
    "ds.loc[ds[\"Name\"].str.contains(\"Dr.\"), \"Title\"] = 6\n",
    "# delete the `Name` column, it is not longer userful\n",
    "ds.drop(columns=\"Name\", inplace=True) \n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6a2364ce",
   "metadata": {},
   "source": [
    "We have defined arbitrary values for the titles,  \n",
    "but there is not an intrinsic hierarchical titles order (`Miss` < `Mrs.`?),  \n",
    "whatever we erroneously introduced it.\n",
    "\n",
    "[**One-hot encoding**](https://it.wikipedia.org/wiki/One-hot) is a common system to solve this issue, for each category a binary codification with a single `1` is created, let us see it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7d8bf107",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create one-hot encode based on `Title`\n",
    "title_onehot = pd.get_dummies(ds[\"Title\"], prefix=\"Title\")\n",
    "# merge the one-hot table to original dataset\n",
    "ds = pd.concat([ds, title_onehot], axis=1) \n",
    "# drop the not longer required `Title` column\n",
    "ds.drop(columns=\"Title\", inplace=True) \n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b8849a6c",
   "metadata": {},
   "source": [
    "Keras expects that all input and output are `np.float`, so we should convert `int` to this type. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c67dcb8f",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds = ds.astype(np.float16) # cast whole dataframe to float\n",
    "ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9d87c681",
   "metadata": {},
   "source": [
    "The dataset is almost ready, so we move to build the model!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8d0219a",
   "metadata": {},
   "source": [
    "### Wait a moment!\n",
    "### Are we sure we can behave like in the regression problem?\n",
    "# 🤔"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e57ba10d",
   "metadata": {},
   "source": [
    "In the first problem we saw simple data and we immediately realized that there was some kind of mathematical relation between $x$ and $y$... maybe a little noise, but the relation was obvious.\n",
    "\n",
    "In this case can we say the some thing?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11a6086c",
   "metadata": {},
   "source": [
    "Also in the noise too, if we search well enough we will able to find some kind of pattern we can memorize, but this cannot help us to improve our model, indeed these \"conclusions\" can be deceptive our **FNN**.\n",
    "\n",
    "This is a well known problem of any kind of model identification system."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "32a4cc71",
   "metadata": {},
   "source": [
    "## [Overfitting](https://en.wikipedia.org/wiki/Overfitting)\n",
    "\n",
    "When it occurs, our model stops learning useful patterns and  \n",
    "starting memorizing the training dataset.\n",
    "\n",
    "This kind of mechanism is not simple to avoid, because if during the training we look only to **loss** value we  will see it tending to zero,  \n",
    "so we could erroneously think that our model is improving, while it is losing generality instead!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b6143d01",
   "metadata": {},
   "source": [
    "#### So, how can we detect overfitting?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d3e0cdd5",
   "metadata": {},
   "source": [
    "### [**Cross validation**](https://en.wikipedia.org/wiki/Cross-validation_(statistics))\n",
    "\n",
    "is a class of techniques aimed at detecting if our model is **overfitting** and how much.\n",
    "\n",
    "From this information we can define a strategy to deal with it."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a996bebf",
   "metadata": {},
   "source": [
    "As we say it, **cross validation** is not a specific algorithm, rather it is a class of algorithms, today we will see only the most simple."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e46b79e",
   "metadata": {},
   "source": [
    "#### [Holdout method](https://en.wikipedia.org/wiki/Cross-validation_(statistics)#Holdout_method)\n",
    "\n",
    "The **initial dataset**, with known output is **split** into two subsets,  \n",
    "the **training dataset** and the **validation dataset** (common split ratio could be 0.7/0.3).\n",
    "\n",
    "During the training, the first set is used to **teach** the desired behavior to the model,  \n",
    "while periodically the second one is used to test the model.\n",
    "\n",
    "The model performance(e.g. the loss function) on the two model, while the time passes, should increase, until the performance on the validation dataset starts to decrease, at that point we can assert that the **model is overfitting**."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "156a48a2",
   "metadata": {},
   "source": [
    "So, let us build our validation dataset!\n",
    "\n",
    "To do this we will use an utility of [**scikit-learn**](https://scikit-learn.org/stable/) library."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9a1dd279",
   "metadata": {},
   "outputs": [],
   "source": [
    "# import the scikit-learn module\n",
    "import sklearn as skl\n",
    "# import the submodule including desired function\n",
    "import sklearn.model_selection  \n",
    "# split the initial dataset in two random subsets.\n",
    "# 70% of initial dataset will be used for training\n",
    "# `random_state=0` makes the result reproducible\n",
    "train_df, val_df = skl.model_selection.train_test_split(ds,\n",
    "                                                  train_size=0.7,\n",
    "                                                  random_state=0)\n",
    "\n",
    "train_df.shape, val_df.shape # show the size of two new subsets"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cdbc3a75",
   "metadata": {},
   "source": [
    "Last step separate inputs and output into tuple, as **Keras** wants"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c808a7a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "train = (train_df.drop(columns=\"Survived\"), train_df[\"Survived\"])\n",
    "val = (val_df.drop(columns=\"Survived\"), val_df[\"Survived\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0d37b8a",
   "metadata": {},
   "source": [
    " #### Now, the training datasets are finally ready!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cc437de2",
   "metadata": {},
   "source": [
    "## Model\n",
    "\n",
    "Let us create a simple model for our problem.\n",
    "\n",
    "Pay attention to two things:\n",
    "- The number of input is equal to the number of dataset columns less the `Survived` one.\n",
    "- This model will be a binary classifier (the passenger lives or dies),  \n",
    "so it should have a single output with an suitable activation function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2d19da65",
   "metadata": {},
   "outputs": [],
   "source": [
    "# create a new empty model\n",
    "model = kr.Sequential()\n",
    "\n",
    "# set the input size\n",
    "# inputs number is number of columns less `Survived`\n",
    "model.add(kr.layers.InputLayer(input_shape=(len(ds.columns)-1,)))\n",
    "# add the three Dense layers to the model\n",
    "model.add(kr.layers.Dense(1000, activation=kr.activations.relu))\n",
    "model.add(kr.layers.Dense(100, activation=kr.activations.relu))\n",
    "model.add(kr.layers.Dense(30, activation=kr.activations.relu))\n",
    "model.add(kr.layers.Dense(10, activation=kr.activations.relu))\n",
    "model.add(kr.layers.Dense(1, activation=kr.activations.sigmoid))\n",
    "\n",
    "# print a summary of the model\n",
    "model.summary()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4da68aff",
   "metadata": {},
   "source": [
    "As we saw before, now we have to set the **optimizer**, and the **loss function**.\n",
    "\n",
    "This time the model will not do a regression, instead it will have to behave like a **binary classifier**, to each input set it will provide one of two class (`alive`, `dead`).\n",
    "\n",
    "For the optimizer we can use again **Adam**.&nbsp;  \n",
    "Instead the **loss function** cannot be **Mean Squared Error** anymore,  \n",
    "but we need a function suitable for a two state output ($0$ or $1$), an appropriate function can be the [`binary_crossentropy`](https://gombru.github.io/2018/05/23/cross_entropy_loss/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cb58f33",
   "metadata": {},
   "source": [
    "Loss functions are perfectly to use them to mathematically teach to model the desired behavior, but for the **human readability** are not the best choice.\n",
    "\n",
    "Luckily **Keras** allows us to monitor the model performance during the train through arbitrarily **metrics functions**.\n",
    "\n",
    "In our case a comfortable metric function in order to check the model performance could be the **percentage of true prediction**,  \n",
    "this kind of metric is implemented by the [`BinaryAccuracy`](https://keras.io/api/metrics/accuracy_metrics/#binaryaccuracy-class) class."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8da29b1d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# complete the FNN setup with optimizer algorithm,\n",
    "# loss function and accuracy metric\n",
    "model.compile(\n",
    "    optimizer=kr.optimizers.Adam(learning_rate=1e-3),\n",
    "    loss=kr.losses.binary_crossentropy,\n",
    "    metrics=[\n",
    "        kr.metrics.BinaryAccuracy(name=\"accuracy\")\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1ddb350",
   "metadata": {},
   "source": [
    "## Training\n",
    "\n",
    "As we saw before, in order to detect overfitting we have to validate the model with the dataset will not used to train the model.\n",
    "\n",
    "##### Let us start"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4c06f3b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(*train,\n",
    "                    validation_data=val,\n",
    "                    epochs=300)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "33010dd2",
   "metadata": {},
   "outputs": [],
   "source": [
    "_, axs = plt.subplots(2)\n",
    "axs[0].plot(history.history[\"loss\"])\n",
    "axs[0].plot(history.history[\"val_loss\"])\n",
    "axs[0].set_ylabel('loss')\n",
    "axs[1].plot(history.history[\"accuracy\"])\n",
    "axs[1].plot(history.history[\"val_accuracy\"])\n",
    "axs[1].set_ylabel('accuracy')\n",
    "axs[1].set_xlabel('epoch');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ced3d7f6",
   "metadata": {},
   "source": [
    "#### Do you see it?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1921b15d",
   "metadata": {},
   "source": [
    "<img alt=\"meme overfitting\" src=\"./images/meme_overfitting.jpg\"/>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "54623250",
   "metadata": {},
   "source": [
    "### How to deal with overfitting\n",
    "\n",
    "There are several ways to try to prevent or reduce overfitting:\n",
    "- [Dropout layers](https://keras.io/api/layers/regularization_layers/dropout/)  \n",
    "    Some **Neurons** randomly are switched off,  \n",
    "    in order to reduce the **Neurons co-adapting**\n",
    "- [Early stopping](https://en.wikipedia.org/wiki/Early_stopping)  \n",
    "    Stop the training when the **validation loss** starting to grow\n",
    "- [Weight decay](https://towardsdatascience.com/this-thing-called-weight-decay-a7cd4bcfccab)  \n",
    "    The learning rate are not constant,  \n",
    "    it is reduced at each epoch\n",
    "- Complexity reduction  \n",
    "    A too complex model tends to overfit more than a simple one "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22e31ffd",
   "metadata": {},
   "source": [
    "Let us try to introduce **Early stopping** to our model\n",
    "\n",
    "In this case the fastest method to reinitialize the model is recreate it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8678511e",
   "metadata": {},
   "outputs": [],
   "source": [
    "model = kr.Sequential()\n",
    "\n",
    "model.add(kr.layers.InputLayer(input_shape=(len(ds.columns)-1,)))\n",
    "model.add(kr.layers.Dense(1000, activation=kr.activations.relu))\n",
    "model.add(kr.layers.Dense(100, activation=kr.activations.relu))\n",
    "model.add(kr.layers.Dense(30, activation=kr.activations.relu))\n",
    "model.add(kr.layers.Dense(10, activation=kr.activations.relu))\n",
    "model.add(kr.layers.Dense(1, activation=kr.activations.sigmoid))\n",
    "\n",
    "model.compile(\n",
    "    optimizer=kr.optimizers.Adam(learning_rate=1e-3),\n",
    "    loss=kr.losses.binary_crossentropy,\n",
    "    metrics=[\n",
    "        kr.metrics.BinaryAccuracy(name=\"accuracy\")\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d9fd94ac",
   "metadata": {},
   "source": [
    "Let us add an **EarlyStopping** instance to the training callbacks  \n",
    "(function which periodically are run during the training)  \n",
    "and start the training"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f676e0ab",
   "metadata": {},
   "outputs": [],
   "source": [
    "history = model.fit(*train,\n",
    "                    validation_data=val,\n",
    "                    epochs=300,\n",
    "                    callbacks=[\n",
    "                        kr.callbacks.EarlyStopping(monitor=\"val_loss\",\n",
    "                                                   patience=20,\n",
    "                                                   restore_best_weights=True)\n",
    "                    ])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66a6b21b",
   "metadata": {},
   "outputs": [],
   "source": [
    "_, axs = plt.subplots(2)\n",
    "axs[0].plot(history.history[\"loss\"])\n",
    "axs[0].plot(history.history[\"val_loss\"])\n",
    "axs[0].set_ylabel('loss')\n",
    "axs[1].plot(history.history[\"accuracy\"])\n",
    "axs[1].plot(history.history[\"val_accuracy\"])\n",
    "axs[1].set_ylabel('accuracy')\n",
    "axs[1].set_xlabel('epoch');"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9b595281",
   "metadata": {},
   "source": [
    "### Now the overfitting is limited"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb6fa04a",
   "metadata": {},
   "source": [
    "## Prediction\n",
    "\n",
    "Now, it is finally the moment to use our trained model  \n",
    "to provide a **prediction** on the test dataset.\n",
    "\n",
    "Before go on, we have to preprocess the **test dataset** exactly as  \n",
    "we did on the training dataset.  \n",
    "We should have a dataset in the exactly same structure of the **training one**!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a0cea571",
   "metadata": {},
   "outputs": [],
   "source": [
    "def dataset_preprocess(ds:pd.DataFrame):\n",
    "    # drop useless(maybe) columns\n",
    "    ds.drop(columns=[\"Embarked\", \"Cabin\", \"SibSp\", \n",
    "                     \"Parch\", \"Ticket\", \"PassengerId\"], inplace=True)\n",
    "    # override NaN ages with the average age\n",
    "    ds.loc[ds[\"Age\"].isna(), \"Age\"] = age_average \n",
    "    \n",
    "    # enumerate sex\n",
    "    ds.loc[ds[\"Sex\"] == \"female\", \"Sex\"] = 0\n",
    "    ds.loc[ds[\"Sex\"] == \"male\", \"Sex\"] = 1\n",
    "\n",
    "    # enumerate title\n",
    "    ds.loc[:,\"Title\"] = 0 # set default value for title to 0\n",
    "    ds.loc[ds[\"Name\"].str.contains(\"Rev.\"), \"Title\"] = 1\n",
    "    ds.loc[ds[\"Name\"].str.contains(\"Miss.\"), \"Title\"] = 2\n",
    "    ds.loc[ds[\"Name\"].str.contains(\"Mr.\"), \"Title\"] = 3\n",
    "    ds.loc[ds[\"Name\"].str.contains(\"Mrs.\"), \"Title\"] = 4\n",
    "    ds.loc[ds[\"Name\"].str.contains(\"Master.\"), \"Title\"] = 5\n",
    "    ds.loc[ds[\"Name\"].str.contains(\"Dr.\"), \"Title\"] = 6\n",
    "    # delete the `Name` column, it is not longer userful\n",
    "    ds.drop(columns=\"Name\", inplace=True) \n",
    "    \n",
    "    # create one-hot encode based on `Title`\n",
    "    title_onehot = pd.get_dummies(ds[\"Title\"], prefix=\"Title\") \n",
    "    # merge the one-hot table to original dataset\n",
    "    ds = pd.concat([ds, title_onehot], axis=1) \n",
    "    # drop the not longer required `Title` column\n",
    "    ds.drop(columns=\"Title\", inplace=True) \n",
    "    \n",
    "    ds = ds.astype(np.float16) # converts all values to `float`\n",
    "    ds = ds.fillna(0) # overwrite NaN with zeros\n",
    "    return ds"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d5f7f2c7",
   "metadata": {},
   "source": [
    "We have a function to reproduce the preprocessing on a dataset  \n",
    "(maybe we should have thought about it before 😅)\n",
    "\n",
    "So, we can import and preprocess the test dataset,  \n",
    "but we will save the `PassengerId` this time  \n",
    "(it will be useful when we will write the csv for the submission to **Kaggle**)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d7deef43",
   "metadata": {},
   "outputs": [],
   "source": [
    "ds_test = pd.read_csv(\"./titanic/test.csv\") # load the training dataset\n",
    "ds_test_passeger_id = ds_test[\"PassengerId\"] # save the PassengerId in order\n",
    "ds_test = dataset_preprocess(ds_test) # preprocess the dataset\n",
    "ds_test"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "232ec0f0",
   "metadata": {},
   "source": [
    "Let us make the prediction based on the test dataset exploiting the trained model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4680369",
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = model.predict(ds_test)\n",
    "prediction = pd.DataFrame(prediction, columns=[\"Survived\"])\n",
    "prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1cfbe727",
   "metadata": {},
   "source": [
    "Let us join the `PassengerId` and the prediction"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64f801f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = pd.concat((ds_test_passeger_id, prediction), axis=1)\n",
    "prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7af8a68f",
   "metadata": {},
   "source": [
    "The prediction is provided by the model as a floating point and not as a binary value,  \n",
    "we can interpret it as a percentage of the confidence in the passenger survival.\n",
    "\n",
    "Anyway, **Kaggle** does not request to us how much we are sure about a passenger survival,  \n",
    "but if it is survived or less.  \n",
    "So, we have to binary the our prediction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "52e90fae",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# if the prediction is less than 0.5 the passenger is dead\n",
    "prediction.loc[prediction[\"Survived\"] < 0.5, \"Survived\"] = 0\n",
    "# if the prediction is more than 0.5 the passenger is alive\n",
    "prediction.loc[prediction[\"Survived\"] >= 0.5, \"Survived\"] = 1\n",
    "# cast the result to integer\n",
    "prediction['Survived'] = prediction['Survived'].astype(int)\n",
    "prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b2965ad0",
   "metadata": {},
   "source": [
    "Let us prepare a `csv` file for the submission to **Kaggle**.\n",
    "\n",
    "Do not worry, **Pandas** has a build-in function in order to do it"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b979ca22",
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction.to_csv(\"result.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "761b55a1",
   "metadata": {},
   "source": [
    "Let us try to send our result to **Kaggle**!\n",
    "\n",
    "To do this, we will use again the **Kaggle API**\n",
    "\n",
    "### Cross the fingers 🤞🤞"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c9f6b073",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%%bash\n",
    "kaggle competitions submit --file result.csv -m \"simple FNN\" titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6eb9ab2",
   "metadata": {},
   "source": [
    "Let us check our score..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "535d5a94",
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "kaggle competitions submissions titanic"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "02d32d55",
   "metadata": {},
   "source": [
    "### So, we participated to our first Machine Learning challenge!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f21ebd9b",
   "metadata": {},
   "source": [
    "# What next?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5b523fd",
   "metadata": {},
   "source": [
    "## [Deep Learning](https://en.wikipedia.org/wiki/Deep_learning)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f78c198e",
   "metadata": {},
   "source": [
    "### Inputs\n",
    "Makes neural networks able to deal directly with complex input: \n",
    "- Images\n",
    "- Sound \n",
    "- Speech \n",
    "- Natural language\n",
    "\n",
    "are some examples"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8b3c5a69",
   "metadata": {},
   "source": [
    "### Models\n",
    "\n",
    "The general idea is use a more complex model respect **FNN** in order to allow the model to discover directly features by complex data:\n",
    "- [**Convolutional Neural Network**](https://en.wikipedia.org/wiki/Convolutional_neural_network) to deal with images\n",
    "- [**Recurrent Neural Network**](https://en.wikipedia.org/wiki/Recurrent_neural_network) to deal with natural language are some examples\n",
    "- ecc.\n",
    "\n",
    "Different kinds of **NN** can be combined in order to exploit the abilities of each of them"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b48663d",
   "metadata": {},
   "source": [
    "### Tasks\n",
    "\n",
    "The introduction of several new type of **NN** allow us to deal with really complex tasks:\n",
    "- [Speech analysis](https://towardsdatascience.com/speech-classification-using-neural-networks-the-basics-e5b08d6928b7)\n",
    "- [Image segmentation](https://en.wikipedia.org/wiki/Image_segmentation)\n",
    "- [Object detection](https://en.wikipedia.org/wiki/Object_detection)\n",
    "- [Q&A with context](https://towardsdatascience.com/implementing-question-answering-networks-with-cnns-5ae5f08e312b)\n",
    "- [Image restoration](http://vision.cs.utexas.edu/projects/on_demand_learning/)\n",
    "- [Word embedding](https://it.wikipedia.org/wiki/Word_embedding)\n",
    "- [Image generation](https://heartbeat.fritz.ai/stylegans-use-machine-learning-to-generate-and-customize-realistic-images-c943388dc672)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1e11e06",
   "metadata": {},
   "source": [
    "### An super active field: [Generative Adversarial Network (GAN)](https://en.wikipedia.org/wiki/Generative_adversarial_network)\n",
    "\n",
    "**GAN** is approach for training to model to generate original complex data, like photos.\n",
    "\n",
    "The idea is put in competition two network, two networks, with two different tasks.\n",
    "One network have to generate an original images, the second one fed with the generate image and an original one have to discriminate them, and find the false.\n",
    "\n",
    "So, they improve themselves in autonomy, when the training ends the discriminator network is discarded and the we have a trained generative network. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "df7f1265",
   "metadata": {},
   "source": [
    "#### [pix2pixHD](https://github.com/NVIDIA/pix2pixHD)\n",
    "\n",
    "<img alt=\"input\" src=\"https://raw.githubusercontent.com/NVIDIA/pix2pixHD/master/imgs/teaser_label.png\" width=\"40%\"/>\n",
    "<img alt=\"input\" src=\"https://raw.githubusercontent.com/NVIDIA/pix2pixHD/master/imgs/teaser_style.gif\" width=\"40%\"/>\n",
    "from <a href=\"https://raw.githubusercontent.com/NVIDIA/pix2pixHD/master/imgs/teaser_style.gif\" target=\"_blank\">pix2pixHD project</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea38e0f1",
   "metadata": {},
   "source": [
    "### [CycleGAN](https://github.com/junyanz/CycleGAN)\n",
    "\n",
    "<img alt=\"trasfert style\" src=\"https://camo.githubusercontent.com/c6757ece72fa4a2a24e53685c0f6fb8defb346fdd1103b1a679e21b5bce182c2/68747470733a2f2f6a756e79616e7a2e6769746875622e696f2f4379636c6547414e2f696d616765732f70686f746f327061696e74696e672e6a7067\" width=\"50%\"/>\n",
    "\n",
    "from [CycleGAN](https://github.com/junyanz/CycleGAN)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a413623c",
   "metadata": {},
   "source": [
    "## Artificial Neural Networks and Deep Learning (054307)\n",
    "### by Matteo Matteucci\n",
    "\n",
    "If you are interested in Deep Learning field you can follow this course in the first semester "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "burning-milton",
   "metadata": {},
   "source": [
    "<h1 class=\"outro_title\" style=\"text-align:center; font-size: 35px;\">Thank you!</h1>\n",
    "\n",
    "<h2 class=\"outro_subtitle\" style=\"text-align:center; font-size: 20px;\">A special thank to Francesca for the text peer review and<br/>to Francesco for the technical peer review!</h2>\n",
    "\n",
    "<img class=\"outro_logo\" style=\"width: 20%;\" src=\"https://static.poul.org/assets/logo/logo_g.svg\" alt=\"POuL logo\">\n",
    "\n",
    "<a class=\"outro_license\" style=\"display: block; margin: 20px auto;\" rel=\"license\" href=\"http://creativecommons.org/licenses/by-nc-sa/4.0/\"><img alt=\"Creative Commons License\" src=\"https://i.creativecommons.org/l/by-nc-sa/4.0/88x31.png\" /></a>\n",
    "<p class=\"outro_license_text\" style=\"font-size: 15px; text-align: center;\">Licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International<br/>\n",
    "    Notebook source code available in <a href=\"https://gitlab.poul.org/corsi/Python/keras/-/tree/2021\">this repo</a></p>\n",
    "\n",
    "<p class=\"outro_author\" style=\"text-align: center; font-size: 18px;\">Roberto Bochet &lt;avrdudo@poul.org&gt;</p>"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  },
  "latex_envs": {
   "LaTeX_envs_menu_present": true,
   "autoclose": false,
   "autocomplete": true,
   "bibliofile": "biblio.bib",
   "cite_by": "apalike",
   "current_citInitial": 1,
   "eqLabelWithNumbers": true,
   "eqNumInitial": 1,
   "hotkeys": {
    "equation": "Ctrl-E",
    "itemize": "Ctrl-I"
   },
   "labels_anchors": false,
   "latex_user_defs": false,
   "report_style_numbering": false,
   "user_envs_cfg": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}