Gr5-Provost_Iantsa-Soucasse_Bastien.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Iantsa Provost et Bastien Soucasse – Groupe 5\n",
    "\n",
    "# Projet d'ACID – 2 janvier 2022"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Introduction\n",
    "\n",
    "Afin de se lancer dans la comparaison des différents algorithmes de _Machine Learning_, il nous faut des données à exploiter pour l'entraînement et en guise de test."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Commençons par importer les modules et classes nécessaires au bon fonctionnement du projet."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import fashion_mnist\n",
    "\n",
    "from sklearn.decomposition import PCA\n",
    "from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA\n",
    "from sklearn.naive_bayes import GaussianNB\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.model_selection import RandomizedSearchCV\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import matplotlib.cm as cm\n",
    "\n",
    "import numpy as np\n",
    "\n",
    "import time\n",
    "\n",
    "%matplotlib inline\n",
    "\n",
    "VERBOSE = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut alors récupérer les données à traiter à l'aide de **keras**."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train.shape = (60000, 28, 28)\n",
      "y_train.shape = (60000,)\n",
      "X_test.shape = (10000, 28, 28)\n",
      "y_test.shape = (10000,)\n"
     ]
    }
   ],
   "source": [
    "(X_train, y_train), (X_test, y_test) = fashion_mnist.load_data()\n",
    "X_train, X_test = X_train / 255.0, X_test / 255.0\n",
    "\n",
    "assert len(X_train.shape) == 3\n",
    "assert len(X_test.shape) == 3\n",
    "assert len(y_train.shape) == 1\n",
    "assert len(y_test.shape) == 1\n",
    "\n",
    "if (VERBOSE):\n",
    "    print('X_train.shape =', X_train.shape)\n",
    "    print('y_train.shape =', y_train.shape)\n",
    "    print('X_test.shape =', X_test.shape)\n",
    "    print('y_test.shape =', y_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Si besoin, on peut avoir un aperçu des données."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train[0]:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAASdklEQVR4nO3da4xVZZYG4HcBhchNQbC4FPerlwiNRzIKUSbtEPGH0DGaJqZDJ0T6h8bu2D9GnRhMDAmZTNPpxEkbesSmJyhp0y0SNTM4SEKI0HJUWu6iWFyKgqqigAKU+5ofte2UWHut8uxzk/U+Camqs853zlenfN1VZ+1vf6KqIKJrX7dKT4CIyoNhJwqCYScKgmEnCoJhJwqiRzmfbNCgQTp69OhyPiVRKPX19WhpaZHOapnCLiIPAPgdgO4A/ktVl1r3Hz16NPL5fJanJCJDLpdLrRX8a7yIdAfwnwDmALgVwHwRubXQxyOi0sryN/t0AJ+r6n5VvQBgNYC5xZkWERVblrAPB3Cow9eHk9u+RUQWiUheRPLNzc0Zno6Isij5u/GqulxVc6qaGzx4cKmfjohSZAl7A4ARHb6uS24joiqUJexbAUwQkTEi0hPATwGsLc60iKjYCm69qeolEXkSwP+ivfW2QlV3Fm1mRFRUmfrsqvougHeLNBciKiGeLksUBMNOFATDThQEw04UBMNOFATDThQEw04UBMNOFATDThQEw04UBMNOFATDThQEw04URFkvJU3l523cKdLpVYe77Pz582Z9z549qbUpU6Zkem7ve7Pq3bpV9jiXZUPVQn9mPLITBcGwEwXBsBMFwbATBcGwEwXBsBMFwbATBcE++zUua5+9tbXVrL/66qtmvXfv3gXVAKBnz55mfdSoUWY9yzkEWXr4XZGlz3/lypXCnrPgZySiHxSGnSgIhp0oCIadKAiGnSgIhp0oCIadKAj22a9xWfvBW7ZsMetvv/22WR8zZkxq7dy5c+bYs2fPmvUhQ4aY9fnz56fW+vTpY471evRZrwNw4cKFgh+7pqamoOfMFHYRqQdwGsBlAJdUNZfl8YiodIpxZP9nVW0pwuMQUQnxb3aiILKGXQGsE5GPRGRRZ3cQkUUikheRfHNzc8anI6JCZQ37TFWdBmAOgCdE5N6r76Cqy1U1p6q5wYMHZ3w6IipUprCrakPysQnAmwCmF2NSRFR8BYddRPqISL9vPgcwG8COYk2MiIory7vxtQDeTHqCPQC8pqr/U5RZUdF079490/iNGzea9V27dpn1ixcvpta8ddnz5s0z65s3bzbrzz//fGptxowZ5tjbb7/drNfV1Zn1vXv3mvUPPvggtXbvvd/5a/hbJk6cmFqzzqsoOOyquh9Atqv8E1HZsPVGFATDThQEw04UBMNOFATDThQEl7heA6x2i7dccufOnWZ906ZNZv2GG24w66dOnUqtbdu2zRzr1WfNmmXWJ02alFqz5gX433dDQ4NZ9y6DPXPmzNTaSy+9ZI59+umnU2vWFto8shMFwbATBcGwEwXBsBMFwbATBcGwEwXBsBMFIVkvNfx95HI5zefzZXu+H4pS/gy8Pvvs2bPNuteH91jfm3dJ5Ouuuy7Tc1uXi/aW/npLYCdPnmzWve9tzZo1qbXt27ebYw8cOJBay+VyyOfznf7QeWQnCoJhJwqCYScKgmEnCoJhJwqCYScKgmEnCoLr2atA1u1/s/B26enVq5dZ79evn1n/6quvUmvWtsUA0NbWZtavv/56s3769OnUmtdnf+edd8z6unXrzPrly5fN+pEjR1Jr1lbTWfDIThQEw04UBMNOFATDThQEw04UBMNOFATDThQE++zBnT171qx7/WKv3r9//9Sa1+P36rt37zbrVi/du4aA93155wD06GFHq1u39OPs/v37zbGFco/sIrJCRJpEZEeH2waKyHsisi/5OKAksyOiounKr/F/BPDAVbc9A2C9qk4AsD75moiqmBt2Vd0IoPWqm+cCWJl8vhLAvOJOi4iKrdA36GpVtTH5/CiA2rQ7isgiEcmLSL65ubnApyOirDK/G6/t73SkvtuhqstVNaeqOe8NFyIqnULDfkxEhgJA8rGpeFMiolIoNOxrASxIPl8A4K3iTIeISsXts4vI6wBmARgkIocBLAawFMCfRWQhgAMAHi3lJK91Xs/Xq1s9W2/N+L59+8x67969zbq33v3cuXMFj+3bt69Zb2lpMevDhg1LrXl98q+//tqsDxhgd5uPHz9u1q392U+cOGGOPXjwYGrN+nm7YVfVtJX0P/bGElH14OmyREEw7ERBMOxEQTDsREEw7ERBcIlrFfAuJX3lypWCH3vDhg1m3WrjAHb7CvCXyFrLTE+dOmWOtdp2gN+6sy5j7W0H7bUsve+7qck+z2zx4sWpta1bt5pjreW3VpuWR3aiIBh2oiAYdqIgGHaiIBh2oiAYdqIgGHaiINhnrwJeH93bXtgyadIks+4tYT1//rxZ9+ZuLb9taGgwx3pbMg8dOtSsW3P3+uTWds+Af5nrsWPHmvWXX345tbZ06VJz7JgxY1Jr1vkDPLITBcGwEwXBsBMFwbATBcGwEwXBsBMFwbATBfGD6rNba3WzXo7Zq1u9bm89usfqRWd11113mfV+/fqZde9yzt6ac+u18frkly5dMuter9xbs27p2bOnWffOffDmvmXLltSa9zMpFI/sREEw7ERBMOxEQTDsREEw7ERBMOxEQTDsREFUVZ89y9rorL3uSvK2TV69erVZf//991Nrffr0Mcd614X3+ugXL1406z16pP8n1r9/f3Os16u2rgsPAGfOnEmteec2eOcXeLwtn63Hf+2118yx06ZNK2hO7pFdRFaISJOI7Ohw2wsi0iAi25J/Dxb07ERUNl35Nf6PAB7o5PbfqurU5N+7xZ0WERWbG3ZV3QigtQxzIaISyvIG3ZMi8mnya/6AtDuJyCIRyYtIvrm5OcPTEVEWhYb99wDGAZgKoBHAb9LuqKrLVTWnqjnvIn1EVDoFhV1Vj6nqZVW9AuAPAKYXd1pEVGwFhV1EOq5N/AmAHWn3JaLq4PbZReR1ALMADBKRwwAWA5glIlMBKIB6AL8oxmRKua7b63t6e4UfOHAgtdbY2GiOXbVqlVn39uP2ru1u7dft9bKPHDli1sePH2/WvT6+1ac/dOiQOdZbU+6tZ58zZ05qzerBA8CaNWvMureefcCA1LexANhr7devX2+OLZQbdlWd38nNr5RgLkRUQjxdligIhp0oCIadKAiGnSgIhp0oiKpa4rp//36z/uyzz6bWDh8+bI49duyYWa+pqTHr1lLO2tpac6zXQho4cKBZ97YutpYGe5clvuOOO8y6tbUwANx///1mvbU1fVlFr169zLHe0l/P5s2bU2snT540x44bN86sey1Nb8tnq9X72WefmWMLxSM7URAMO1EQDDtREAw7URAMO1EQDDtREAw7URBl77NbPeHHH3/cHPvFF1+k1qxLFgN+H93rm1q85bPe3LJu0Wtd7mvv3r3m2CVLlph1b3ntiy++aNZHjhxZ8GM/8sgjZt3rhVv96oaGBnOsd26Dd4lta9kxYP/3OGTIEHNsoXhkJwqCYScKgmEnCoJhJwqCYScKgmEnCoJhJwqirH32trY28zK5u3fvNsdPmTIltXbixAlzrFc/evSoWbdcuHDBrO/cudOse/3iCRMmmPW2trbUWl1dnTl29uzZZt1aEw4ADz/8sFmvr69PrVnzBoAtW7aY9bVr15p165wOby29tx2012f3WOdeeNtgW6+b1d/nkZ0oCIadKAiGnSgIhp0oCIadKAiGnSgIhp0oiLL22Xv06IHBgwen1idNmmSOb2lpSa317dvXHOutEfb68FZf1ZoX4F9X/pZbbjHr3nbS1np4b0tl75r299xzj1mfMWOGWd+xY0dqzVqHD9jbGgPATTfdVPB47xoDXh/+/PnzZt3b0llVU2veeRvWWnyrR+8e2UVkhIhsEJFdIrJTRH6Z3D5QRN4TkX3JR3tDaiKqqK78Gn8JwK9V9VYA/wTgCRG5FcAzANar6gQA65OviahKuWFX1UZV/Tj5/DSA3QCGA5gLYGVyt5UA5pVojkRUBN/rDToRGQ3gRwD+BqBWVRuT0lEAnf5hKiKLRCQvInlvfy0iKp0uh11E+gL4C4Bfqeq3zsTX9ncbOn3HQVWXq2pOVXM33nhjlrkSUQZdCruI1KA96KtU9a/JzcdEZGhSHwqgqTRTJKJicFtvIiIAXgGwW1WXdSitBbAAwNLk41veY9XU1Jitt/anSjdx4sTU2pkzZ8yx3pbON998s1kfNmxYam3EiBHmWG/Jordc0mvzWN/78ePHzbHWMlDAb1l++OGHZt1qiY4fPz7Tc3vLUK2fmXdp8ayXJvcuL37w4MHUmtWWA4BPPvkktWa9Jl3ps88A8DMA20VkW3Lbc2gP+Z9FZCGAAwAe7cJjEVGFuGFX1U0A0g65Py7udIioVHi6LFEQDDtREAw7URAMO1EQDDtREGVd4lpTU4Phw4en1h977DFz/LJly1Jr3uWWb7vtNrPuLWm0etlen/zs2bNm3evJXrp0yaxbWx97/WDv3AZvK+uxY8eadWupp9fL9pZ6WudsAPbSYO/nPWCAvYjTq3tLh63XzbukupUh6+fNIztREAw7URAMO1EQDDtREAw7URAMO1EQDDtREGXts3sWLlxo1u+8887U2pIlS8yxu3btMusjR44069ZVdrzLNVvb6AJ+P9nrs1uP762N9vrs3ty8tfbWOQbe+Qne3D3W+FGjRpljvesjeNcJ6NbNPo5++eWXqbW7777bHHvfffel1qzLivPIThQEw04UBMNOFATDThQEw04UBMNOFATDThRE2fvsVu/T6/lOnTo1tfbGG2+YY/fs2WPWn3rqKbNubT3c2tpqjvWuze714b3rzltrxr1edV1dnVnPci1/wF5r722z7b0uHmvu3jp/79wJ72f60EMPmXXr+gveNQIKxSM7URAMO1EQDDtREAw7URAMO1EQDDtREAw7URBd2Z99BIA/AagFoACWq+rvROQFAI8DaE7u+pyqvtuFxyt8thlMnjzZrK9bt67gx25ubjbrJ0+eNOvWGmQAaGpqMuvWPubetdkHDhxo1una0ZWTai4B+LWqfiwi/QB8JCLvJbXfqup/lG56RFQsXdmfvRFAY/L5aRHZDSB9Swoiqkrf6292ERkN4EcA/pbc9KSIfCoiK0Sk0/1wRGSRiORFJO/9uktEpdPlsItIXwB/AfArVW0D8HsA4wBMRfuR/zedjVPV5aqaU9WctzcXEZVOl8IuIjVoD/oqVf0rAKjqMVW9rKpXAPwBwPTSTZOIsnLDLu1vn78CYLeqLutw+9AOd/sJgPRlYURUcV15N34GgJ8B2C4i25LbngMwX0Smor0dVw/gFyWY3w+C9+dJ1j9frNYaUVd15d34TQA6a467PXUiqh48g44oCIadKAiGnSgIhp0oCIadKAiGnSgIhp0oCIadKAiGnSgIhp0oCIadKAiGnSgIhp0oCIadKAjxtvQt6pOJNAM40OGmQQBayjaB76da51at8wI4t0IVc26jVLXTCyiUNezfeXKRvKrmKjYBQ7XOrVrnBXBuhSrX3PhrPFEQDDtREJUO+/IKP7+lWudWrfMCOLdClWVuFf2bnYjKp9JHdiIqE4adKIiKhF1EHhCRvSLyuYg8U4k5pBGRehHZLiLbRCRf4bmsEJEmEdnR4baBIvKeiOxLPna6x16F5vaCiDQkr902EXmwQnMbISIbRGSXiOwUkV8mt1f0tTPmVZbXrex/s4tIdwCfAfgXAIcBbAUwX1V3lXUiKUSkHkBOVSt+AoaI3AvgDIA/qertyW3/DqBVVZcm/6McoKr/WiVzewHAmUpv453sVjS04zbjAOYB+Dkq+NoZ83oUZXjdKnFknw7gc1Xdr6oXAKwGMLcC86h6qroRQOtVN88FsDL5fCXa/2Mpu5S5VQVVbVTVj5PPTwP4Zpvxir52xrzKohJhHw7gUIevD6O69ntXAOtE5CMRWVTpyXSiVlUbk8+PAqit5GQ64W7jXU5XbTNeNa9dIdufZ8U36L5rpqpOAzAHwBPJr6tVSdv/Bqum3mmXtvEul062Gf+HSr52hW5/nlUlwt4AYESHr+uS26qCqjYkH5sAvInq24r62Dc76CYfmyo8n3+opm28O9tmHFXw2lVy+/NKhH0rgAkiMkZEegL4KYC1FZjHd4hIn+SNE4hIHwCzUX1bUa8FsCD5fAGAtyo4l2+plm2807YZR4Vfu4pvf66qZf8H4EG0vyP/BYB/q8QcUuY1FsDfk387Kz03AK+j/de6i2h/b2MhgJsArAewD8D/ARhYRXP7bwDbAXyK9mANrdDcZqL9V/RPAWxL/j1Y6dfOmFdZXjeeLksUBN+gIwqCYScKgmEnCoJhJwqCYScKgmEnCoJhJwri/wEAWB+BNM85DgAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train[0] = 9\n",
      "X_train[1]:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAR80lEQVR4nO3dbWxVZbYH8P8SyouUt9oCFXphhH7AqJchJ+QmGON14kT0A84XhQ/ITcgwGokzYUw0aoKJX8jNnZmguU4CVxzGjI6TzHBFY67jJRMImEw8IoOgudYX5CWFltfyJgis+6EbU6B7rXL22Wcfuv6/pGl71tk9zzntv6c9az/PI6oKIhr8bih6AERUGww7URAMO1EQDDtREAw7URBDa3ljzc3NOm3atFre5KBw/vx5s75nz57U2k033WQeO3LkSLMuImbdc/bs2dTakSNHzGOHDx9u1idMmGDWs479erR7924cOnSo3zueKewich+AVQCGAPgvVV1pXX/atGkol8tZbjI3XguyyB+crq4us75s2bLU2iOPPGIee/vtt5v1oUPtH5EhQ4aY9S+//DK19vrrr5vHtre3m/VHH33UrI8YMcKsD0alUim1VvGf8SIyBMB/ApgH4FYAC0Xk1kq/HhHlK8v/7HMAfKGqX6nqOQB/BDC/OsMiomrLEvbJAPb2+XxfctllRGSpiJRFpNzd3Z3h5ogoi9xfjVfV1apaUtVSS0tL3jdHRCmyhH0/gLY+n09JLiOiOpQl7B8CaBeRH4jIMAALAGyozrCIqNoqbr2p6nkRWQbgPfS23taq6q6qjezax2PWvdZZltbavn37zPqbb75p1teuXWvWGxoazPqxY8dSa5s2bTKP9dp6efLafh988IFZX758uVm/+eabU2sPP/yweeyTTz5p1ltbW816PcrUZ1fVdwG8W6WxEFGOeLosURAMO1EQDDtREAw7URAMO1EQDDtREFLL1WVLpZLW6xRXa941ADz22GOpta1bt5rHXrhwwayPHz/erI8ePdqsW/O+vSmo586dM+vefIZx48aZ9RtuSH8+sWrVcPLkyYpqAHDmzBmz/sADD5j1VatWmfW8lEollMvlfk8a4TM7URAMO1EQDDtREAw7URAMO1EQDDtREDVdSrqeLVq0yKzv2pU+e3fq1KnmsVnbX8OGDTPrVvvUW4baa71OnnzVSmOX8dqKeR07EFbLsrGx0TzWm/L8zjvvmPVnn33WrHvLYOeBz+xEQTDsREEw7ERBMOxEQTDsREEw7ERBMOxEQYTps+/du9esW310AGhra0uteX1yr9d94sQJs/7111+b9VOnTqXWvF6218P/7rvvzLq3y6vVx/ceN28J7bFjx5r1GTNmpNa8++3x7verr75q1p966qlMt18JPrMTBcGwEwXBsBMFwbATBcGwEwXBsBMFwbATBRGmz75582az7i0dfPr06dSaN1/d61V7c6tfe+01s25tH+zNm/aWim5ubjbrFy9eNOtWP9o7/8D7nmzbts2sr1ixIrU2ZcoU81jve+Z9z9esWWPWi+izZwq7iOwGcALABQDnVbVUjUERUfVV45n9X1X1UBW+DhHliP+zEwWRNewK4K8i8pGILO3vCiKyVETKIlL2/j8kovxkDfudqjobwDwAj4vIXVdeQVVXq2pJVUstLS0Zb46IKpUp7Kq6P3nfBWA9gDnVGBQRVV/FYReRUSIy+tLHAH4MYGe1BkZE1ZXl1fiJANYn62sPBfC6qv5PVUaVg3Xr1pl1b+601Xf15jZb880BwPv3Zt68eWZ9587037HePP17773XrHvro8+cOdOsW3P1s25lvWDBArP+wgsvpNa8Prq3pfOYMWPMuncOwKFD6Q0s79yGSlUcdlX9CsA/V3EsRJQjtt6IgmDYiYJg2ImCYNiJgmDYiYIIM8V1y5YtZn369Olm3ZqOaU1/HQirDTMQd9xxR2rNmz773HPPmXVvKuaSJUvMutXy9Ka43nXXVSdkXmbTpk1m3Vou+tixY+axXjvVm+JqLWMNADt27Eit3XPPPeaxleIzO1EQDDtREAw7URAMO1EQDDtREAw7URAMO1EQg6bP3tnZadYnTZpk1r0prtZ0zKxbMk+ePNmse6z77t2vI0eOmPUnnniiojFd8vLLL6fWrO2cAaCjoyPTbVu9bu9re310rz5q1Ciz/t5776XW2GcnokwYdqIgGHaiIBh2oiAYdqIgGHaiIBh2oiAGTZ995cqVZt2bcz5u3DizbvWrvT66N6fcmncNAHv27DHrx48fT61587a9cwQOHz5s1r1538OHD0+tecs59/T0mPWtW7ea9YMHD6bWvO+Jt1WZd46At920t75CHvjMThQEw04UBMNOFATDThQEw04UBMNOFATDThTEoOmze3OADxw4YNY//vhjs3706NHUmre9r7WuO+D3qm+55RazfsMN6b+zs87L9rZV9nrlVj/au+2LFy+a9bFjx5r12bNnp9aynn/gPS7t7e1m3dtuOg/uM7uIrBWRLhHZ2eeyJhF5X0Q6kvf2RtpEVLiB/Bn/OwD3XXHZ0wA2qmo7gI3J50RUx9ywq+pmAFeuXTQfwKV9fdYBeLC6wyKiaqv0BbqJqnpp4bMDACamXVFElopIWUTK3vnGRJSfzK/Ga+8rMKmvwqjqalUtqWqppaUl680RUYUqDftBEWkFgOR9V/WGRER5qDTsGwAsTj5eDOCt6gyHiPIi3rxcEXkDwN0AmgEcBLACwH8D+BOAfwLwDYCHVNVegBxAqVTScrmcbcQ5+fbbb826NTf6pZdeMo99++23zfrMmTPNuvdax4QJE1JrZ8+eNY/1+sl58n72vF62tza79bjNmTPHPPbFF1806/WqVCqhXC5LfzX3pBpVXZhS+lGmURFRTfF0WaIgGHaiIBh2oiAYdqIgGHaiIAbNFNesRowYYdanTp2aWlu+fLl57Pr16826SL+dku95yxJbS0l7rTVvmqnHm4aaZYqrN3XY+56dOnUqtZbXtsj1jM/sREEw7ERBMOxEQTDsREEw7ERBMOxEQTDsREGE6bN70ym9fnGWfnRTU5NZz9oL9/r0Fu9+W8tUFy3L9Fzve+LxHjfve5Lle1ap+v1OElFVMexEQTDsREEw7ERBMOxEQTDsREEw7ERBhOmze33NLP3k5uZms+7thONte3zjjTde85gu8e63d/5BVlY/2jt/wLvf3jLZlsbGxoqPBfzHrR7PT6i/ERFRLhh2oiAYdqIgGHaiIBh2oiAYdqIgGHaiIML02T0D2Lo6tdbQ0GAeO3LkSLN++vRpsz5s2DCzfu7cudRalvsFZFsXHrC3XfZ60V6f3VovH7DHXo998Ly591hE1opIl4js7HPZ8yKyX0S2J2/35ztMIspqIL/efgfgvn4u/42qzkre3q3usIio2tywq+pmAEdqMBYiylGWf1yWiciO5M/88WlXEpGlIlIWkXJ3d3eGmyOiLCoN+28BTAcwC0AngF+lXVFVV6tqSVVL3oQQIspPRWFX1YOqekFVLwJYA2BOdYdFRNVWUdhFpLXPpz8BsDPtukRUH9w+u4i8AeBuAM0isg/ACgB3i8gsAApgN4Cf5TfE2siyjrd3bNZ137261cv2eOcIeHPtPVYf3ht31sc1zz57Eeu+Z+WGXVUX9nPxKzmMhYhyFO80IqKgGHaiIBh2oiAYdqIgGHaiIDjFtQY6OjrM+qRJk8y6tzWx1Uby2lveFNYieWMfPny4WbfuWz3f77zwmZ0oCIadKAiGnSgIhp0oCIadKAiGnSgIhp0oCPbZa2Do0GwP85kzZ8y6NU0161LQWZeito73pqh6WzJ7S01b5x9knbp7PeIzO1EQDDtREAw7URAMO1EQDDtREAw7URAMO1EQ7LPXQGNjo1n35qt7WzZbx2ddxtob24gRIyo+3tpqGvDHNmbMGLNu6enpqfjY6xWf2YmCYNiJgmDYiYJg2ImCYNiJgmDYiYJg2ImCYJ+9BrJuD+yx5qx7fXaPt3a7N9/dkmUuPOA/rtY8/1OnTpnHeq7HLZvdn0IRaRORv4nIpyKyS0R+nlzeJCLvi0hH8n58/sMlokoN5CnnPIBfquqtAP4FwOMiciuApwFsVNV2ABuTz4moTrlhV9VOVd2WfHwCwGcAJgOYD2BdcrV1AB7MaYxEVAXX9M+kiEwD8EMAfwcwUVU7k9IBABNTjlkqImURKXd3d2cZKxFlMOCwi0gjgD8D+IWqXjaLQHtfSen31RRVXa2qJVUttbS0ZBosEVVuQGEXkQb0Bv0PqvqX5OKDItKa1FsBdOUzRCKqBrf1Jr09hlcAfKaqv+5T2gBgMYCVyfu3chnhIOAt55x1++A8W3t5bvnsjdubXuu1v6ypwSdPnjSPHYwG0mefC2ARgE9EZHty2TPoDfmfRGQJgG8APJTLCImoKtywq+oWAGm/Qn9U3eEQUV54uixREAw7URAMO1EQDDtREAw7URCc4poocspintsHe73sLFNUAb8XbvEec6+H750DYG2VnWXc1ys+sxMFwbATBcGwEwXBsBMFwbATBcGwEwXBsBMFwT57wus3Z+nDe9sae1sXZ+EtJe31+K1eNeD3urMsZZ31HAH22S/HZ3aiIBh2oiAYdqIgGHaiIBh2oiAYdqIgGHaiINhnrwNZ5mUDdr/Z+9pZt2TOuu1ylq+dZc16734PRnxmJwqCYScKgmEnCoJhJwqCYScKgmEnCoJhJwpiIPuztwH4PYCJABTAalVdJSLPA/gpgO7kqs+o6rt5DTRvea4b39bWZtaPHj1q1q19xgF7zrg3n/zs2bMVf+2B1K056d75A1nn+Vu3nXU+e5H7DFRqICfVnAfwS1XdJiKjAXwkIu8ntd+o6n/kNzwiqpaB7M/eCaAz+fiEiHwGYHLeAyOi6rqm/9lFZBqAHwL4e3LRMhHZISJrRWR8yjFLRaQsIuXu7u7+rkJENTDgsItII4A/A/iFqvYA+C2A6QBmofeZ/1f9Haeqq1W1pKqllpaW7CMmoooMKOwi0oDeoP9BVf8CAKp6UFUvqOpFAGsAzMlvmESUlRt26X3Z8RUAn6nqr/tc3trnaj8BsLP6wyOiahnIq/FzASwC8ImIbE8uewbAQhGZhd523G4AP8thfIPCsWPHzPrx48fNuteC6uzsTK1l3fbYa81l0dDQYNa99tiMGTPMek9PT2rt888/N4/15Ln0eF4G8mr8FgD9jfy67akTRcQz6IiCYNiJgmDYiYJg2ImCYNiJgmDYiYLgUtKJPPumpVLJrN92221mvampyaxnmQrq9eHHjBlj1rMsJe1NcfWmz3pTfw8fPpxamzt3rnmspx776B4+sxMFwbATBcGwEwXBsBMFwbATBcGwEwXBsBMFIVm21L3mGxPpBvBNn4uaARyq2QCuTb2OrV7HBXBslarm2Kaqar/rv9U07FfduEhZVe0zTgpSr2Or13EBHFulajU2/hlPFATDThRE0WFfXfDtW+p1bPU6LoBjq1RNxlbo/+xEVDtFP7MTUY0w7ERBFBJ2EblPRP5PRL4QkaeLGEMaEdktIp+IyHYRKRc8lrUi0iUiO/tc1iQi74tIR/K+3z32Chrb8yKyP3nstovI/QWNrU1E/iYin4rILhH5eXJ5oY+dMa6aPG41/59dRIYA+BzAvQD2AfgQwEJV/bSmA0khIrsBlFS18BMwROQuACcB/F5Vb0su+3cAR1R1ZfKLcryqPlUnY3sewMmit/FOditq7bvNOIAHAfwbCnzsjHE9hBo8bkU8s88B8IWqfqWq5wD8EcD8AsZR91R1M4AjV1w8H8C65ON16P1hqbmUsdUFVe1U1W3JxycAXNpmvNDHzhhXTRQR9skA9vb5fB/qa793BfBXEflIRJYWPZh+TFTVS/s9HQAwscjB9MPdxruWrthmvG4eu0q2P8+KL9Bd7U5VnQ1gHoDHkz9X65L2/g9WT73TAW3jXSv9bDP+vSIfu0q3P8+qiLDvB9DW5/MpyWV1QVX3J++7AKxH/W1FffDSDrrJ+66Cx/O9etrGu79txlEHj12R258XEfYPAbSLyA9EZBiABQA2FDCOq4jIqOSFE4jIKAA/Rv1tRb0BwOLk48UA3ipwLJepl22807YZR8GPXeHbn6tqzd8A3I/eV+S/BPBsEWNIGdctAP6RvO0qemwA3kDvn3Xfofe1jSUAbgKwEUAHgP8F0FRHY3sNwCcAdqA3WK0Fje1O9P6JvgPA9uTt/qIfO2NcNXnceLosURB8gY4oCIadKAiGnSgIhp0oCIadKAiGnSgIhp0oiP8HVUQkKAeNZ6MAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train[1] = 0\n",
      "X_train[2]:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAPDklEQVR4nO3dX2wd9ZnG8efNHwdCCoqJCYZEm6biBlYiraxoUaOKVbURcBOKEGouSlZCpBcgtVIvCulFkbhBq22rCq2K3E3UdNWlqpQicoF2y0aVoDcFg7KQgFj+BTUmsU+URHZDEsfJuxceKif4/H7OmZlzxn6/H8ny8bxnPK9P8niOz3tmxtxdABa/Jb1uAEB3EHYgCMIOBEHYgSAIOxDEsm5ubM2aNb5hw4ZubnJBuHTpUrJ+6tSpZH316tVta0uWNPf3+YULF5L18+fPJ+urVq2qsp1F4ciRIzpx4oTNVSsVdjO7R9LPJS2V9O/u/kzq/hs2bNDIyEiZTS5KZ86cSdb37duXrD/wwANta00OxOjoaLL+8ccfJ+t33XVXsr506dKr7mmhGxoaalvr+Ne+mS2V9G+S7pV0u6TtZnZ7p98PQL3KPMfbLOkDd//I3ack/VbStmraAlC1MmG/VdJfZn19tFh2GTPbaWYjZjbSarVKbA5AGbW/euPuw+4+5O5DAwMDdW8OQBtlwj4qaf2sr9cVywA0UJmwvy7pNjP7spn1Sfq2pP3VtAWgah2P3tx92swel/Tfmhm97XH3w5V1tohMTU0l66+++mqy/uyzzybru3fvblu75ZZbkuv29fWVqk9OTibr586da1t77733kus+/PDDyXputJYbzUVTas7u7i9JeqmiXgDUqLlvrwJQKcIOBEHYgSAIOxAEYQeCIOxAEF09nj2q3Kz6hhtuSNaHh4eT9SeffLJt7bXXXkuu++mnnybrqTm5JN14443J+vXXX9+29uCDDybX3bYtfVxV7tBgXI49OxAEYQeCIOxAEIQdCIKwA0EQdiAIRm8NkDulcmp8JUm7du1qW3v66aeT665cuTJZ/+yzz5L1m266KVnfsmVL29qjjz6aXDd39tn+/v5kHZdjzw4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQTBnb4DcrPvkyZPJ+uDgYNvac889l1z3+PHjyfrY2FiyvnHjxmQ9dfju6dOnk+tOT08n6+6erONy7NmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjm7A2wbFm5f4bcvDold0z4wMBAsn727NlkfXx8vG1tyZL0vsbMStVxuVL/y8zsiKRJSRclTbv7UBVNAaheFXv2f3T3ExV8HwA14m92IIiyYXdJfzCzN8xs51x3MLOdZjZiZiOtVqvk5gB0qmzYt7j71yTdK+kxM/vGlXdw92F3H3L3odyLPQDqUyrs7j5afB6X9IKkzVU0BaB6HYfdzK4zsy99flvSVkmHqmoMQLXKvBq/VtILxaxzmaT/dPf/qqSrYC5dupSs5+bJS5cubVu7ePFict2JiYlkvU6549FzP3fuZ8PlOg67u38k6c4KewFQI0ZvQBCEHQiCsANBEHYgCMIOBMEhrg1w5syZZH1qaipZX7FiRdta2bFebv06T+ec2/b58+dr2/ZixJ4dCIKwA0EQdiAIwg4EQdiBIAg7EARhB4Jgzt4AuUM1c7PsVL3snD237TJz+NwptHOPC4e4Xh327EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQBHP2BsjNqq+99tqO1y97PHrqNNXzUeayytdcc02pbeNy7NmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjm7A2Qm4XnpI7rXrIk/fs8d0x42d5Spqenk/W+vr5kfWxsrMp2Fr3snt3M9pjZuJkdmrWs38xeNrP3i8+r620TQFnzeRr/K0n3XLHsCUkH3P02SQeKrwE0WDbs7v6KpJNXLN4maW9xe6+k+6ttC0DVOn2Bbq27HytuH5e0tt0dzWynmY2Y2Uir1epwcwDKKv1qvM8cSdH2aAp3H3b3IXcfGhgYKLs5AB3qNOxjZjYoScXn8epaAlCHTsO+X9KO4vYOSS9W0w6AumTn7Gb2vKS7Ja0xs6OSfizpGUm/M7NHJH0i6aE6m1zozp07l6zn5s25Y85Tx4zXOSefj9ScP/dzp647L0kTExPJeur7585Zvxhlf2J3396m9M2KewFQI94uCwRB2IEgCDsQBGEHgiDsQBDx5g89kDsdc+6UyWVOx5xT9nuXvaRzSu7w2/7+/mQ94ngthT07EARhB4Ig7EAQhB0IgrADQRB2IAjCDgTBILIBcvPk5cuXd6mT7srN+KemprrUSQzs2YEgCDsQBGEHgiDsQBCEHQiCsANBEHYgCObsXZA75jsnd9nlOk8X3ctt57537nj11ONe5zkCmoo9OxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EwZy9C3KXJs7NfHP11Dw5d8763Cw7d6x9GWUuRT2feqr3iOeUz+7ZzWyPmY2b2aFZy54ys1EzO1h83FdvmwDKms/T+F9JumeO5T9z903Fx0vVtgWgatmwu/srkk52oRcANSrzAt3jZvZW8TR/dbs7mdlOMxsxs5FWq1VicwDK6DTsv5D0FUmbJB2T9JN2d3T3YXcfcvehgYGBDjcHoKyOwu7uY+5+0d0vSfqlpM3VtgWgah2F3cwGZ335LUmH2t0XQDNkh41m9rykuyWtMbOjkn4s6W4z2yTJJR2R9N36Wlz8cvPmsvPoMt+7l8r21uSfrReyYXf37XMs3l1DLwBqxNtlgSAIOxAEYQeCIOxAEIQdCCLecX49UOfplssqc/jsfJQ5nXPZw2+b/Lj3Ant2IAjCDgRB2IEgCDsQBGEHgiDsQBCEHQiCOXsD5E73XGYWXnbWXPbSxqn1y/aWW//8+fNtaytWrCi17YWIPTsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBMGcvQvKHndd5pjyOk9DXbclS9L7olz9woULVbaz4LFnB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgmLN3Qdk5em5O3+RZecqyZeX++/X19SXrXLL5ctk9u5mtN7M/mtk7ZnbYzL5XLO83s5fN7P3i8+r62wXQqfk8jZ+W9AN3v13SP0h6zMxul/SEpAPufpukA8XXABoqG3Z3P+bubxa3JyW9K+lWSdsk7S3utlfS/TX1CKACV/UCnZltkPRVSX+WtNbdjxWl45LWtllnp5mNmNlIq9Uq0yuAEuYddjNbJWmfpO+7+8Tsms+8EjLnqyHuPuzuQ+4+NDAwUKpZAJ2bV9jNbLlmgv4bd/99sXjMzAaL+qCk8XpaBFCF7OzDZuY6uyW96+4/nVXaL2mHpGeKzy/W0uEiMD09XWr93GgtdajnQr5sce7nXr58ebJ+7ty5KttZ8OYz6Py6pO9IetvMDhbLdmkm5L8zs0ckfSLpoVo6BFCJbNjd/U+S2v2K/Wa17QCoC2+XBYIg7EAQhB0IgrADQRB2IAgOce2C3CmNc/Pk3KGgC/VQztx7AHKXss49LkePHm1bW7duXXLdxYg9OxAEYQeCIOxAEIQdCIKwA0EQdiAIwg4EwZy9C06cOFFq/dw8OjWnz13WuO7TVKd6z/WWe/9Abs6+ejUnPJ6NPTsQBGEHgiDsQBCEHQiCsANBEHYgCMIOBMGcvQtWrFiRrE9NTSXrK1euTNZTs/LcrDp3zHhuDp+TOrd77nvn5vBnzpxJ1m+++eZkPRr27EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQxHyuz75e0q8lrZXkkobd/edm9pSkRyW1irvucveX6mp0Idu4cWOy/uGHHybrp0+fTtZzc/iUsseMlz3ePWV0dDRZz71HgDn75ebzppppST9w9zfN7EuS3jCzl4vaz9z9X+trD0BV5nN99mOSjhW3J83sXUm31t0YgGpd1d/sZrZB0lcl/blY9LiZvWVme8xsznMAmdlOMxsxs5FWqzXXXQB0wbzDbmarJO2T9H13n5D0C0lfkbRJM3v+n8y1nrsPu/uQuw8NDAyU7xhAR+YVdjNbrpmg/8bdfy9J7j7m7hfd/ZKkX0raXF+bAMrKht1mXm7dLeldd//prOWDs+72LUmHqm8PQFXm82r81yV9R9LbZnawWLZL0nYz26SZcdwRSd+tob9FIXWYpyRt3bo1WT98+HCyfurUqba1ycnJ5Lq5w0xzo7ec1PfPPS65keUdd9yRrOe+fzTzeTX+T5LmGqYyUwcWEN5BBwRB2IEgCDsQBGEHgiDsQBCEHQiCU0l3QdnDSO+8886Ot3327NlkfWJiIlnPna45d4hr6rLJq1atSq5bdsafetzrPDS3qdizA0EQdiAIwg4EQdiBIAg7EARhB4Ig7EAQlpsBV7oxs5akT2YtWiPpRNcauDpN7a2pfUn01qkqe/s7d5/z/G9dDfsXNm424u5DPWsgoam9NbUvid461a3eeBoPBEHYgSB6HfbhHm8/pam9NbUvid461ZXeevo3O4Du6fWeHUCXEHYgiJ6E3czuMbP3zOwDM3uiFz20Y2ZHzOxtMztoZiM97mWPmY2b2aFZy/rN7GUze7/43P6A8e739pSZjRaP3UEzu69Hva03sz+a2TtmdtjMvlcs7+ljl+irK49b1/9mN7Olkv5P0j9JOirpdUnb3f2drjbShpkdkTTk7j1/A4aZfUPSXyX92t3/vlj2L5JOuvszxS/K1e7+w4b09pSkv/b6Mt7F1YoGZ19mXNL9kv5ZPXzsEn09pC48br3Ys2+W9IG7f+TuU5J+K2lbD/poPHd/RdLJKxZvk7S3uL1XM/9Zuq5Nb43g7sfc/c3i9qSkzy8z3tPHLtFXV/Qi7LdK+susr4+qWdd7d0l/MLM3zGxnr5uZw1p3P1bcPi5pbS+bmUP2Mt7ddMVlxhvz2HVy+fOyeIHui7a4+9ck3SvpseLpaiP5zN9gTZqdzusy3t0yx2XG/6aXj12nlz8vqxdhH5W0ftbX64pljeDuo8XncUkvqHmXoh77/Aq6xefxHvfzN026jPdclxlXAx67Xl7+vBdhf13SbWb2ZTPrk/RtSft70McXmNl1xQsnMrPrJG1V8y5FvV/SjuL2Dkkv9rCXyzTlMt7tLjOuHj92Pb/8ubt3/UPSfZp5Rf5DST/qRQ9t+too6X+Lj8O97k3S85p5WndBM69tPCLpRkkHJL0v6X8k9Teot/+Q9LaktzQTrMEe9bZFM0/R35J0sPi4r9ePXaKvrjxuvF0WCIIX6IAgCDsQBGEHgiDsQBCEHQiCsANBEHYgiP8HXyAEHHATpX4AAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train[2] = 0\n",
      "X_train[3]:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAR10lEQVR4nO3dW2xd5ZUH8P+Kc785Dsc4IQk4Y5wEhDROdBSNCFQZqqkgSEkqIWgeqoyIJnkAqZX6AAJB4QkYpi19GFVKB2gyKlRBLSICNFOIKqEKqeQkBAiB1gyY1LnYzv0ex86aB+8gJ3ivdXL2uTnr/5Ms22f5O+fzif/Z9ln725+oKojo2jem1hMgoupg2ImCYNiJgmDYiYJg2ImCGFvNB8vlctra2lrNh6wLg4ODZv3MmTNm/fDhw2Z97Nj0f8ZcLmeOnTBhglm/cOGCWT969KhZP3bsWGqtoaHBHNvc3GzWGxsbzXpEXV1dOHTokIxUyxR2EbkbwC8BNAD4L1V91vr61tZWFAqFLA85Klk/8ACwY8cOs/7KK6+YdSvQDz74oDnW+8/34MGDZv21114z62+99VZqbfr06ebYDRs2mPUVK1aY9Yjy+XxqreRf40WkAcB/ArgHwK0A1ojIraXeHxFVVpa/2ZcC+EJVv1TVfgC/A7CqPNMionLLEvY5AP4+7PPu5LbLiMh6ESmISKGvry/DwxFRFhV/NV5VN6pqXlXz3gsuRFQ5WcK+D8C8YZ/PTW4jojqUJezbAbSLyHwRGQ/gBwC2lmdaRFRuJbfeVHVARB4G8L8Yar29pKqflm1mdeajjz5KrT3zzDPm2EmTJpn1/v5+sz5x4kSz/tVXX6XW7rrrLnPs/v37zXp7e7tZHzdunFm/+eabU2szZswwx27evNmsP/nkk2Z95cqVJY+9FmXqs6vq2wDeLtNciKiCeLosURAMO1EQDDtREAw7URAMO1EQDDtREFVdz17Pjhw5YtZffvnl1NrixYvNsadPnzbrFy9eNOtjxtj/J1vr2adNm2aO9XiP7dWtXrrXo7e+LwC48847zfrevXtTa88995w59pFHHjHroxGP7ERBMOxEQTDsREEw7ERBMOxEQTDsREGw9ZZ44YUXzHpLS0vJ9+211s6ePWvWvRaUVbeWmAL+MlNvbl7r7fz582bd4l1q2rvMdVtbW2pt165d5liv3tHRYdbrEY/sREEw7ERBMOxEQTDsREEw7ERBMOxEQTDsREGwz55Yt26dWX/++edTa9dff705dvbs2Wb9xIkTZt1bCmrxtmTu6ekp+b4Bv08/efLkTPdv8b43azvpm266yRw7GvvoHh7ZiYJg2ImCYNiJgmDYiYJg2ImCYNiJgmDYiYJgnz3h9V2XL1+eWtuyZYs59vbbbzfr3rps71LUuVwuteb1or11+l6f/NSpU2bd+t6amprMsQcPHjTrHut5e/zxxzPd92iUKewi0gXgJIBBAAOqmi/HpIio/MpxZP9nVT1Uhvshogri3+xEQWQNuwL4o4jsEJH1I32BiKwXkYKIFPr6+jI+HBGVKmvY71DVJQDuAfCQiHznyi9Q1Y2qmlfVfHNzc8aHI6JSZQq7qu5L3vcCeB3A0nJMiojKr+Swi8gUEZl26WMA3wOwu1wTI6LyyvJqfAuA10Xk0v28oqr/U5ZZ1aH77rsvtfbEE0+YY70evtfrnjp1asl1b725xzsHwFurb43v7+83x06fPt2sHzt2zKyvXr06teadf3AtKjnsqvolgH8s41yIqILYeiMKgmEnCoJhJwqCYScKgmEnCoJLXBPetsrW1sTbt283xz799NMlzekSr/VmXWraWx47ZcoUsz4wMJCpPmnSpNSa95x7BgcHzfqSJUsy3f+1hkd2oiAYdqIgGHaiIBh2oiAYdqIgGHaiIBh2oiDYZ09YfXSP1wdfuHChWe/s7DTrVq8aABobG1NrDQ0N5ljvUtFeL9t6bADYv39/as3bitrrw7e1tZl1uhyP7ERBMOxEQTDsREEw7ERBMOxEQTDsREEw7ERBsM9eBV6/+OTJk2bdOwfg3LlzqTXvUtLWWMDvw2e5JPPYsdl+/ObMmZNpfDQ8shMFwbATBcGwEwXBsBMFwbATBcGwEwXBsBMFwT57kVQ1tZZsW51q/vz5Zn3Hjh1m3evTT5w4MbXm9ejPnDlj1r318GfPnjXr1nXpvR5+T0+PWfe2urZk2SdgtHK/IxF5SUR6RWT3sNtmisg7ItKZvG+q7DSJKKti/vv6DYC7r7jtUQDbVLUdwLbkcyKqY27YVfU9AEeuuHkVgE3Jx5sArC7vtIio3Er9w6RFVQ8kHx8EkPrHk4isF5GCiBT6+vpKfDgiyirzqxA69MpV6qtXqrpRVfOqmm9ubs76cERUolLD3iMiswEged9bvikRUSWUGvatANYmH68F8EZ5pkNEleL22UXkVQDLAeREpBvATwE8C2CLiKwD8DWA+ys5ydFu3rx5Zt3r+Z4/f96sHzp0KLW2YMECc6x37Xav1+39aWbdv3V+AOA/L945AHQ5N+yquial9N0yz4WIKujaO02IiEbEsBMFwbATBcGwEwXBsBMFwSWuVeBdbjnrckprvLflsrfE1ZtbU5O94LG3N/18q+PHj5tjPQMDA5nGR8MjO1EQDDtREAw7URAMO1EQDDtREAw7URAMO1EQ7LMXybtctMVbiuldEtnr02e5AlAulzPr3jJUb8vnWbNmpdasHjwATJ8+3azT1eGRnSgIhp0oCIadKAiGnSgIhp0oCIadKAiGnSgI9tmLlGXLZm/N+JEjV26ldzlr22MAyLKtltfjP336tFk/duyYWR8/fvzVTukb3qWk9+3bZ9YXLVqUWrsWt2T2xPuOiYJi2ImCYNiJgmDYiYJg2ImCYNiJgmDYiYJgn71IWdazNzY2mvWOjg6z3tbWZtZPnTqVWps8ebI5tru726x7a+nb29vNuvX43vkFc+fONet79+4163Q598guIi+JSK+I7B5221Misk9EdiVvKyo7TSLKqphf438D4O4Rbv+FqnYkb2+Xd1pEVG5u2FX1PQD271tEVPeyvED3sIh8nPyan7rhl4isF5GCiBSynMNNRNmUGvZfAWgD0AHgAICfpX2hqm5U1byq5rNcGJGIsikp7Krao6qDqnoRwK8BLC3vtIio3EoKu4jMHvbp9wHsTvtaIqoPbp9dRF4FsBxATkS6AfwUwHIR6QCgALoAbKjcFEe/Dz74wKwvXLjQrM+ZM8esjxs3LrXm9dkXLFhg1r317F4ffuzY9B+xw4cPm2M93np2a+7eNQKs6xcA2c67qBU37Kq6ZoSbX6zAXIiogni6LFEQDDtREAw7URAMO1EQDDtREFzimsjSajlx4oQ59sMPPzTrXuvNWwra09OTWrvlllvMsV5r7fPPPzfr1113nVn3LjWdhbel85tvvplae+CBB8yxo7G15uGRnSgIhp0oCIadKAiGnSgIhp0oCIadKAiGnSgI9tkTWfqq77//vllfvHixWfe2dG5qSr3qFwCgs7MztTZr1qySxwJAQ0ODWW9tbTXr27dvT615S3d7e3vNei6XM+tdXV2pNW95rXf+wGjEIztREAw7URAMO1EQDDtREAw7URAMO1EQDDtREOyzl4F3qeilS+09NAYHB836+fPnzfq5c+fMuuXChQsljwWAMWPs44VV9y5zbfXJAWDGjBkl161rAADssxPRKMawEwXBsBMFwbATBcGwEwXBsBMFwbATBcE+e5Gs65/PnTvXHOutV29sbDTrXi/c2rI5Sw8esLdcBvw+u3eOgMXbVnn//v1m/cYbb0ytZd0uejRyj+wiMk9E/iQie0TkUxH5UXL7TBF5R0Q6k/f2FRaIqKaK+TV+AMBPVPVWAP8E4CERuRXAowC2qWo7gG3J50RUp9ywq+oBVd2ZfHwSwGcA5gBYBWBT8mWbAKyu0ByJqAyu6gU6EWkFsBjAXwC0qOqBpHQQQEvKmPUiUhCRQl9fX5a5ElEGRYddRKYC+D2AH6vqZTsZ6tCuiCPujKiqG1U1r6r55ubmTJMlotIVFXYRGYehoP9WVf+Q3NwjIrOT+mwA9qVAiaim3NabDF1j+UUAn6nqz4eVtgJYC+DZ5P0bFZlhnbD+BPEut+y1zrz2lNe6sx7fWz7r8VpUWb5373lpb28363v27DHrN9xwQ2rN20r67NmzZn3SpElmvR4V02dfBuCHAD4RkV3JbY9hKORbRGQdgK8B3F+RGRJRWbhhV9U/A0jbQeG75Z0OEVUKT5clCoJhJwqCYScKgmEnCoJhJwqCS1yLZPWrvV72tGnTzPrJkyfNen9/v1mfMGFCas3bitpbourNzVsCO3HixNSad6noZcuWmfV3333XrFtLj70e/7XYZ+eRnSgIhp0oCIadKAiGnSgIhp0oCIadKAiGnSgI9tmLdPTo0dSatx49l8uZ9e7ubrPurWdvakq/sG+Wy1ADfp/du3+rH71z505z7MqVK8269X0D9r+Lt04/63UA6hGP7ERBMOxEQTDsREEw7ERBMOxEQTDsREEw7ERBsM9epN7e9D0wLl68aI71+sFWDx/we9nW1sTeWG9u3rbJ3veehbdmfObMmWbdWqs/depUc+yRI0fM+mjc3YhHdqIgGHaiIBh2oiAYdqIgGHaiIBh2oiAYdqIgitmffR6AzQBaACiAjar6SxF5CsC/Abi0cfljqvp2pSZaaydOnEiteT3b48ePZ3rsc+fOmXXr2uwDAwPm2J6eHrPe0tJi1r219tb9e4/trfP3rnlvnQPgjT19+rRZH42KOalmAMBPVHWniEwDsENE3klqv1DV/6jc9IioXIrZn/0AgAPJxydF5DMAcyo9MSIqr6v6m11EWgEsBvCX5KaHReRjEXlJREY871JE1otIQUQKfX19I30JEVVB0WEXkakAfg/gx6p6AsCvALQB6MDQkf9nI41T1Y2qmlfV/Gg8n5joWlFU2EVkHIaC/ltV/QMAqGqPqg6q6kUAvwawtHLTJKKs3LDL0DagLwL4TFV/Puz22cO+7PsAdpd/ekRULsW8Gr8MwA8BfCIiu5LbHgOwRkQ6MNSO6wKwoQLzqxt79uxJrbW3t5tjvUtNe7zLGluXe/aWiS5dav9C1tnZada9JbT33ntvas1bHuvVvaXBVkt00aJF5tjbbrvNrI9Gxbwa/2cAI23yfc321ImuRTyDjigIhp0oCIadKAiGnSgIhp0oCIadKAhR1ao9WD6f10KhULXHK6csyyW953jovKV03mWNZ8yYkVqzluZ6Y2n0yefzKBQKI/5A8chOFATDThQEw04UBMNOFATDThQEw04UBMNOFERV++wi0gfg62E35QAcqtoErk69zq1e5wVwbqUq59xuUtURr/9W1bB/68FFCqqar9kEDPU6t3qdF8C5lapac+Ov8URBMOxEQdQ67Btr/PiWep1bvc4L4NxKVZW51fRvdiKqnlof2YmoShh2oiBqEnYRuVtE/ioiX4jIo7WYQxoR6RKRT0Rkl4jUdPF9soder4jsHnbbTBF5R0Q6k/cj7rFXo7k9JSL7kudul4isqNHc5onIn0Rkj4h8KiI/Sm6v6XNnzKsqz1vV/2YXkQYAfwPwLwC6AWwHsEZV03dhqCIR6QKQV9Wan4AhIt8BcArAZlW9Lbnt3wEcUdVnk/8om1T1kTqZ21MATtV6G+9kt6LZw7cZB7AawL+ihs+dMa/7UYXnrRZH9qUAvlDVL1W1H8DvAKyqwTzqnqq+B+DKy9SsArAp+XgThn5Yqi5lbnVBVQ+o6s7k45MALm0zXtPnzphXVdQi7HMA/H3Y592or/3eFcAfRWSHiKyv9WRG0KKqB5KPDwJoqeVkRuBu411NV2wzXjfPXSnbn2fFF+i+7Q5VXQLgHgAPJb+u1iUd+husnnqnRW3jXS0jbDP+jVo+d6Vuf55VLcK+D8C8YZ/PTW6rC6q6L3nfC+B11N9W1D2XdtBN3vfWeD7fqKdtvEfaZhx18NzVcvvzWoR9O4B2EZkvIuMB/ADA1hrM41tEZErywglEZAqA76H+tqLeCmBt8vFaAG/UcC6XqZdtvNO2GUeNn7uab3+uqlV/A7ACQ6/I/x+Ax2sxh5R5/QOAj5K3T2s9NwCvYujXugsYem1jHYDrAGwD0AngXQAz62hu/w3gEwAfYyhYs2s0tzsw9Cv6xwB2JW8rav3cGfOqyvPG02WJguALdERBMOxEQTDsREEw7ERBMOxEQTDsREEw7ERB/D8YDabFNTPevQAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train[3] = 3\n",
      "X_train[4]:\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAPsAAAD4CAYAAAAq5pAIAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAPyElEQVR4nO3dX4yV9Z3H8c9XYED+hD9CYATcKQ0XEpMdmgnZBNO4abaKN9gLTLmobEKWJmrSJr1Y417gpdlsW0uyaaArKd10bZoUlESyi0satTcNI6GC6DoujjLDMAwQZRDl73cv5tEdcc7vN3Oe5/yR7/uVTM6Z53uec74c+fgcnt/5PT9zdwG4/d3R6gYANAdhB4Ig7EAQhB0IgrADQUxv5ostXrzYu7q6mvmSaGM3b95M1u+4g2PRVPX39+vcuXM2Ua1U2M3sIUm/kDRN0r+5+7Opx3d1dam3t7fMS+I2Mjo6mqzPmzevSZ3cPnp6emrW6v5fp5lNk/SvkjZIWiNps5mtqff5ADRWmc9J6yS95+4n3f2qpN9J2lhNWwCqVibsyyWdGvf7QLHtS8xsm5n1mlnvyMhIiZcDUEbDz4C4+y5373H3niVLljT65QDUUCbsg5JWjvt9RbENQBsqE/bDklab2TfMrEPS9yXtr6YtAFWre+jN3a+b2ZOS/ktjQ2+73f2tyjq7jWzatClZHx4eTtY7OzuT9Z07d9asLViwILlvWbnhs/Xr19esXbp0KbnvmjXpwZ29e/cm6x0dHcl6NKXG2d39gKQDFfUCoIH4ihIQBGEHgiDsQBCEHQiCsANBEHYgiKbOZ4/q+vXryfqpU6eS9b6+vmT9nnvuqVnLjbNv3bo1Wd+xY0eyfuPGjWT9zjvvrFlbvHhxct/z588n64yjTw1HdiAIwg4EQdiBIAg7EARhB4Ig7EAQDL01QW6IKXfJ5GXLliXrqSmyAwMDyX23b9+erB85ciRZf/3115P11NWJrl27ltx3+fKvXOUMJXBkB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgGGdvgtWrVyfrBw8eTNanT0//Z5o1a1bNmrsn981ZtWpVsr5/f3qpgNT+uUtJX7x4MVnH1HBkB4Ig7EAQhB0IgrADQRB2IAjCDgRB2IEgGGdvgrVr1ybruUtN5+a7z58/v2Ytd7nl06dPJ+s5s2fPTtZv3rxZs5b7cy9atKiunjCxUmE3s35Jo5JuSLru7j1VNAWgelUc2f/W3c9V8DwAGoh/swNBlA27SzpoZm+Y2baJHmBm28ys18x6R0ZGSr4cgHqVDfv97v4tSRskPWFm3771Ae6+y9173L0ndfFBAI1VKuzuPljcnpW0T9K6KpoCUL26w25mc8xs3uf3JX1X0vGqGgNQrTJn45dK2mdmnz/Pf7j7f1bS1W2ms7MzWZ8xY0aynlsW+cqVKzVrK1euTO6bWu5ZSo/hS1JXV1eynhpLT43BT+a1MTV1h93dT0r66wp7AdBADL0BQRB2IAjCDgRB2IEgCDsQBFNcm+Cuu+5K1mfOnJms54aopk2bVrM2d+7c5L49PemJiqlhPSk/LJj61uTly5eT+5a9DDa+jCM7EARhB4Ig7EAQhB0IgrADQRB2IAjCDgTBOHsTzJs3L1nv6+tL1ru7u5P1OXPm1KzlxqqvXr2arOfkviOQev3c1N7cZbAxNRzZgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiAIxtmbIDenPCc3nz015zw33zwnt1z0tWvXkvXUpaRzY/ws2VwtjuxAEIQdCIKwA0EQdiAIwg4EQdiBIAg7EATj7G1g1qxZpfYvls2uS24cPVdPXbNekqZPr/1XLDfXniWbq5U9spvZbjM7a2bHx21bZGavmFlfcbuwsW0CKGsyH+N/LemhW7Y9JemQu6+WdKj4HUAby4bd3V+TdOGWzRsl7Snu75H0SLVtAahavSfolrr7UHH/jKSltR5oZtvMrNfMekdGRup8OQBllT4b72NnWWqeaXH3Xe7e4+49qUX+ADRWvWEfNrNOSSpuz1bXEoBGqDfs+yVtKe5vkfRSNe0AaJTsOLuZvSDpAUmLzWxA0nZJz0r6vZltlfSBpEcb2eTtLjeW3cjnzs2Vz43h554/NZaee+7c9fYxNdmwu/vmGqXvVNwLgAbi67JAEIQdCIKwA0EQdiAIwg4EwRTXNpAb/mrkc6cu9TwZuUtVp4bXUtNfJWloaChZx9RwZAeCIOxAEIQdCIKwA0EQdiAIwg4EQdiBIBhnbwO5SyqX2T/33Llx8o6OjmQ9N46fWpY599wnT55M1jE1HNmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjG2dtA2fnsqbHyss+d2z93KenUfPncfPZ33nknWcfUcGQHgiDsQBCEHQiCsANBEHYgCMIOBEHYgSAYZ2+Cc+fOJeupOd9Sfiw7Nye9zL65+fBl6rn57IODg8k6piZ7ZDez3WZ21syOj9v2jJkNmtnR4ufhxrYJoKzJfIz/taSHJtj+c3fvLn4OVNsWgKplw+7ur0m60IReADRQmRN0T5rZm8XH/IW1HmRm28ys18x6R0ZGSrwcgDLqDfsvJX1TUrekIUk/rfVAd9/l7j3u3rNkyZI6Xw5AWXWF3d2H3f2Gu9+U9CtJ66ptC0DV6gq7mXWO+/V7ko7XeiyA9pAdZzezFyQ9IGmxmQ1I2i7pATPrluSS+iX9sHEtfv319/cn611dXcn6zJkzk/UrV65MsaP/lxtnb+Rc+9mzZyf3PX36dLKeu678qlWrkvVosmF3980TbH6+Ab0AaCC+LgsEQdiBIAg7EARhB4Ig7EAQTHFtggMH0vOEzCxZzw1/paaR5p47N0U1N702J3Up6Vxv9957b7K+Y8eOZP25555L1qPhyA4EQdiBIAg7EARhB4Ig7EAQhB0IgrADQTDO3gSvvvpqsp6bwnrt2rVkvcw4e2ocvAqpKa6ffPJJct/cFNiDBw/W1VNUHNmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjG2ZvgxIkTyXpupZzcfPbcWHpK7lLSZZ47JzfGf/ny5WR9YGCg7uefPj3eX32O7EAQhB0IgrADQRB2IAjCDgRB2IEgCDsQRLzBxgb47LPPkvXh4eFkfcWKFcl6brw5NRaeuy58bhw9Nx6d+w5Aqn716tXkvps2bUrWd+7cmax/+OGHNWsRl3POHtnNbKWZ/dHMTpjZW2b2o2L7IjN7xcz6ituFjW8XQL0m8zH+uqSfuPsaSX8j6QkzWyPpKUmH3H21pEPF7wDaVDbs7j7k7keK+6OS3pa0XNJGSXuKh+2R9EiDegRQgSmdoDOzLklrJf1Z0lJ3HypKZyQtrbHPNjPrNbPekZGRMr0CKGHSYTezuZL+IOnH7n5xfM3HzgJNeCbI3Xe5e4+79+QmfABonEmF3cxmaCzov3X3vcXmYTPrLOqdks42pkUAVcgOvdnY2Mzzkt5295+NK+2XtEXSs8XtSw3p8Gugv78/Wc8Nf3V0dCTruUsup4bPclNYP/3002Q9N6w4bdq0ZD215HPuz52bGpybItvX11ezFnHobTLj7Osl/UDSMTM7Wmx7WmMh/72ZbZX0gaRHG9IhgEpkw+7uf5JU69DxnWrbAdAofF0WCIKwA0EQdiAIwg4EQdiBIJjiWoF9+/Yl60uXTvhN4i/klmzOjWWn6h9//HFy39x3AK5cuZKs58bpFy6sPRky9+d+//33k/Xc9NvDhw/XrD344IPJfW9HHNmBIAg7EARhB4Ig7EAQhB0IgrADQRB2IAjG2Svw7rvvJuu5se7ceHLucs3Lli2rWRscHEzue+TIkWR97dq1yfrs2bOT9UuXLtWsLViwILlvzujoaLJ+7NixUs9/u+HIDgRB2IEgCDsQBGEHgiDsQBCEHQiCsANBMM5egUcfTV9F+8UXX0zWyy6rnBtvTsmNk+fk5pTn5qyn5Obx53rv7u6u+7VvRxzZgSAIOxAEYQeCIOxAEIQdCIKwA0EQdiCIyazPvlLSbyQtleSSdrn7L8zsGUn/IGmkeOjT7n6gUY22sw0bNiTrH330UbKeG2fPXXc+Nx6dklo/fTJSc+kl6cKFCzVrM2bMSO6buw5A7n19/PHHk/VoJvOlmuuSfuLuR8xsnqQ3zOyVovZzd/+XxrUHoCqTWZ99SNJQcX/UzN6WtLzRjQGo1pQ+w5lZl6S1kv5cbHrSzN40s91mNuE6P2a2zcx6zax3ZGRkoocAaIJJh93M5kr6g6Qfu/tFSb+U9E1J3Ro78v90ov3cfZe797h7z5IlS8p3DKAukwq7mc3QWNB/6+57Jcndh939hrvflPQrSesa1yaAsrJht7EpV89LetvdfzZue+e4h31P0vHq2wNQlcmcjV8v6QeSjpnZ0WLb05I2m1m3xobj+iX9sAH93RZOnz6drN99992lnn/WrFl173v+/PlkffXq1cl67lLVly9frlm7fv16ct/c0FvuUtHz589P1qOZzNn4P0maaEJ1yDF14OuKb9ABQRB2IAjCDgRB2IEgCDsQBGEHguBS0k3Q2dmZrOemuPb19dVdf/nll5P73nfffcl6zvbt25P1M2fO1Kw99thjyX25FHS1OLIDQRB2IAjCDgRB2IEgCDsQBGEHgiDsQBCWG+Ot9MXMRiR9MG7TYknnmtbA1LRrb+3al0Rv9aqyt79y9wmv/9bUsH/lxc163b2nZQ0ktGtv7dqXRG/1alZvfIwHgiDsQBCtDvuuFr9+Srv21q59SfRWr6b01tJ/swNonlYf2QE0CWEHgmhJ2M3sITP7HzN7z8yeakUPtZhZv5kdM7OjZtbb4l52m9lZMzs+btsiM3vFzPqK2wnX2GtRb8+Y2WDx3h01s4db1NtKM/ujmZ0ws7fM7EfF9pa+d4m+mvK+Nf3f7GY2TdK7kv5O0oCkw5I2u/uJpjZSg5n1S+px95Z/AcPMvi3pkqTfuPt9xbZ/lnTB3Z8t/ke50N3/sU16e0bSpVYv412sVtQ5fplxSY9I+nu18L1L9PWomvC+teLIvk7Se+5+0t2vSvqdpI0t6KPtuftrki7csnmjpD3F/T0a+8vSdDV6awvuPuTuR4r7o5I+X2a8pe9doq+maEXYl0s6Ne73AbXXeu8u6aCZvWFm21rdzASWuvtQcf+MpKWtbGYC2WW8m+mWZcbb5r2rZ/nzsjhB91X3u/u3JG2Q9ETxcbUt+di/wdpp7HRSy3g3ywTLjH+hle9dvcufl9WKsA9KWjnu9xXFtrbg7oPF7VlJ+9R+S1EPf76CbnF7tsX9fKGdlvGeaJlxtcF718rlz1sR9sOSVpvZN8ysQ9L3Je1vQR9fYWZzihMnMrM5kr6r9luKer+kLcX9LZJeamEvX9Iuy3jXWmZcLX7vWr78ubs3/UfSwxo7I/+/kv6pFT3U6GuVpL8UP2+1ujdJL2jsY901jZ3b2CrpLkmHJPVJ+m9Ji9qot3+XdEzSmxoLVmeLertfYx/R35R0tPh5uNXvXaKvprxvfF0WCIITdEAQhB0IgrADQRB2IAjCDgRB2IEgCDsQxP8BlEcjV2Xp6ikAAAAASUVORK5CYII=",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "y_train[4] = 0\n"
     ]
    }
   ],
   "source": [
    "if VERBOSE:\n",
    "    NUM_EXAMPLES = 5\n",
    "\n",
    "    for i in range(NUM_EXAMPLES):\n",
    "        print('X_train[%d]:' % i)\n",
    "        plt.imshow(X_train[i], cmap = cm.Greys)\n",
    "        plt.show()\n",
    "        print('y_train[%d] =' % i, y_train[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On a également besoin d'applatir les données."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "X_train.shape = (60000, 784)\n",
      "X_test.shape = (10000, 784)\n"
     ]
    }
   ],
   "source": [
    "X_train, X_test = X_train.reshape(X_train.shape[0], X_train.shape[1] * X_train.shape[2]), X_test.reshape(X_test.shape[0], X_test.shape[1] * X_test.shape[2])\n",
    "\n",
    "assert len(X_train.shape) == 2\n",
    "assert len(X_test.shape) == 2\n",
    "\n",
    "if (VERBOSE):\n",
    "    print('X_train.shape =', X_train.shape)\n",
    "    print('X_test.shape =', X_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Réductions de dimension\n",
    "\n",
    "Afin de comparer un large panel de classifications, on va mettre en place deux réductions de dimensions qui seront appliquées une à une à chaque modèle de classification. Cela permettra de tenter d'améliorer nos accuracy tout en gagnant du temps de calcul.\n",
    "\n",
    "- Analyse en composantes principales\n",
    "- Analyse linéaire discriminante"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyse en composantes principales"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On commence par centrer les données."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X = np.concatenate((X_train, X_test))\n",
    "z = scaler.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut maintenant commencer l'ACP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "n_components = 784\n"
     ]
    }
   ],
   "source": [
    "pca = PCA()\n",
    "coord = pca.fit_transform(z)\n",
    "\n",
    "if VERBOSE:\n",
    "    print('n_components =', pca.n_components_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fions-nous au critère du coude pour définir le nombre d'axes que l'on va retenir. Pour ce faire, nous allons construire l'éboulis des valeurs propres grâce à une fonction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_scree_plot(n_components, explained_variance_ratio):\n",
    "    plt.grid()\n",
    "    plt.plot(np.arange(1, n_components+1), explained_variance_ratio)\n",
    "    plt.title('Scree Plot')\n",
    "    plt.ylabel('Eigen Values')\n",
    "    plt.xlabel('Factor Number')\n",
    "    plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut ainsi afficher notre éboulis et tenter de trouver le décrochement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAh50lEQVR4nO3deZxcZZ3v8c+3O+kshCwQjEDYQRHuRSQRBBQbRxA3GO7ANbgAM2iu94qjjqK4vFAZ9bpcN5RRGWVUHAgIjmYwggi0oyIQoiAERMKaBGTLRmdP9+/+cZ7qPl19ulNd3aerSH/fr9SrT531W92V/vXznFPPUURgZmZWraXRAczMrDm5QJiZWSEXCDMzK+QCYWZmhVwgzMyskAuEmZkVcoEwa3KSzpb020bnsLHHBcLGHEmvlHSLpLWSVkn6naSXNzjTpyRtldQpaU3Kd3Qd++mQ9M4yMtrY4wJhY4qkqcC1wDeAXYA9gU8Dm4e4n3Ejn44rI2IKsBvwW+AnklTCccxq4gJhY82LACLiiojoioiNEfHLiPhTZQVJ75J0n6TnJN0r6Yg0/xFJH5H0J2C9pHGSXpH+2l8j6S5J7bn9TJP0PUlPSFop6TOSWrcXMCK2Aj8AXgjsWr1c0jGSFqcW0GJJx6T5nwVeBXwztUS+OZxvlJkLhI01fwG6JP1A0uslzcgvlHQ68CngTGAqcDLwbG6VM4A3AtOBWcDPgc+QtUY+BFwjabe07veBbcCBwMuAE4Htdv9ImgCcDSyPiGeqlu2SjnkRWfH4CvBzSbtGxMeB3wDnRsSUiDh3+98Os4G5QNiYEhHrgFcCAfwr8LSkhZJmpVXeCXwxIhZHZllEPJrbxUURsTwiNgJvBxZFxKKI6I6IG4A7gDek/b0BeH9ErI+Ip4CvAvMGifc/Ja0BlgNzgFML1nkj8EBEXBYR2yLiCuDPwJvr+46YDayMflSzphYR95H9hY6kg4EfAV8jax3sBTw4yObLc9P7AKdLyv9yHg/cnJaNB57InUZoqdq+2lUR8fbtxN8DeLRq3qNk51LMRpQLhI1pEfFnSd8H/leatRw4YLBNctPLgcsi4l3VK0nanezE98yI2DZCcQEeJys+eXsD1xXkMxsWdzHZmCLpYEkflDQ7Pd+LrOVwa1rlu8CHJM1R5kBJ1b+QK34EvFnS6yS1SpooqV3S7Ih4Avgl8GVJUyW1SDpA0quH+RIWAS+S9NZ0kvwtwCFkV2YBPAnsP8xjmAEuEDb2PAccBdwmaT1ZYbgH+CBARPwY+CxweVr3p2QnoPuJiOXAKcDHgKfJWhTn0fv/6kygDbgXWA1cDew+nPAR8SzwppT3WeDDwJtyJ7O/DpwmabWki4ZzLDP5hkFmZlbELQgzMyvkAmFmZoVcIMzMrJALhJmZFdphPgcxc+bM2Hfffevadv369ey0004jG2iEOFt9nK0+zjZ0zZoLasu2ZMmSZyJit8KFEbFDPObMmRP1uvnmm+vetmzOVh9nq4+zDV2z5oqoLRtwRwzwe9VdTGZmVsgFwszMCrlAmJlZIRcIMzMr5AJhZmaFXCDMzKyQC4SZmRUa8wVi/eZtXPPAFv742OpGRzEzaypjvkBs2trFfz64lbtXrm10FDOzpjLmC0TlfsHd3b4vhplZngtE+uryYGbW15gvEC2pBeEb65mZ9TXmC0SlCdHtCmFm1seYLxDS9tcxMxuLxnyBcBeTmVmxMV8gKg0IdzGZmfXlApEqhMuDmVlfY75AuIvJzKzYmC8QFe5iMjPra8wXCF/FZGZWbMwXiN4uJrcgzMzyxnyB6L2KqaExzMyajguET1KbmRUa8wWipecyV1cIM7O8UguEpJMk3S9pmaTzC5b/k6R7Jf1J0o2S9sktO0vSA+lxVokZAXcxmZlVK61ASGoFLgZeDxwCnCHpkKrV/gjMjYjDgKuBL6ZtdwE+CRwFHAl8UtKMsrIC7mMyM6tSZgviSGBZRDwUEVuABcAp+RUi4uaI2JCe3grMTtOvA26IiFURsRq4ATiprKDCn6Q2M6s2rsR97wkszz1fQdYiGMg5wC8G2XbP6g0kzQfmA8yaNYuOjo66gorg4UcepaPjibq2L1NnZ2fdr6tszlYfZ6tPs2Zr1lww/GxlFoiaSXo7MBd49VC2i4hLgEsA5s6dG+3t7fUd//qfs/fee9PefnBd25epo6ODel9X2ZytPs5Wn2bN1qy5YPjZyuxiWgnslXs+O83rQ9JrgY8DJ0fE5qFsO1LcxWRm1l+ZBWIxcJCk/SS1AfOAhfkVJL0M+A5ZcXgqt+h64ERJM9LJ6RPTvHLIYzGZmVUrrYspIrZJOpfsF3srcGlELJV0IXBHRCwEvgRMAX6cLjd9LCJOjohVkv6ZrMgAXBgRq8rKKnATwsysSqnnICJiEbCoat4FuenXDrLtpcCl5aXrJbk+mJlVG/OfpIasBdHtT8qZmfXhAoFPUpuZFXGBIHUxuUKYmfXhApH4KiYzs75cIOi9J4SZmfVygaDSxeQWhJlZngsE6Som1wczsz5cIKhcxeQKYWaW5wKBr2IyMyviAgGA3MVkZlbFBYKsBeGPypmZ9eUCQToH4fpgZtaHCwSVq5hcIczM8lwg8ElqM7MiLhB4sD4zsyIuEIm7mMzM+nKBIF3F5PpgZtaHCwTuYjIzK+ICQdaCcBeTmVlfLhD4cxBmZkVcIHAXk5lZERcIAHcxmZn14wJBuqOc64OZWR8uEKRPUrtCmJn14QJBGoupu9EpzMyaiwsEvqOcmVkRFwhAki9zNTOr4gKR+I5yZmZ9uUCQrmJyF5OZWR8uEPh+EGZmRVwg8B3lzMyKuEDgoTbMzIq4QOAuJjOzIi4QibuYzMz6coGgchWTmZnluUDgLiYzsyKlFghJJ0m6X9IySecXLD9O0h8kbZN0WtWyLkl3psfCUnPiLiYzs2rjytqxpFbgYuAEYAWwWNLCiLg3t9pjwNnAhwp2sTEiDi8rX55bEGZm/ZVWIIAjgWUR8RCApAXAKUBPgYiIR9Kyho6l6sH6zMz6U5T0p3PqMjopIt6Znr8DOCoizi1Y9/vAtRFxdW7eNuBOYBvw+Yj4acF284H5ALNmzZqzYMGCurJ+5pZOWltb+ehRk+ravkydnZ1MmTKl0TEKOVt9nK0+zZqtWXNBbdmOP/74JRExt2hZmS2I4donIlZK2h+4SdLdEfFgfoWIuAS4BGDu3LnR3t5e14E+f/svmDp1Ou3tRw8384jr6Oig3tdVNmerj7PVp1mzNWsuGH62Mk9SrwT2yj2fnebVJCJWpq8PAR3Ay0YyXJ67mMzM+iuzQCwGDpK0n6Q2YB5Q09VIkmZImpCmZwLHkjt3MdIkD/dtZlattAIREduAc4HrgfuAqyJiqaQLJZ0MIOnlklYApwPfkbQ0bf4S4A5JdwE3k52DKK9AZHnL2r2Z2fPSds9BSDoAWBERmyW1A4cBP4yINdvbNiIWAYuq5l2Qm15M1vVUvd0twH/f3v5HipA7mMzMqtTSgrgG6JJ0INkJ4b2Ay0tNNdrcxWRm1k8tBaI7dRedCnwjIs4Ddi831ugS+JNyZmZVaikQWyWdAZwFXJvmjS8v0ujzSWozs/5qKRB/DxwNfDYiHpa0H3BZubFGly9zNTPrb7snqSPiXkkfAfZOzx8GvlB2sNHmHiYzs76224KQ9GayIS+uS88PL3t01dHW4i4mM7N+auli+hTZwHtrACLiTmD/0hI1iD8HYWbWV00nqSNibdW8ho6+OtJ8Rzkzs/5qGaxvqaS3Aq2SDgL+Ebil3FijK7uKyS0IM7O8WloQ7wUOBTYDVwDrgPeXmGnUZUNtNDqFmVlzqeUqpg3Ax9Njh+Q7ypmZ9VfLWEw3Q/8PCUTEa0pJ1AC+J7WZWX+1nIPI3y96IvB3ZHd527G4PpiZ9VFLF9OSqlm/k3R7SXkaokWuD2Zm1WrpYtol97QFmANMKy1Rg7iLycysr1q6mJaQ/YEtsq6lh4Fzygw12nwVk5lZf7V0Me03GkEaSZIH6zMzqzJggZD0PwbbMCJ+MvJxGkNA9w712XAzs+EbrAXx5kGWBbDDFAgzM+tvwAIREX8/mkEaqUUerM/MrFotJ6mR9Eay4TYmVuZFxIVlhWoED/dtZtZXLfeD+DbwFrIxmQScDuxTcq5RJfmOcmZm1WoZrO+YiDgTWB0Rnya7/eiLyo01unyZq5lZf7UUiI3p6wZJewBbgd3LizT6srGYGp3CzKy51HIO4lpJ04EvAX8gu4LpX8sMNeoEHmzDzKyvwT4HsQi4HPhqRHQC10i6FphYcIe557UW3MVkZlZtsC6m7wBvBB6SdJWkU4HY0YoDAL6jnJlZPwMWiIj4WUScAewLXAOcCTwm6d8knTBK+UaFcAeTmVm17Z6kjogNEXFlRJwKnAgcDlxXdrDR5KuYzMz6q+VzELMkvVfS74CfAtcDR5QdbDTJXUxmZv0MdpL6XcAZwIvJupjOi4hbRivYqHN9MDPrY7DLXI8G/i9wY0Ts0GOdtuD6YGZWbbDB+v5hNIM0lLuYzMz6qeWT1Ds8n6Q2M+vPBQLfUc7MrEhNBUJSq6Q9JO1dedS43UmS7pe0TNL5BcuPk/QHSdsknVa17CxJD6THWbW9nPp4LCYzs/62OxaTpPcCnwSeBConqwM4bDvbtQIXAycAK4DFkhZGxL251R4DzgY+VLXtLumYc9OxlqRtV9fwmurjAmFm1kctg/W9D3hxRDw7xH0fCSyLiIcAJC0ATgF6CkREPJKWVV8l9TrghohYlZbfAJwEXDHEDDVp8f0gzMz6qaVALAfqGX9pz7RtxQrgqGFsu2f1SpLmA/MBZs2aRUdHRx0xYcuWLXR1q+7ty9TZ2dmUucDZ6uVs9WnWbM2aC4afrZYC8RDQIennwObKzIj4St1HHSERcQlwCcDcuXOjvb29rv1c85dfAlupd/sydXR0NGUucLZ6OVt9mjVbs+aC4Wer5ST1Y8ANQBuwc+6xPSuBvXLPZ6d5tRjOtkOW3XLUzMzyttuCSLcZRdLkiNgwhH0vBg6StB/ZL/d5wFtr3PZ64HOSZqTnJwIfHcKxh8SfgzAz66+WwfqOlnQv8Of0/KWS/mV720XENuBcsl/29wFXRcRSSRdKOjnt6+WSVgCnA9+RtDRtuwr4Z7Iisxi4sHLCukzhKmFm1qOWcxBfI7uqaCFARNwl6bhadh4Ri4BFVfMuyE0vJus+Ktr2UuDSWo4zXC2qHDPrbjIzsxo/KBcRy6tmdZWQpeE8HpOZWa+aLnOVdAwQksaTfS7ivnJjja5Kq8HlwcysVy0tiHcD7yH7HMJKsjvKvafETKOu0qvkBoSZWa9armJ6BnjbKGRpmEqBcBeTmVmvWsZiuqhg9lrgjoj42chHagCfmDYz66eWLqaJZN1KD6THYWRXHp0j6WulJRtFlW+CGxBmZr1qOUl9GHBsRHQBSPoW8BvglcDdJWYbPakF4S4mM7NetbQgZgBTcs93AnZJBWNz8SbPL0oVwuXBzKxXLS2ILwJ3Suog+1v7OLJhMHYCflVitlHTexWTS4SZWUUtVzF9T9Iisvs7AHwsIh5P0+eVlmwUqaeLqbE5zMyayYBdTJIOTl+PAHYnuz/DcuCFad6OxwXCzKzHYC2IDwLvAr5csCyA15SSqAF6rmJyhTAz6zFggYiId6Wvx49enAZxF5OZWT+DdTF9ODd9etWyz5UZarT5JLWZWX+DXeY6LzddfbOek0rI0jAerM/MrL/BCoQGmC56/rzmsZjMzPobrEDEANNFz3cMO+arMjOry2BXMb1U0jqyP7AnpWnS84mlJxtFLe5iMjPrZ7CrmFpHM0gzcBeTmVmvmm45uqPzDYPMzPpzgcBXMZmZFXGBIHcVkz8pZ2bWwwXCzMwKuUCQu4rJDQgzsx4uEDm+isnMrJcLBCD5jnJmZtVcIPBgfWZmRVwgyI/F1NAYZmZNxQUCckMPukKYmVW4QJC7o5zrg5lZDxcI8B3lzMwKuECQO0ntLiYzsx4uEHiwPjOzIi4Q9A7W5w/KmZn1coHIcX0wM+vlAgFMGpc1IdZt2trgJGZmzaPUAiHpJEn3S1om6fyC5RMkXZmW3yZp3zR/X0kbJd2ZHt8uM+euE7MC8cSaTWUexszseWWwe1IPi6RW4GLgBGAFsFjSwoi4N7faOcDqiDhQ0jzgC8Bb0rIHI+LwsvLlzUgF4vE1G0fjcGZmzwtltiCOBJZFxEMRsQVYAJxStc4pwA/S9NXA36gyct4oamsVMyaP58nn3IIwM6tQWQPUSToNOCki3pmevwM4KiLOza1zT1pnRXr+IHAUMAVYCvwFWAd8IiJ+U3CM+cB8gFmzZs1ZsGBBXVk7Ozv51JIWXjSjlfmHTahrH2Xp7OxkypQpjY5RyNnq42z1adZszZoLast2/PHHL4mIuUXLSutiGqYngL0j4llJc4CfSjo0ItblV4qIS4BLAObOnRvt7e11Hayjo4MZO4tpu0yhvX3OMKOPrI6ODup9XWVztvo4W32aNVuz5oLhZyuzi2klsFfu+ew0r3AdSeOAacCzEbE5Ip4FiIglwIPAi0rMyuS2VjZs6SrzEGZmzytlFojFwEGS9pPUBswDFlatsxA4K02fBtwUESFpt3SSG0n7AwcBD5WYlUltrWx0gTAz61FaF1NEbJN0LnA90ApcGhFLJV0I3BERC4HvAZdJWgasIisiAMcBF0raCnQD746IVWVlBZg0vpVnOreUeQgzs+eVUs9BRMQiYFHVvAty05uA0wu2uwa4psxs1Sa3jWPDlg2jeUgzs6bmT1In7mIyM+vLBSKZ3NbKhq0uEGZmFS4QyaTxbkGYmeW5QCST2lrZvK2bLt9WzswMcIHoMbmtFYCN7mYyMwNcIHpMassu6NqwZVuDk5iZNQcXiGTS+KwFsWlLd4OTmJk1BxeIpNLFtGGrWxBmZuAC0WNSpUD4SiYzM8AFosfkni4mFwgzM3CB6OEWhJlZXy4QSe85CBcIMzNwgeix284TAVi52velNjMDF4ge0yaNZ9bUCTzw1HONjmJm1hRcIHL2nzmFh59Z3+gYZmZNwQUi54XTJvLUus2NjmFm1hRcIHJeMHUCTz+3mQgP2Gdm5gKR84KdJ7Klq5s1G7Y2OoqZWcO5QOTMnNIGwDOd7mYyM3OByJkxOSsQaze6BWFm5gKRM33yeAB3MZmZ4QLRx/RJWQtijVsQZmYuEHnTeloQWxqcxMys8VwgcnaeMI4WuYvJzAxcIPpoaREv2Hkif123qdFRzMwazgWiyh7TJ/LEWg/YZ2bmAlFl9+mTPKKrmRkuEP0c9IIpPLpqA6vX+0S1mY1tLhBVXnXQbkTA7x96ttFRzMwaygWiyn/bcyrjW8VdK9Y0OoqZWUO5QFSZMK6VI/aewYLbl/veEGY2prlAFPjSaS+lqzv41MKldHd76G8zG5tcIArsvetkzjx6H379l6f5xM/uYa0/OGdmY9C4RgdoVue97sWsWr+Fy297jFsfepYf/sORzJ4xudGxzMxGjVsQA5DE5//uMC55xxyWr9rAiV/9Ly742T385oGnfQmsmY0JbkFsx4mHvpBffuDVfOPGB1hw+3J++PtHAdh92kQO3WMqh+wxjUP3mMqhe0xlz+mTkNTgxGZmI6PUAiHpJODrQCvw3Yj4fNXyCcAPgTnAs8BbIuKRtOyjwDlAF/CPEXF9mVkHs9/MnfjKWw7nkycfyj0r17L08bUsfXwdSx9fx01/forKeeydJ4xj710ns+f0Sey28wR23amNGTu1MXXieCa1tWaP8dljclsrE8dn8ya3tTJxXCstLS4uZtY8SisQklqBi4ETgBXAYkkLI+Le3GrnAKsj4kBJ84AvAG+RdAgwDzgU2AP4laQXRURXWXlrMW3SeI49cCbHHjizZ97GLV3c99esWDzw5HM8tmoDjzy7niWPrmb1hi0M5SKoCeNasoIxvpWJqZhs3biR7z14W09RmZQKy7gW0dIixrWIVmXTrRKtrelri2iRGNeafW1t6Z3f2pJbv6XyIFu/pYWWFmiVelpDElRKV28DSSxb08XUx1bnlqX1c+sJVW3Xd1nPdNG8gmP2X6Z+6wnx9IZulq/a0OeY+XzbOybqn10Mvg/6vZbi9Td3BRu3dA34PallH26p2mgoswVxJLAsIh4CkLQAOAXIF4hTgE+l6auBbyp7558CLIiIzcDDkpal/f2+xLx1mdSWfW7iiL1n9FvW3R2s2biVzk3b2LB1Gxu3dLFxa1dNXzds7WLTli4e39TJc5u28fRzm9m4tYsNW7L527qDrgi6u4NtjbwU99ZbGnfs7fmvmxudYGA3XDeiu+tbxAcu1PQpPH2XSdDd1UXrTdf1LUoFRbmWY0LRer3HLCr2g9m0aRMTb72p3+seyKDLKF44+DbFNm7cyOQ7Ooa0zfYWDrZd0ffpJbtP5RtnvGywo9WlzAKxJ7A893wFcNRA60TENklrgV3T/Furtt2z+gCS5gPzAWbNmkVHR0ddQTs7O+veth6twJT06DGOwp9G58wupkypXGarAVfsjqA76P8gK1TdFCyLgu2Arm6olJyI3unKVGXepk2bmDhxYm5Jtqzv2n3nb3de1UTknxbN69lX35SbNm1mwoQJ/fY70P63l6fwdeXWry7Rfb9vlfWymZu3bKGtra3G11W9jwHmFWSq3sfgrysIYOuWYPx4Dfq6Kuv23VdU/vU9VtG8AV7DQCrrbmvrZty42i45H3R/Ay0dZKPB9re1pZvx4/rfIqD6Nda6v8EMtF08t7nwd9hwf7c9r09SR8QlwCUAc+fOjfb29rr209HRQb3bls3Z6uNs9XG2oWvWXDD8bGVe5roS2Cv3fHaaV7iOpHHANLKT1bVsa2ZmJSqzQCwGDpK0n6Q2spPOC6vWWQiclaZPA26KrK9gITBP0gRJ+wEHAbeXmNXMzKqU1sWUzimcC1xP1u1+aUQslXQhcEdELAS+B1yWTkKvIisipPWuIjuhvQ14T6OvYDIzG2tKPQcREYuARVXzLshNbwJOH2DbzwKfLTOfmZkNzENtmJlZIRcIMzMr5AJhZmaFXCDMzKyQYrCP+z2PSHoaeLTOzWcCz4xgnJHkbPVxtvo429A1ay6oLds+EbFb0YIdpkAMh6Q7ImJuo3MUcbb6OFt9nG3omjUXDD+bu5jMzKyQC4SZmRVygchc0ugAg3C2+jhbfZxt6Jo1Fwwzm89BmJlZIbcgzMyskAuEmZkVGvMFQtJJku6XtEzS+Q04/qWSnpJ0T27eLpJukPRA+jojzZeki1LWP0k6osRce0m6WdK9kpZKel8TZZso6XZJd6Vsn07z95N0W8pwZRpmnjRs/JVp/m2S9i0rWy5jq6Q/Srq2mbJJekTS3ZLulHRHmtfwn2k63nRJV0v6s6T7JB3dDNkkvTh9vyqPdZLe3wzZ0vE+kP4f3CPpivT/Y2TebxExZh9kw5A/COwPtAF3AYeMcobjgCOAe3Lzvgicn6bPB76Qpt8A/ILs3qOvAG4rMdfuwBFpemfgL8AhTZJNwJQ0PR64LR3zKmBemv9t4H+n6f8DfDtNzwOuHIWf6z8BlwPXpudNkQ14BJhZNa/hP9N0vB8A70zTbcD0ZsmWy9gK/BXYpxmykd2K+WFgUu59dvZIvd9K/4Y28wM4Grg+9/yjwEcbkGNf+haI+4Hd0/TuwP1p+jvAGUXrjULGnwEnNFs2YDLwB7L7nT8DjKv+2ZLdk+ToND0uracSM80GbgReA1ybflE0S7ZH6F8gGv4zJbub5MPVr70ZslXlORH4XbNkIysQy4Fd0vvnWuB1I/V+G+tdTJVvbsWKNK/RZkXEE2n6r8CsNN2QvKkZ+jKyv9SbIlvqwrkTeAq4gawluCYithUcvydbWr4W2LWsbMDXgA8D3en5rk2ULYBfSloiaX6a1ww/0/2Ap4F/S11z35W0U5Nky5sHXJGmG54tIlYC/w94DHiC7P2zhBF6v431AtH0Iiv1DbsWWdIU4Brg/RGxLr+skdkioisiDif7a/1I4OBG5Kgm6U3AUxGxpNFZBvDKiDgCeD3wHknH5Rc28Gc6jqyr9VsR8TJgPVm3TTNkAyD1458M/Lh6WaOypfMep5AV2D2AnYCTRmr/Y71ArAT2yj2fneY12pOSdgdIX59K80c1r6TxZMXh3yPiJ82UrSIi1gA3kzWjp0uq3CUxf/yebGn5NODZkiIdC5ws6RFgAVk309ebJFvlL04i4ingP8iKazP8TFcAKyLitvT8arKC0QzZKl4P/CEinkzPmyHba4GHI+LpiNgK/ITsPTgi77exXiAWAwelM/5tZM3HhQ3OBFmGs9L0WWT9/5X5Z6arJF4BrM01cUeUJJHdM/y+iPhKk2XbTdL0ND2J7NzIfWSF4rQBslUynwbclP7iG3ER8dGImB0R+5K9n26KiLc1QzZJO0nauTJN1p9+D03wM42IvwLLJb04zfobsnvSNzxbzhn0di9VMjQ622PAKyRNTv9nK9+3kXm/lX1Sp9kfZFcc/IWsD/vjDTj+FWR9h1vJ/oo6h6xP8EbgAeBXwC5pXQEXp6x3A3NLzPVKsibzn4A70+MNTZLtMOCPKds9wAVp/v7A7cAysm6ACWn+xPR8WVq+/yj9bNvpvYqp4dlShrvSY2nl/d4MP9N0vMOBO9LP9afAjCbKthPZX9rTcvOaJdungT+n/wuXARNG6v3moTbMzKzQWO9iMjOzAbhAmJlZIRcIMzMr5AJhZmaFXCDMzKyQC4TtkCR1VY3Aue8Qt/9bSYcMM8PZkrolHZabd89Qswyy/86R2I/ZQFwgbEe1MSIOzz0eGeL2f0s2em3Ncp9czVsBfHyIxy7dAFnN+nCBsDFB0hRJN0r6g7L7IZySW3ZmGrf/LkmXSTqGbMydL6XWxwGSDpd0a1rvP9Q79n+HpK8pu7fC+woOfS1waO4TwvlMnbnp0yR9P01/X9K30vEektSu7L4h91XWyW33VWX3ArhR0m5p3gGSrksD8v1G0sG5/X5b0m1kQ1WbDcp/RdiOapKy0V4hG0b6dODUiFgnaSZwq6SFZK2ETwDHRMQzknaJiFVp2bURcTWApD8B742IX0u6EPgk8P60/7aImDtAjm6yX8Yfo3eIg1rMIBtf6mSy4RGOBd4JLJZ0eETcSfbp3jsi4gOSLkiZziW7Uf27I+IBSUcB/0I2JhRk4/IcExFdQ8hiY5QLhO2oNkY22ivQM/Dg55SNXtpNNuzxLLJfnD+OiGcAImJV9Y4kTQOmR8Sv06wf0HdEzyu3k+Vy4OOS9htC/v+MiJB0N/BkRNydsiwlu3/Inel1VI79I+AnykbfPQb4cTY0D5ANvVDxYxcHq5ULhI0VbwN2A+ZExFZlo61OHKF9rx9sYURsk/Rl4CPVi3LT1Vk2p6/duenK84H+3wZZt/GafHEcSlazPJ+DsLFiGtl9GrZKOp7slpEANwGnS9oVsvszp/nPkd1qlYhYC6yW9Kq07B3Arxma75MNzbxbbt6Tkl4iqQU4dYj7g+z/b2XEzrcCv43snh0PSzodeu6P/NI69m3mAmFjxr8Dc1OXzZlko18SEUuBzwK/lnQXUBnafAFwnrK7mx1Adv7gS+lcxOHAhUM5eERsAS4CXpCbfT7ZSexbyEb0Har1wJGS7iHrKqtkehtwTno9S8luKGM2ZB7N1czMCrkFYWZmhVwgzMyskAuEmZkVcoEwM7NCLhBmZlbIBcLMzAq5QJiZWaH/Dz70IZmZSuanAAAAAElFTkSuQmCC",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_scree_plot(pca.n_components_, pca.explained_variance_ratio_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En l'état, l'éboulis n'est pas vraiment lisible. Cependant, on remarque qu'à partir d'un certain facteur, la valeur propre est plutôt constante. On peut donc tronquer cette partie, ce qui nous permettra de « zoomer » sur le décrochement que l'on cherche."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAnuElEQVR4nO3deXxddZ3/8dfn3uxJ0yZNmpYm3dhaQKy0LIJLRMWiI9UZEFAUZlDGGXF0XEZH54eKy7iM4z4oKm6jVhaVihVEJcgyQAuU0oWlLaVp6UJpS5suaZbP749zkt7e3qQ3TU7uvee+n49HHjnrvZ9vb3vfPdv3a+6OiIhIukSuCxARkfykgBARkYwUECIikpECQkREMlJAiIhIRgoIERHJSAEhkufM7AozuzfXdUjxUUBI0TGzV5jZ/Wb2opltN7P7zOz0HNf0aTPrMrMOM9sZ1vfyo3idNjN7dxQ1SvFRQEhRMbNa4DbgW0A9MBn4DNA5xNcpGfnq+JW71wCNwL3Ar83MIngfkawoIKTYnADg7r909x533+fuf3T3ZX0bmNl7zGyVme02s5Vmdlq4fJ2ZfczMlgF7zKzEzM4K/7e/08weM7PWlNcZa2Y/NLNNZrbRzD5nZskjFejuXcBPgInA+PT1Zna2mS0Oj4AWm9nZ4fLPA68Evh0eiXx7OH9QIgoIKTZPAT1m9hMzO9/M6lJXmtlFwKeBdwG1wAXACymbXAq8CRgHNAG/Bz5HcDTyEeAWM2sMt/0x0A0cB7wMOA844ukfMysHrgDa3X1b2rr68D2/SRAe/w383szGu/sngXuAq929xt2vPvIfh8jAFBBSVNx9F/AKwIHvA8+b2UIzawo3eTfwZXdf7IHV7v5sykt8093b3X0fcBmwyN0XuXuvu98JLAHeGL7eG4EPuvsed98KfA24ZJDy3mZmO4F2YA7w1gzbvAl42t1/5u7d7v5L4AngzUf3JyIysCjOo4rkNXdfRfA/dMxsJvC/wNcJjg5agDWD7N6eMj0VuMjMUr+cS4G7wnWlwKaUywiJtP3T3ejulx2h/GOAZ9OWPUtwLUVkRCkgpKi5+xNm9mPgH8NF7cCxg+2SMt0O/Mzd35O+kZlNIrjw3eDu3SNULsBzBOGTagpwe4b6RIZFp5ikqJjZTDP7sJk1h/MtBEcOD4Sb/AD4iJnNscBxZpb+hdznf4E3m9kbzCxpZhVm1mpmze6+Cfgj8FUzqzWzhJkda2avHmYTFgEnmNnbw4vkFwMnEdyZBbAFmDHM9xABFBBSfHYDZwIPmtkegmBYDnwYwN1vAj4P/CLc9rcEF6AP4+7twHzgE8DzBEcUH+Xgv6t3AWXASmAHcDMwaTjFu/sLwN+E9b4A/BvwNykXs78BXGhmO8zsm8N5LxHTgEEiIpKJjiBERCQjBYSIiGSkgBARkYwUECIiklFsnoNoaGjwadOmHbJsz549VFdX56agiMStTXFrD8SvTXFrD8SvTcNpz8MPP7zN3RszrYtNQEybNo0lS5YcsqytrY3W1tbcFBSRuLUpbu2B+LUpbu2B+LVpOO0xs/Qn8/vpFJOIiGSkgBARkYwUECIikpECQkREMlJAiIhIRgoIERHJSAEhIiIZFX1AvLi3i2/86WmWbdiZ61JERPJKbB6UO1qJBHztT09RXprg1OZxuS5HRCRvFP0RxJiKUhrHlLP2+Y5clyIikleKPiAApjdU88y2PbkuQ0QkrygggGMbFRAiIukUEARHENs6DvDivq5clyIikjcUEMD0hhoAHUWIiKRQQAAzGoN+1HWhWkTkIAUE0FJXRTJhOoIQEUmhgADKShK01FWyVgEhItJPARGa0VjD2ucVECIifRQQoekN1azbtofeXs91KSIieUEBEZrRWM2+rh4279qf61JERPKCAiI0vSG4k0kXqkVEAgqI0IzwWQhdqBYRCSggQk215VSVJfUshIhISAERMjN12icikiLSgDCzeWb2pJmtNrOPZ1j/ITNbaWbLzOzPZjY1Zd3lZvZ0+HN5lHX20a2uIiIHRRYQZpYEvgOcD5wEXGpmJ6Vt9igw191PBW4GvhzuWw98CjgTOAP4lJnVRVVrn+kN1WzYsZfO7p6o30pEJO9FeQRxBrDa3de6+wFgATA/dQN3v8vd94azDwDN4fQbgDvdfbu77wDuBOZFWCsAMxqq6XVo3773yBuLiMRclEOOTgbaU+Y3EBwRDORK4A+D7Ds5fQczuwq4CqCpqYm2trZD1nd0dBy2bDA7XgyOHG6960HmNOXnaKxDbVO+i1t7IH5tilt7IH5tiqo9efEtaGaXAXOBVw9lP3e/HrgeYO7cud7a2nrI+ra2NtKXDWbO/i4+839/pHridFpffexQShk1Q21TvotbeyB+bYpbeyB+bYqqPVGeYtoItKTMN4fLDmFmrwM+CVzg7p1D2XekaXxqEZGDogyIxcDxZjbdzMqAS4CFqRuY2cuA7xGEw9aUVXcA55lZXXhx+rxwWeR0q6uISCCygHD3buBqgi/2VcCN7r7CzK41swvCzb4C1AA3mdlSM1sY7rsd+CxByCwGrg2XRW6GAkJEBIj4GoS7LwIWpS27JmX6dYPsewNwQ3TVZTajsZpti4PxqcdWlo7224uI5A09SZ1G41OLiAQUEGk0PrWISEABkUbjU4uIBBQQaTQ+tYhIQAGRgTrtExFRQGSk8alFRBQQGU1v0PjUIiIKiAz67mTShWoRKWYKiAw0PrWIiAIiI41PLSKigMhI41OLiCggBjS9oVq3uopIUVNADGBGY43GpxaRoqaAGIDGpxaRYqeAGEDfra5rdJpJRIqUAmIA0xr0LISIFDcFxABqK0ppqNH41CJSvBQQg5jRqFtdRaR4KSAGofGpRaSYKSAGMaOxmm0dwfjUIiLFRgExCI1PLSLFTAExiOkNGp9aRIqXAmIQU+o1PrWIFC8FxCA0PrWIFDMFxBFofGoRKVYKiCPQ+NQiUqwUEEeg8alFpFgpII5A41OLSLFSQByBxqcWkWKlgDgCjU8tIsVKAXEEGp9aRIqVAiILGp9aRIqRAiILGp9aRIqRAiILGp9aRIqRAiILfZ32aXxqESkmCogsTNezECJShBQQWdD41CJSjCINCDObZ2ZPmtlqM/t4hvWvMrNHzKzbzC5MW9djZkvDn4VR1pkNjU8tIsWmJKoXNrMk8B3g9cAGYLGZLXT3lSmbrQeuAD6S4SX2ufvsqOobqhkN1fxp1ZZclyEiMmqiPII4A1jt7mvd/QCwAJifuoG7r3P3ZUBvhHWMiOkNGp9aRIpLZEcQwGSgPWV+A3DmEPavMLMlQDfwRXf/bfoGZnYVcBVAU1MTbW1th6zv6Og4bNnR2ru1G4Bb7vgrM8YlR+Q1j8ZItikfxK09EL82xa09EL82RdWeKANiuKa6+0YzmwH8xcwed/c1qRu4+/XA9QBz58711tbWQ16gra2N9GVHq3lrB9945G7GTTmR1tOaR+Q1j8ZItikfxK09EL82xa09EL82RdWeKE8xbQRaUuabw2VZcfeN4e+1QBvwspEsbqg0PrWIFJsoA2IxcLyZTTezMuASIKu7kcyszszKw+kG4Bxg5eB7RUvjU4tIsTliQJjZsSlf1q1m9i9mNu5I+7l7N3A1cAewCrjR3VeY2bVmdkH4eqeb2QbgIuB7ZrYi3H0WsMTMHgPuIrgGkdOAAHXaJyLFJZtrELcAc83sOILz/bcCvwDeeKQd3X0RsCht2TUp04sJTj2l73c/8JIsahtV0xtqePCZ7bg7ZpbrckREIpXNKabe8GjgrcC33P2jwKRoy8pPLfWV7D3Qw/Y9B3JdiohI5LIJiC4zuxS4HLgtXFYaXUn5q6WuCoD2HftyXImISPSyCYi/B14OfN7dnzGz6cDPoi0rP7XUhwGhbr9FpAgc8RqEu680s48BU8L5Z4AvRV1YPmquqwSgfYcCQkTiL5u7mN4MLAVuD+dn50PneblQXV7C+OoyHUGISFHI5hTTpwn6VdoJ4O5LgRmRVZTnmuuraN+uaxAiEn9ZXaR29xfTluV953pRaamr1CkmESkK2QTECjN7O5A0s+PN7FvA/RHXlbda6qt4buc+eno916WIiEQqm4B4P3Ay0An8EtgFfDDCmvJaS10VXT3O5l37c12KiEiksrmLaS/wyfCn6LXUh3cybd/L5HGVOa5GRCQ6RwwIM7sLOOx8irufG0lFea7/YbntezlrxvgcVyMiEp1s+mJKHQ60Avg7gkF8itIx4yox09PUIhJ/2Zxiejht0X1m9lBE9eS9spIEk2or2KBnIUQk5rI5xVSfMpsA5gBjI6uoADTXV+lWVxGJvWxOMT1McA3CCE4tPQNcGWVR+a6lror7Vm/LdRkiIpHK5hTT9NEopJC01FeyZfd+Ort7KC9J5rocEZFIDBgQZva3g+3o7r8e+XIKQ0tdFe6wccc+ZjTW5LocEZFIDHYE8eZB1jlQvAFRf3BcCAWEiMTVgAHh7n8/moUUktSH5URE4iqbi9SY2ZsIutuo6Fvm7tdGVVS+axpTQVkyoTuZRCTWshkP4rvAxQR9MhlwETA14rryWiJhTK6rZIO6/RaRGMums76z3f1dwA53/wzB8KMnRFtW/mtWt98iEnPZBETff5P3mtkxQBcwKbqSCkNLfZWuQYhIrGUTELeZ2TjgK8AjwDrgFxHWVBBa6qrYsbeLjs6i7ZZKRGJusOcgFhEEwdfcvQO4xcxuAyoyjDBXdFLvZJo1qTbH1YiIjLzBjiC+B7wJWGtmN5rZWwFXOARSu/0WEYmjAQPC3W9190uBacAtwLuA9Wb2IzN7/SjVl7dSH5YTEYmjI16DcPe97v4rd38rcB4wG7g96sLyXV1VKdVlSR1BiEhsZfMcRJOZvd/M7gN+C9wBnBZ1YfnOzGipr2KDbnUVkZga7CL1e4BLgRMJTjF91N3vH63CCkFznW51FZH4GqyrjZcD/wn82d17R6megtJSX8n9a7bh7phZrssRERlRg3XW9w+jWUghaqmrYu+BHrbvOcD4mvJclyMiMqKyeVBOBqA7mUQkzhQQw6Buv0UkzrLt7jsJNKVu7+7royqqUPQ/LKc7mUQkho4YEGb2fuBTwBag72K1A6dGWFdBqC4vob66jHZ1+y0iMZTNKaYPACe6+8nu/pLwJ6twMLN5Zvakma02s49nWP8qM3vEzLrN7MK0dZeb2dPhz+XZNWf0tdRV6hSTiMRSNgHRDgy5/6XwtNR3gPOBk4BLzeyktM3WA1eQ1jusmdUTHLWcCZwBfMrM6oZaw2horq/SKSYRiaVsrkGsBdrM7PdAZ99Cd//vI+x3BrDa3dcCmNkCYD6wMuU11oXr0p+zeANwp7tvD9ffCcwDfplFvaOqpa6KP67YTE+vk0zoWQgRiY9sAmJ9+FMW/mRrMsHRR58NBEcER7vv5PSNzOwq4CqApqYm2traDlnf0dFx2LKRtu/5Lrp6nN/ecRfjK6O/KWw02jSa4tYeiF+b4tYeiF+bomrPEQMiHGYUM6ty97w6l+Lu1wPXA8ydO9dbW1sPWd/W1kb6spGWeOp5frLyISaf+FLOmjE+0veC0WnTaIpbeyB+bYpbeyB+bYqqPdl01vdyM1sJPBHOv9TM/ieL194ItKTMN4fLsjGcfUdV/8NyulAtIjGTzTmRrxNcE3gBwN0fA16VxX6LgePNbLqZlQGXAAuzrOsO4DwzqwsvTp8XLss7x4yrwExPU4tI/GR10tzd29MW9WSxTzdwNcEX+yrgRndfYWbXmtkFAGZ2upltAC4CvmdmK8J9twOfJQiZxcC1fRes8015SZKJtRVs0BGEiMRMNhep283sbMDNrJTguYhV2by4uy8CFqUtuyZlejHB6aNM+94A3JDN++RaS51udRWR+MnmCOK9wPsI7iLaSDCi3PsirKngNNdX6mlqEYmdbO5i2ga8YxRqKVgtdVVs2b2Rzu4eykuSuS5HRGREZNMX0zczLH4RWOLut458SYWnpb4Kd9i4Yx8zGmtyXY6IyIjI5hRTBcFppafDn1MJrhtcaWZfj6yyAtJSF3b7rTuZRCRGsrlIfSpwjrv3AJjZdcA9wCuAxyOsrWDoWQgRiaNsjiDqgNTzJtVAfRgYnZl3KS5NtRWUJk13MolIrGRzBPFlYKmZtQFG8JDcF8ysGvhThLUVjGTCmDyukg26k0lEYiSbu5h+aGaLCHpnBfiEuz8XTn80ssoKTIu6/RaRmBnwFJOZzQx/nwZMIuhdtR2YGC6TFM11VboGISKxMtgRxIeB9wBfzbDOgXMjqahAtdRXsmNvFx2d3dSUZzXUt4hIXhvwm8zd3xP+fs3olVO4pqTcyTRrUm2OqxERGb7BTjH9W8r0RWnrvhBlUYWopU63uopIvAx2m+slKdP/nrZuXgS1FLT+ZyH0sJyIxMRgAWEDTGeaL3p1VaVUlyV1BCEisTFYQPgA05nmi56Z0VJfxQbd6ioiMTHY7TYvNbNdBEcLleE04XxF5JUVIN3qKiJxMthdTOq3eoha6iu5f8023B0znYUTkcKW1ZCjkp2Wuir2Huhh+54DuS5FRGTYFBAjSHcyiUicKCBGUEt9OC6ErkOISAwoIEZQ/8NyupNJRGJAATGCqstLqK8uo13dfotIDCggRlhLXaWehRCRWFBAjLDmej0LISLxoIAYYS11VWzcuY+eXj1sLiKFTQExwlrqK+nqcTbv2p/rUkREhkUBMcLU7beIxIUCYoS11CsgRCQeFBAj7JhxFZjpaWoRKXwKiBFWXpJkYm0FG3QEISIFTgERgZa6Kj1NLSIFTwERgeb6Sj1NLSIFTwERgZa6Krbs3k9nd0+uSxEROWoKiAi01FfhDht1oVpECpgCIgItdWG33woIESlgCogITGuoBuBff7WUj928jLue3MqB7t4cVyUiMjQDjkktR6+ptoIbrpjLbx99jt8/volfLWlnTEUJr5vVxLxTJvLqExqpKNWQ3yKS3yINCDObB3wDSAI/cPcvpq0vB34KzAFeAC5293VmNg1YBTwZbvqAu783ylpH2rkzmzh3ZhP7u3q4b/U2/rB8M3eu3MJvHt1IVVmS15w4gXmnTOQ1MydQU66cFpH8E9k3k5klge8Arwc2AIvNbKG7r0zZ7Epgh7sfZ2aXAF8CLg7XrXH32VHVN1oqSpO8dlYTr53VRFdPLw+sfYE/LN/MH1ds5vePb6KsJMGbTz2Gz73lFCrLdFQhIvkjyv+6ngGsdve1AGa2AJgPpAbEfODT4fTNwLfNzCKsKadKkwleeXwjrzy+kc/OP4WHn93Bbcue42cPPMvabR388PLTqa8uy3WZIiIAmHs04xaY2YXAPHd/dzj/TuBMd786ZZvl4TYbwvk1wJlADbACeArYBfyHu9+T4T2uAq4CaGpqmrNgwYJD1nd0dFBTUxNB60bWks3dfHdZJw2VxofnVNBYNfC9A4XSpmzFrT0QvzbFrT0QvzYNpz2vec1rHnb3uRlXunskP8CFBNcd+ubfCXw7bZvlQHPK/BqgASgHxofL5gDtQO1g7zdnzhxPd9dddx22LF89uPYFf8mnbvfTP3enr9j44oDbFVKbshG39rjHr01xa497/No0nPYAS3yA79Uob3PdCLSkzDeHyzJuY2YlwFjgBXfvdPcXANz9YYLgOCHCWnPujOn13PxPZ5NMGBd/7/+4f822XJckIkUuyoBYDBxvZtPNrAy4BFiYts1C4PJw+kLgL+7uZtYYXuTGzGYAxwNrI6w1L5zQNIZb/ulsJo6t4IobFnPbsudyXZKIFLHIAsLdu4GrgTsIblm90d1XmNm1ZnZBuNkPgfFmthr4EPDxcPmrgGVmtpTg4vV73X17VLXmk2PGVXLze8/mpS1jef8vH+VH9z2T65JEpEhFegO+uy8CFqUtuyZlej9wUYb9bgFuibK2fDa2qpSfXXkmH1jwKJ/53Uq27OrkY/NOJMY3eIlIHlJXG3mqojTJ/7xjDpedNYXv3r2GD9/4GF096q5DREaPHuHNY8mE8dn5pzCxtoL/+uNTbNtzgLdPiea2ZBGRdAqIPGdmXH3u8TSOKecTv1lO+2Zj7lmdNNSU57o0EYk5nWIqEBefPoXr3zmHjR29XHjd/ax/QUOaiki0FBAF5LWzmvi30yvYua+Lv73ufpZvfDHXJYlIjCkgCsxxdUlufu/LKUsal1z/APev1gN1IhINBUQBOm7CGG7557M5ZlwFl//oIX73mB6oE5GRp4AoUJPGVnLTP57N7JZx/MsCPVAnIiNPAVHA+h6oe/2sJj7zu5V86fYn+jo9FBEZNgVEgasoTXLdZXO49IwpXNe2ho/ctEwP1InIiNBzEDGQTBhfeOspTBhTzjf+/DTb93TynXecRlWZPl4ROXr6BokJM+NfX38CjWPKuebW5cz7+j288vgG5kytY+7UelrqK9WXk4gMiQIiZi47ayrHjKvgR/etY+HS5/j5g+sBaKgpZ+7UOuZMrWPOtDpOPqaW8hKNgS0iA1NAxNC5M5s4d2YTPb3OU1t28/CzO/p/bl+xGYCykgQvbR7LnKn1nH/KRE5tHqsjDBE5hAIixpIJY9akWmZNquWys6YCsHXX/v6wWPLsDn5471q+e/caTmiq4cI5zbzlZZOZMKYix5WLSD5QQBSZCbUVnP+SSZz/kkkA7NrfxW2PbeLmh9v5wqIn+NLtT9J6QiMXzmnmtbOaKCvRjW4ixUoBUeRqK0p5+5lTePuZU1i9tYNbHtnArx/ZwJ+f2EpdVSnzZ0/mwjnNnDJ5bK5LFZFRpoCQfsdNqOFj82bykfNO5J6nn+fmhzfwi4fW8+P71zFrUi1vP6OFt53eoovbIkVCASGHSSaM1hMn0HriBF7c28XCZc9x05J2/t+tK7j+nrV8+PUncsFLjyGR0EVtkTjTCWYZ1NiqUt551lRufd85/PQfzmBMeSkf/NVS3vSte2l7cqu69hCJMQWEZMXMeNUJjdz2/lfwjUtms6ezmyt+tJhLv/8AS9t35ro8EYmAAkKGJJEw5s+ezJ8+9Go+c8HJrN7awVu+cx///POHWft8R67LE5ERpGsQclTKShJcfvY0/m5OMz+4Zy3f/+ta7lixhbfNbeGDrzueplo9SyFS6BQQMiw15SV88HUncNlZU/n2X1bz8wef5TePbuANJ0/k/FMm8eoTGqks011PIoVIASEjoqGmnE9fcDL/cM50rrt7Dbcv38StS5+jsjTJuTMnMO+UiZw7cwLV5forJ1Io9K9VRtSU8VX859++hM/OP5kHn9nOosc3cceKLfz+8U2UlyR41QmNnH/KRF47q4mxlaW5LldEBqGAkEiUJBOcc1wD5xzXwLXzT2HJuu38Yflmbl++mTtXbqE0aZxzXAPHWBf+xFZa6qtorqukolSno0TyhQJCIpdMGGfOGM+ZM8Zzzd+cxNINO/nD45v4w/LNtO04wC+eWNy/7cTaClrqK2mpr2JK2k/jmHL1OCsyihQQMqoSCeO0KXWcNqWOT7xxFrfecRcts2azfvte1r+wj/Xb99K+fS//t+YFfv3IxkP2HVdVysyJY4IeaicGvdQe31Sjow6RiCggJGfMjHEVCeZMrWfO1PrD1u/v6mHjziA01m3bw1NbdrNq024WPNTOvq4eABIGMxprDgbHpDFMqa+mqixJVVmSyrIkZcmEjjxEjoICQvJWRWmSYxtrOLaxBk48uLyn11m/fS9PbNrFqk27WLV5N0vbd3Lbsk0ZXyeZMKpKg7AIQqOkP0CCn5JgXemh6yvLklSH07WVJTTWVDChtlxHLFI0FBBScJIJY3pDNdMbqvvHtYBgbIsnN+/muZ372Hegh70HetjX1cPeA93BdLgsWN5NR2c3z+/uDJd1929/pO6lxlSUMGFMORPGVNA4pjyYrj04//SOHsau30FpMkEyYZQkjGTCDpkvSSYoL0lQVZbU0Y3kLQWExEZtRSmnTzv8VNVQuDv7u3r7A6MvPF7c18XW3Z08v7uTrbv283xHJ1t3dbK0fSdbd+9nf1fvoS/04P1ZvV8yYdSUlzCmooQxFaWMqSihNmW6b3l5SYKykgSlYbCUJYPpsrTlJUkjYUbCglN4fdMJMyz8nTAjkYCyZILykiRlJUFwiaRTQIikMDMqw9NL47Pcx93Z3dnN1l2dbOvoZMkjj3LyKafS3et09/TS3ev09Pph8/u6eti9v4vd+7vDny527e9m48797N6/u39Z7yh0mFuSMMpKEv1B1Bcc5SUJDuzdx4+feehgaIWBFgRbsKymooQx5aX9AZVMHAyjZMJIJIKgSlowXZpIUF4avL6OoPKXAkJkmMyM2opSaitKOW5CDfvXl9A6c8KIvLZ7ECSdXb0c6OnlQPfB31198ynLunsdd+h1p9dTpwnng+nuXqcr3C947YPv0dnVS2d3T//0c/s7eKHjAOu27aGjs5td+7s50N175OKz1BdEFaVJyg+bTlKSPPz0XEn/tIXLE/2n8hIJ6w+i/mVhaCUTCZIGa9d3sfmh9cG+yUP3P3w+0f9eJYlE+DutjkQCDMzAOHjEZgS/gf751CO6fA9HBYRIHjMzqspKqCrLXQ1tbW20tr7ikGWd3T107A+u4/QdAXV0dtPT20tPL/SEYdQTHi25B8uCaedAj9PZHYTS/vB3Z3cvnV09dHb3sj/83dndw76uYL+unt7+1+vq7aWnx+nqOzpLOTLr9b7fR2jYysej+0PL0sHTfn2nBMPwIFzed+SVMMyC4Aum6Q++hEFDcj+trSNfX6QBYWbzgG8ASeAH7v7FtPXlwE+BOcALwMXuvi5c9+/AlUAP8C/ufkeUtYpI9spLkpTXJBlfU57rUgbUH1Du9Iah1dMTzP/13ns586yX093jYbD0hqcAU+bD6dRTg8GyYF1/UPU6XT3eP3iWOzhBQPVN99340HcE52lHdL39yz3tiI/+0EsNvt6+IAynbc+BSP4MIwsIM0sC3wFeD2wAFpvZQndfmbLZlcAOdz/OzC4BvgRcbGYnAZcAJwPHAH8ysxPcvSeqekUkXszC00IZ1o0rTzBpbOWo1xSVtra2SF43ygGDzgBWu/tadz8ALADmp20zH/hJOH0z8FoLTsrNBxa4e6e7PwOsDl9PRERGSZSnmCYD7SnzG4AzB9rG3bvN7EVgfLj8gbR9J6e/gZldBVwF0NTUdFiKdnR0RJasuRK3NsWtPRC/NsWtPRC/NkXVnoK+SO3u1wPXA8ydO9db067SBBfXWg/fsYDFrU1xaw/Er01xaw/Er01RtSfKU0wbgZaU+eZwWcZtzKwEGEtwsTqbfUVEJEJRBsRi4Hgzm25mZQQXnRembbMQuDycvhD4iwe3AiwELjGzcjObDhwPPBRhrSIikiayU0zhNYWrgTsIbnO9wd1XmNm1wBJ3Xwj8EPiZma0GthOECOF2NwIrgW7gfbqDSURkdEV6DcLdFwGL0pZdkzK9H7hogH0/D3w+yvpERGRgUZ5iEhGRAmZ9T/8VOjN7Hng2bXEDsC0H5UQpbm2KW3sgfm2KW3sgfm0aTnumuntjphWxCYhMzGyJu8/NdR0jKW5tilt7IH5tilt7IH5tiqo9OsUkIiIZKSBERCSjuAfE9bkuIAJxa1Pc2gPxa1Pc2gPxa1Mk7Yn1NQgRETl6cT+CEBGRo6SAEBGRjGIbEGY2z8yeNLPVZvbxXNczXGa2zsweN7OlZrYk1/UcDTO7wcy2mtnylGX1ZnanmT0d/q7LZY1DMUB7Pm1mG8PPaamZvTGXNQ6VmbWY2V1mttLMVpjZB8LlBfk5DdKegv2czKzCzB4ys8fCNn0mXD7dzB4Mv/N+FfaBN7z3iuM1iHA0u6dIGc0OuDRtNLuCYmbrgLnuXrAP95jZq4AO4Kfufkq47MvAdnf/Yhjkde7+sVzWma0B2vNpoMPd/yuXtR0tM5sETHL3R8xsDPAw8BbgCgrwcxqkPW+jQD+ncFC1anfvMLNS4F7gA8CHgF+7+wIz+y7wmLtfN5z3iusRRDaj2ckoc/e/EnTKmCp1VMGfEPzjLQgDtKegufsmd38knN4NrCIYrKsgP6dB2lOwPNARzpaGPw6cSzAyJ4zQZxTXgMg0ml1B/6Ug+AvwRzN7OBxJLy6a3H1TOL0ZaMplMSPkajNbFp6CKohTMZmY2TTgZcCDxOBzSmsPFPDnZGZJM1sKbAXuBNYAO929O9xkRL7z4hoQcfQKdz8NOB94X3h6I1bCsUAK/ZzndcCxwGxgE/DVnFZzlMysBrgF+KC770pdV4ifU4b2FPTn5O497j6bYDC1M4CZUbxPXAMidiPSufvG8PdW4DcEfyniYEt4nrjvfPHWHNczLO6+JfzH2wt8nwL8nMLz2rcAP3f3X4eLC/ZzytSeOHxOAO6+E7gLeDkwLhyZE0boOy+uAZHNaHYFw8yqwwtsmFk1cB6wfPC9CkbqqIKXA7fmsJZh6/sSDb2VAvucwgugPwRWuft/p6wqyM9poPYU8udkZo1mNi6criS4GWcVQVBcGG42Ip9RLO9iAghvW/s6B0ezK9jBh8xsBsFRAwSDPP2iENtjZr8EWgm6Jt4CfAr4LXAjMIWgu/a3uXtBXPgdoD2tBKctHFgH/GPKufu8Z2avAO4BHgd6w8WfIDhvX3Cf0yDtuZQC/ZzM7FSCi9BJgv/k3+ju14bfEwuAeuBR4DJ37xzWe8U1IEREZHjieopJRESGSQEhIiIZKSBERCQjBYSIiGSkgBARkYwUEBJLZtaT0lPn0rCbhaHs/xYzO2mYNVxhZr3hbYl9y5YPtZZBXr/jyFuJHD0FhMTVPnefnfKzboj7vwUYUkCkPMWaagPwySG+d+QGqFXkEAoIKQpmVmNmfzazRywYV2N+yrp3hZ22PWZmPzOzs4ELgK+ERx/HmtlsM3sg3O43fZ27mVmbmX3dgjE6PpDhrW8DTjazEzPU1JEyfaGZ/Tic/rGZXRe+31ozaw07lFvVt03Kfl8LxwT4s5k1hsuONbPbw44d7zGzmSmv+10zexD48vD+RKUY6H8REleVYW+XAM8AFwFvdfddZtYAPGBmCwmOEv4DONvdt5lZvbtvD9fd5u43A5jZMuD97n63mV1L8NT0B8PXL3P3uQPU0UvwZfwJDnZVkY06gv51LiDo5uIc4N3AYjOb7e5LgWpgibv/q5ldE9Z0NcEA9u9196fN7Ezgfwi6goagj56z3b1nCLVIkVJASFztC3u7BPo7bPtC2AtuL0FXyE0EX5w39Q3ElKn7CDMbC4xz97vDRT8BbkrZ5FdHqOUXwCfNbPoQ6v+du7uZPQ5scffHw1pWANOApWE7+t77f4Ffh72Wng3cFHRDBEB5yuvepHCQbCkgpFi8A2gE5rh7lwUj9FWM0GvvGWylu3eb2VeB9BHYUvu5Sa+lrw+d3pTpvvmB/t06wWnjnanhOJRaRVLpGoQUi7HA1jAcXgNMDZf/BbjIzMZDMPZyuHw3MAbA3V8EdpjZK8N17wTuZmh+DLyOIKT6bDGzWWaWIOhRdKgSHOy98+3AveFYB8+Y2UUQ9GZqZi89itcWUUBI0fg5MDc8ZfMu4AkAd18BfB6428weA/q6hF4AfNTMHjWzYwmuH3wlvBYxG7h2KG8eDn37TWBCyuKPE1zEvp9g0Jqh2gOcYWbLCU6V9dX0DuDKsD0r0HC7cpTUm6uIiGSkIwgREclIASEiIhkpIEREJCMFhIiIZKSAEBGRjBQQIiKSkQJCREQy+v9gm1jmvNiQBAAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "display_scree_plot(30, pca.explained_variance_ratio_[:30])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous ne sommes pas sûrs si le « décrochement » se fait plutôt pour la valeur 3 ou 6. Observons alors la variance totale pour ces valeurs. On va utiliser une fonction pour simplifier ce processus."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "def display_total_variance(p):\n",
    "    pca = PCA(n_components=p)\n",
    "    pca.fit_transform(z)\n",
    "    print('Total variance for p = %d: %.2f%%' % (p, pca.explained_variance_ratio_.sum() * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut alors comparer nos deux valeurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Total variance for p = 3: 41.96%\n",
      "Total variance for p = 6: 54.12%\n"
     ]
    }
   ],
   "source": [
    "display_total_variance(3)\n",
    "display_total_variance(6)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Choisissons 6 car elle permet de garder au moins 50% de la variance.\n",
    "\n",
    "Notons tout de même que dans les 2 cas, le pourcentage de variance conservée est plutôt bas. Également, 6 est un nombre d'axes plutôt élevé. On peut en déduire que l'ACP n'est sûrement pas la meilleure réduction de dimension."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "ACP_N_COMPONENTS = 6"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Et voilà, notre ACP finale peut alors voir le jour."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Xp_train.shape = (60000, 6)\n",
      "Xp_test.shape = (10000, 6)\n"
     ]
    }
   ],
   "source": [
    "pca = PCA(n_components=ACP_N_COMPONENTS)\n",
    "\n",
    "X = np.concatenate((X_train, X_test))\n",
    "z = scaler.fit_transform(X)\n",
    "Xp = pca.fit_transform(z)\n",
    "Xp_train = Xp[:X_train.shape[0]]\n",
    "Xp_test = Xp[X_train.shape[0]:]\n",
    "\n",
    "if VERBOSE:\n",
    "    print('Xp_train.shape =', Xp_train.shape)\n",
    "    print('Xp_test.shape =', Xp_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Analyse linéaire discriminante"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut maintenant mettre en place l'ALD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Xl_train.shape = (60000, 9)\n",
      "Xl_test.shape = (10000, 9)\n"
     ]
    }
   ],
   "source": [
    "lda = LDA()\n",
    "\n",
    "X = np.concatenate((X_train, X_test))\n",
    "y = np.concatenate((y_train, y_test))\n",
    "\n",
    "X_lda = lda.fit_transform(X, y)\n",
    "Xl_train = X_lda[:X_train.shape[0]]\n",
    "Xl_test = X_lda[X_train.shape[0]:]\n",
    "\n",
    "if VERBOSE:\n",
    "    print('Xl_train.shape =', Xl_train.shape)\n",
    "    print('Xl_test.shape =', Xl_test.shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ratio de la variance expliquée par les axes obtenus :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0.44775652, 0.21875177, 0.09237004, 0.07396008, 0.06062643,\n",
       "       0.04305586, 0.03801965, 0.01584194, 0.00961772])"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "lda.explained_variance_ratio_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut en déduire qu'on garde 9 axes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Classifications des données\n",
    "\n",
    "Cette partie est dédiée concrètement au _Machine Learning_. Pour classifier les données on a plusieurs méthodes que l'on peut appliquer.\n",
    "\n",
    "- Méthode des _k_ plus proches voisins\n",
    "- Classification naïve bayésienne\n",
    "- Abre de décision\n",
    "- Forêt aléatoire"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Méthode des _k_ plus proches voisins\n",
    "\n",
    "Grâce à cette méthode, on regarde les classes d'un nombre donné de voisins les plus proches afin d'en déduire la classe de celui-ci. On va directement rechercher la meilleure valeur de _k_ possible parmi un ensemble déterminé en amont."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mise en place"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On commence par définir des valeurs possibles pour _k_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "K_VALS = [1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 21, 31, 41, 51, 61, 71, 81, 91]\n"
     ]
    }
   ],
   "source": [
    "K_VALS = list(range(1, 11)) + list(range(11, 101, 10))\n",
    "\n",
    "if VERBOSE:\n",
    "    print('K_VALS =', K_VALS)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut récupérer le temps d'entraînement et l'accuracy pour le meilleur _k_ grâce à une fonction. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_best_knn(X_train, y_train, X_test, y_test, k_vals, patience=3):\n",
    "    best_k, best_k_training_time, best_k_accuracy = 0, 0, 0\n",
    "    p = patience\n",
    "\n",
    "    for k in k_vals:\n",
    "        if patience <= 0:\n",
    "            break\n",
    "\n",
    "        if VERBOSE:\n",
    "            print('Computing %d-Nearest Neighbors classification…' % k)\n",
    "\n",
    "        knn = KNeighborsClassifier(n_neighbors=k)\n",
    "        start_time = time.time()\n",
    "        knn.fit(X_train, y_train)\n",
    "        training_time = time.time() - start_time\n",
    "        accuracy = knn.score(X_test, y_test)\n",
    "\n",
    "        if VERBOSE:\n",
    "            print('training_time =', training_time)\n",
    "            print('accuracy =', accuracy)\n",
    "        \n",
    "        if accuracy > best_k_accuracy:\n",
    "            best_k = k\n",
    "            best_k_training_time = training_time\n",
    "            best_k_accuracy = accuracy\n",
    "            p = patience + 1\n",
    "        \n",
    "        p -= 1\n",
    "    \n",
    "    return best_k, best_k_training_time, best_k_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification des données brutes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle _k_-NN pour le meilleur _k_ sur les données brutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing 1-Nearest Neighbors classification…\n",
      "training_time = 0.07255005836486816\n",
      "accuracy = 0.8497\n",
      "Computing 2-Nearest Neighbors classification…\n",
      "training_time = 0.012056827545166016\n",
      "accuracy = 0.846\n",
      "Computing 3-Nearest Neighbors classification…\n",
      "training_time = 0.012485027313232422\n",
      "accuracy = 0.8541\n",
      "Computing 4-Nearest Neighbors classification…\n",
      "training_time = 0.011662006378173828\n",
      "accuracy = 0.8577\n",
      "Computing 5-Nearest Neighbors classification…\n",
      "training_time = 0.011801004409790039\n",
      "accuracy = 0.8554\n",
      "Computing 6-Nearest Neighbors classification…\n",
      "training_time = 0.011735677719116211\n",
      "accuracy = 0.8544\n",
      "Computing 7-Nearest Neighbors classification…\n",
      "training_time = 0.011578083038330078\n",
      "accuracy = 0.854\n",
      "Computing 8-Nearest Neighbors classification…\n",
      "training_time = 0.01273488998413086\n",
      "accuracy = 0.8534\n",
      "Computing 9-Nearest Neighbors classification…\n",
      "training_time = 0.011889219284057617\n",
      "accuracy = 0.8519\n",
      "Computing 10-Nearest Neighbors classification…\n",
      "training_time = 0.01324319839477539\n",
      "accuracy = 0.8515\n",
      "Computing 11-Nearest Neighbors classification…\n",
      "training_time = 0.011809110641479492\n",
      "accuracy = 0.8495\n",
      "Computing 21-Nearest Neighbors classification…\n",
      "training_time = 0.013054847717285156\n",
      "accuracy = 0.8407\n",
      "Computing 31-Nearest Neighbors classification…\n",
      "training_time = 0.012391328811645508\n",
      "accuracy = 0.8348\n",
      "Computing 41-Nearest Neighbors classification…\n",
      "training_time = 0.01216268539428711\n",
      "accuracy = 0.8287\n",
      "Computing 51-Nearest Neighbors classification…\n",
      "training_time = 0.012603044509887695\n",
      "accuracy = 0.8258\n",
      "Computing 61-Nearest Neighbors classification…\n",
      "training_time = 0.01318812370300293\n",
      "accuracy = 0.8228\n",
      "Computing 71-Nearest Neighbors classification…\n",
      "training_time = 0.013070821762084961\n",
      "accuracy = 0.8213\n",
      "Computing 81-Nearest Neighbors classification…\n",
      "training_time = 0.015358924865722656\n",
      "accuracy = 0.8187\n",
      "Computing 91-Nearest Neighbors classification…\n",
      "training_time = 0.012609004974365234\n",
      "accuracy = 0.818\n",
      "4-Nearest Neighbors:\n",
      "  - Training Time: 0.01s.\n",
      "  - Accuracy: 85.77%.\n"
     ]
    }
   ],
   "source": [
    "knn_k, knn_training_time, knn_accuracy = find_best_knn(X_train, y_train, X_test, y_test, K_VALS)\n",
    "\n",
    "print('%d-Nearest Neighbors:' % knn_k)\n",
    "print('  - Training Time: %.2fs.' % knn_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (knn_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le meilleur modèle _k_-NN sur les données brutes est celui dont le _k_ vaut 4 et son accuracy est d'environ 86%. Quant aux autres valeurs de _k_, on constate malgré tout qu'on arrive généralement au delà des 80% d'accuracy."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification après une ACP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle _k_-NN pour le meilleur _k_ après réduction de dimension grâce à l'ACP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing 1-Nearest Neighbors classification…\n",
      "training_time = 0.024950027465820312\n",
      "accuracy = 0.7205\n",
      "Computing 2-Nearest Neighbors classification…\n",
      "training_time = 0.01714801788330078\n",
      "accuracy = 0.7169\n",
      "Computing 3-Nearest Neighbors classification…\n",
      "training_time = 0.017603158950805664\n",
      "accuracy = 0.7427\n",
      "Computing 4-Nearest Neighbors classification…\n",
      "training_time = 0.01724696159362793\n",
      "accuracy = 0.7507\n",
      "Computing 5-Nearest Neighbors classification…\n",
      "training_time = 0.017938852310180664\n",
      "accuracy = 0.7558\n",
      "Computing 6-Nearest Neighbors classification…\n",
      "training_time = 0.0174558162689209\n",
      "accuracy = 0.7591\n",
      "Computing 7-Nearest Neighbors classification…\n",
      "training_time = 0.01792597770690918\n",
      "accuracy = 0.7642\n",
      "Computing 8-Nearest Neighbors classification…\n",
      "training_time = 0.01722121238708496\n",
      "accuracy = 0.767\n",
      "Computing 9-Nearest Neighbors classification…\n",
      "training_time = 0.021203041076660156\n",
      "accuracy = 0.767\n",
      "Computing 10-Nearest Neighbors classification…\n",
      "training_time = 0.01790475845336914\n",
      "accuracy = 0.7691\n",
      "Computing 11-Nearest Neighbors classification…\n",
      "training_time = 0.01756119728088379\n",
      "accuracy = 0.7674\n",
      "Computing 21-Nearest Neighbors classification…\n",
      "training_time = 0.016989946365356445\n",
      "accuracy = 0.7673\n",
      "Computing 31-Nearest Neighbors classification…\n",
      "training_time = 0.02081775665283203\n",
      "accuracy = 0.7648\n",
      "Computing 41-Nearest Neighbors classification…\n",
      "training_time = 0.016943931579589844\n",
      "accuracy = 0.7648\n",
      "Computing 51-Nearest Neighbors classification…\n",
      "training_time = 0.0190122127532959\n",
      "accuracy = 0.7659\n",
      "Computing 61-Nearest Neighbors classification…\n",
      "training_time = 0.017514944076538086\n",
      "accuracy = 0.764\n",
      "Computing 71-Nearest Neighbors classification…\n",
      "training_time = 0.018743038177490234\n",
      "accuracy = 0.7641\n",
      "Computing 81-Nearest Neighbors classification…\n",
      "training_time = 0.017647981643676758\n",
      "accuracy = 0.7613\n",
      "Computing 91-Nearest Neighbors classification…\n",
      "training_time = 0.016820907592773438\n",
      "accuracy = 0.7596\n",
      "10-Nearest Neighbors after PCA:\n",
      "  - Training Time: 0.02s.\n",
      "  - Accuracy: 76.91%.\n"
     ]
    }
   ],
   "source": [
    "pca_knn_k, pca_knn_training_time, pca_knn_accuracy = find_best_knn(Xp_train, y_train, Xp_test, y_test, K_VALS)\n",
    "\n",
    "print('%d-Nearest Neighbors after PCA:' % pca_knn_k)\n",
    "print('  - Training Time: %.2fs.' % pca_knn_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (pca_knn_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le meilleur modèle _k_-NN après réduction de dimension grâce à l'ACP est celui dont le _k_ vaut 10 et son accuracy est d'environ 77%. Quant aux autres valeurs de _k_, on constate malgré tout qu'on arrive généralement au delà des 70% d'accuracy. \n",
    "\n",
    "On peut déduire qu'une classification _k_-nn après ACP n'est pas pertinente, puisque les temps d'entraînement sont similaires et on observe une différence d'accuracy de 10%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification après une ALD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle _k_-NN pour le meilleur _k_ après réduction de dimension grâce à l'ALD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing 1-Nearest Neighbors classification…\n",
      "training_time = 0.02554917335510254\n",
      "accuracy = 0.7935\n",
      "Computing 2-Nearest Neighbors classification…\n",
      "training_time = 0.01961231231689453\n",
      "accuracy = 0.8025\n",
      "Computing 3-Nearest Neighbors classification…\n",
      "training_time = 0.02342510223388672\n",
      "accuracy = 0.8177\n",
      "Computing 4-Nearest Neighbors classification…\n",
      "training_time = 0.018456697463989258\n",
      "accuracy = 0.8259\n",
      "Computing 5-Nearest Neighbors classification…\n",
      "training_time = 0.02400827407836914\n",
      "accuracy = 0.8293\n",
      "Computing 6-Nearest Neighbors classification…\n",
      "training_time = 0.018183231353759766\n",
      "accuracy = 0.8328\n",
      "Computing 7-Nearest Neighbors classification…\n",
      "training_time = 0.01784205436706543\n",
      "accuracy = 0.8363\n",
      "Computing 8-Nearest Neighbors classification…\n",
      "training_time = 0.018096208572387695\n",
      "accuracy = 0.8363\n",
      "Computing 9-Nearest Neighbors classification…\n",
      "training_time = 0.01875901222229004\n",
      "accuracy = 0.8396\n",
      "Computing 10-Nearest Neighbors classification…\n",
      "training_time = 0.01886892318725586\n",
      "accuracy = 0.8393\n",
      "Computing 11-Nearest Neighbors classification…\n",
      "training_time = 0.0181272029876709\n",
      "accuracy = 0.8386\n",
      "Computing 21-Nearest Neighbors classification…\n",
      "training_time = 0.018520832061767578\n",
      "accuracy = 0.8411\n",
      "Computing 31-Nearest Neighbors classification…\n",
      "training_time = 0.017617225646972656\n",
      "accuracy = 0.8413\n",
      "Computing 41-Nearest Neighbors classification…\n",
      "training_time = 0.018056154251098633\n",
      "accuracy = 0.8415\n",
      "Computing 51-Nearest Neighbors classification…\n",
      "training_time = 0.018262863159179688\n",
      "accuracy = 0.8392\n",
      "Computing 61-Nearest Neighbors classification…\n",
      "training_time = 0.018296003341674805\n",
      "accuracy = 0.8388\n",
      "Computing 71-Nearest Neighbors classification…\n",
      "training_time = 0.017726898193359375\n",
      "accuracy = 0.8388\n",
      "Computing 81-Nearest Neighbors classification…\n",
      "training_time = 0.018044233322143555\n",
      "accuracy = 0.839\n",
      "Computing 91-Nearest Neighbors classification…\n",
      "training_time = 0.017910003662109375\n",
      "accuracy = 0.8379\n",
      "41-Nearest Neighbors after LDA:\n",
      "  - Training Time: 0.02s.\n",
      "  - Accuracy: 84.15%.\n"
     ]
    }
   ],
   "source": [
    "lda_knn_k, lda_knn_training_time, lda_knn_accuracy = find_best_knn(Xl_train, y_train, Xl_test, y_test, K_VALS)\n",
    "\n",
    "print('%d-Nearest Neighbors after LDA:' % lda_knn_k)\n",
    "print('  - Training Time: %.2fs.' % lda_knn_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (lda_knn_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque alors que le meilleur modèle _k_-NN après réduction de dimension grâce à l'ALD est celui dont le _k_ vaut 41 et son accuracy est d'environ 84%. Quant aux autres valeurs de _k_, on constate malgré tout qu'on arrive généralement au delà des 80% d'accuracy.\n",
    "\n",
    " Malgré un _k_ élevé, le temps d'entraînement est très court. Cependant, on trouve une accuracy similaire au modèle sans réduction, qui est même légèrement inférieure. Donc ce modèle n'est donc pas forcément pertinent pour ce jeu de données. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Classification naïve bayésienne\n",
    "\n",
    "La classification naïve bayésienne se base sur le Théorème de Bayes pour classifier les données, en considérant chaque caractéristique (ici pixel) indépendamment des autres."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mise en place"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut récupérer le temps d'entraînement et l'accuracy grâce à une fonction."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_nb(X_train, y_train, X_test, y_test):\n",
    "    if VERBOSE:\n",
    "        print('Computing naive Bayes classification…')\n",
    "\n",
    "    nb = GaussianNB()\n",
    "    start_time = time.time()\n",
    "    nb.fit(X_train, y_train)\n",
    "    nb_training_time = time.time() - start_time\n",
    "    nb_accuracy = nb.score(X_test, y_test)\n",
    "\n",
    "    if VERBOSE:\n",
    "        print('training_time =', nb_training_time)\n",
    "        print('accuracy =', nb_accuracy)\n",
    "    \n",
    "    return nb_training_time, nb_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification des données brutes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle NB sur les données brutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing naive Bayes classification…\n",
      "training_time = 0.2613201141357422\n",
      "accuracy = 0.5856\n",
      "Naive Bayes:\n",
      "  - Training Time: 0.26s.\n",
      "  - Accuracy: 58.56%.\n"
     ]
    }
   ],
   "source": [
    "nb_training_time, nb_accuracy = find_nb(X_train, y_train, X_test, y_test)\n",
    "\n",
    "print('Naive Bayes:')\n",
    "print('  - Training Time: %.2fs.' % nb_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (nb_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle NB sur les données brutes a une accuracy d'environ 59%. C'est une accuracy plutôt basse par rapport aux résultats qu'on a obtenus jusqu'ici. Ce modèle ne semble pas pertinent."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification après une ACP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle NB après réduction de dimension grâce à l'ACP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing naive Bayes classification…\n",
      "training_time = 0.016337156295776367\n",
      "accuracy = 0.6772\n",
      "Naive Bayes after PCA:\n",
      "  - Training Time: 0.02s.\n",
      "  - Accuracy: 67.72%.\n"
     ]
    }
   ],
   "source": [
    "pca_nb_training_time, pca_nb_accuracy = find_nb(Xp_train, y_train, Xp_test, y_test)\n",
    "\n",
    "print('Naive Bayes after PCA:')\n",
    "print('  - Training Time: %.2fs.' % pca_nb_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (pca_nb_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle NB après réduction de dimension grâce à l'ACP a une accuracy d'environ 68%. On a gagné presque 10%, par rapport au modèle sans réduction, c'est donc un meilleur modèle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification après une ALD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle NB après réduction de dimension grâce à l'ALD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing naive Bayes classification…\n",
      "training_time = 0.012111186981201172\n",
      "accuracy = 0.8172\n",
      "Naive Bayes after LDA:\n",
      "  - Training Time: 0.01s.\n",
      "  - Accuracy: 81.72%.\n"
     ]
    }
   ],
   "source": [
    "lda_nb_training_time, lda_nb_accuracy = find_nb(Xl_train, y_train, Xl_test, y_test)\n",
    "\n",
    "print('Naive Bayes after LDA:')\n",
    "print('  - Training Time: %.2fs.' % lda_nb_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (lda_nb_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle NB après réduction de dimension grâce à l'ALD a une accuracy d'environ 82%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Une nette amélioration se démarque au niveau de l'accuracy. En effet, on gagne : \n",
    "- plus de 20% par rapport à une classification bayésienne sans réduction de dimension ;\n",
    "- plus de 10% par rapport à une classification bayésienne avec ACP.\n",
    "\n",
    "On peut en conclure que c'est un bon modèle."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Arbre de décision\n",
    "\n",
    "Comme on travaille sur des images et que chaque pixel fait office de descripteur, on a de très grandes chances de se retrouver avec un arbre très touffu et donc assez long à construire. Ce n'est donc sûrement pas la meilleure méthode à utiliser, du moins peut-être pas avant réduction de dimension. Testons cela."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mise en place"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut récupérer le temps d'entraînement et l'accuracy grâce à une fonction, qui utilise dans un premier temps les hyper paramètres par défaut d'un classifieur arbre de décision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_dt(X_train, y_train, X_test, y_test, params: dict = None):\n",
    "    if VERBOSE:\n",
    "        if params == None:\n",
    "            print('Computing Decision Tree classification…')\n",
    "        else:\n",
    "            print('Computing %s Decision Tree classification…' % params)\n",
    "\n",
    "    if params == None:\n",
    "        dt = DecisionTreeClassifier()\n",
    "    else:\n",
    "        dt = DecisionTreeClassifier(\n",
    "                        criterion=params.get('criterion'),\n",
    "                        max_depth=params.get('max_depth'),\n",
    "                        min_samples_leaf=params.get('min_samples_leaf'),\n",
    "                        min_samples_split=params.get('min_samples_split')\n",
    "                    )\n",
    "\n",
    "    start_time = time.time()\n",
    "    dt.fit(X_train, y_train)\n",
    "    dt_training_time = time.time() - start_time\n",
    "    dt_accuracy = dt.score(X_test, y_test)\n",
    "\n",
    "    if VERBOSE:\n",
    "        print('training_time =', dt_training_time)\n",
    "        print('accuracy =', dt_accuracy)\n",
    "    \n",
    "    return dt_training_time, dt_accuracy, dt.tree_.max_depth"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification (initiale) des données brutes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle DT initial sur les données brutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing Decision Tree classification…\n",
      "training_time = 27.455711126327515\n",
      "accuracy = 0.7926\n",
      "Decision Tree:\n",
      "  - Training Time: 27.46s.\n",
      "  - Accuracy: 79.26%.\n"
     ]
    }
   ],
   "source": [
    "default_dt_training_time, default_dt_accuracy, dt_max_depth = find_dt(X_train, y_train, X_test, y_test)\n",
    "\n",
    "print('Decision Tree:')\n",
    "print('  - Training Time: %.2fs.' % default_dt_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (default_dt_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle DT initial sur les données brutes a une accuracy d'environ 79%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comme on l'avait prédit, le temps d'entraînement est considérablement important comparé aux 2 méthodes testées précédemment (entre 30 secondes et 1 minute). Cependant on obtient une accuracy :\n",
    "- bien meilleure que celle d'une classification bayésienne ;\n",
    "- très légèrement inférieure à celle du meilleur modèle de k-NN, mais qui reste similaire. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Voyons si on peut améliorer ce modèle en changeant les hyper paramètres, sachant que par défaut ils sont définis comme ci-dessous :\n",
    "- `criterion='gini'`\n",
    "- `max_depth=None`\n",
    "- `min_samples_leaf=1`\n",
    "- `min_samples_split=2`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous avons donc besoin de calculer la profondeur par défaut."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "dt_max_depth = 49\n"
     ]
    }
   ],
   "source": [
    "if VERBOSE:\n",
    "    print('dt_max_depth =', dt_max_depth)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On trouve donc une profondeur de 49 par défaut."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Élaborons une nouvelle fonction qui permet de trouver le meilleur modèle d'arbre de décision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "criterion = ['gini', 'entropy']\n",
    "max_depth = [3, 5, 10]\n",
    "min_samples_leaf = [1, 2, 5]\n",
    "min_samples_split = [2, 3, 4]\n",
    "\n",
    "def find_best_dt(X_train, y_train, X_test, y_test):\n",
    "    best_params = None\n",
    "    best_dt_training_time, best_dt_accuracy = 0, 0\n",
    "\n",
    "    for c in criterion:\n",
    "        for depth in max_depth:\n",
    "            for l in min_samples_leaf:\n",
    "                for s in min_samples_split:\n",
    "\n",
    "                    params = {\n",
    "                        'criterion': c,\n",
    "                        'max_depth': depth,\n",
    "                        'min_samples_leaf': l,\n",
    "                        'min_samples_split': s\n",
    "                    }\n",
    "                    dt_training_time, dt_accuracy, _ = find_dt(X_train, y_train, X_test, y_test, params)\n",
    "\n",
    "                    if best_dt_accuracy == 0 or dt_accuracy > best_dt_accuracy:\n",
    "                        best_params = params\n",
    "                        best_dt_training_time = dt_training_time\n",
    "                        best_dt_accuracy = dt_accuracy\n",
    "                        \n",
    "\n",
    "    if VERBOSE:\n",
    "        print('best_params =', best_params)\n",
    "        print('training_time =', best_dt_training_time)\n",
    "        print('accuracy =', best_dt_accuracy)\n",
    "    \n",
    "    return best_params, best_dt_training_time, best_dt_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous avons élaboré cette fonction dans un premier temps, avant de trouver une méthode plus propre à l'aide de `RandomizedSearchCV`. On va également utiliser cette méthode pour vérifier et/ou comparer les résultats."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On commence par créer la _random grid_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'criterion': ['gini', 'entropy'], 'max_depth': [3, 5, 10], 'min_samples_leaf': [1, 2, 5], 'min_samples_split': [2, 3, 4]}\n"
     ]
    }
   ],
   "source": [
    "random_grid = {\n",
    "    'criterion': criterion,\n",
    "    'max_depth': max_depth,\n",
    "    'min_samples_leaf': min_samples_leaf,\n",
    "    'min_samples_split': min_samples_split\n",
    "}\n",
    "\n",
    "if VERBOSE:\n",
    "    print(random_grid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Une fois créée on peut procéder à la recherche des meilleurs hyper paramètres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n",
      "[CV] END criterion=gini, max_depth=10, min_samples_leaf=1, min_samples_split=3; total time=  19.2s\n",
      "[CV] END criterion=gini, max_depth=10, min_samples_leaf=1, min_samples_split=3; total time=  19.7s\n",
      "[CV] END criterion=gini, max_depth=10, min_samples_leaf=1, min_samples_split=3; total time=  20.0s\n",
      "[CV] END criterion=entropy, max_depth=10, min_samples_leaf=2, min_samples_split=3; total time=  21.8s\n",
      "[CV] END criterion=entropy, max_depth=10, min_samples_leaf=2, min_samples_split=2; total time=  22.0s\n",
      "[CV] END criterion=entropy, max_depth=10, min_samples_leaf=2, min_samples_split=3; total time=  22.0s\n",
      "[CV] END criterion=entropy, max_depth=10, min_samples_leaf=2, min_samples_split=2; total time=  22.3s\n",
      "[CV] END criterion=entropy, max_depth=10, min_samples_leaf=2, min_samples_split=3; total time=  22.8s\n",
      "[CV] END criterion=gini, max_depth=3, min_samples_leaf=2, min_samples_split=4; total time=   6.2s\n",
      "[CV] END criterion=gini, max_depth=5, min_samples_leaf=2, min_samples_split=2; total time=  10.3s\n",
      "[CV] END criterion=gini, max_depth=5, min_samples_leaf=2, min_samples_split=2; total time=  10.4s\n",
      "[CV] END criterion=gini, max_depth=5, min_samples_leaf=2, min_samples_split=2; total time=  10.0s\n",
      "[CV] END criterion=entropy, max_depth=5, min_samples_leaf=5, min_samples_split=4; total time=  10.9s\n",
      "[CV] END criterion=entropy, max_depth=5, min_samples_leaf=5, min_samples_split=4; total time=  10.9s\n",
      "[CV] END criterion=entropy, max_depth=5, min_samples_leaf=5, min_samples_split=4; total time=  10.8s\n",
      "[CV] END criterion=gini, max_depth=3, min_samples_leaf=2, min_samples_split=4; total time=   6.4s\n",
      "[CV] END criterion=gini, max_depth=3, min_samples_leaf=2, min_samples_split=4; total time=   6.3s\n",
      "[CV] END criterion=gini, max_depth=5, min_samples_leaf=5, min_samples_split=4; total time=  10.1s\n",
      "[CV] END criterion=entropy, max_depth=10, min_samples_leaf=2, min_samples_split=2; total time=  21.7s\n",
      "[CV] END criterion=gini, max_depth=5, min_samples_leaf=5, min_samples_split=4; total time=   9.8s\n",
      "[CV] END criterion=gini, max_depth=3, min_samples_leaf=2, min_samples_split=2; total time=   6.1s\n",
      "[CV] END criterion=gini, max_depth=5, min_samples_leaf=5, min_samples_split=4; total time=  10.4s\n",
      "[CV] END criterion=gini, max_depth=3, min_samples_leaf=2, min_samples_split=2; total time=   6.0s\n",
      "[CV] END criterion=gini, max_depth=3, min_samples_leaf=2, min_samples_split=2; total time=   6.3s\n",
      "[CV] END criterion=entropy, max_depth=3, min_samples_leaf=2, min_samples_split=4; total time=   6.2s\n",
      "[CV] END criterion=entropy, max_depth=3, min_samples_leaf=2, min_samples_split=4; total time=   6.0s\n",
      "[CV] END criterion=entropy, max_depth=3, min_samples_leaf=2, min_samples_split=4; total time=   5.7s\n",
      "[CV] END criterion=entropy, max_depth=10, min_samples_leaf=5, min_samples_split=3; total time=  18.7s\n",
      "[CV] END criterion=entropy, max_depth=10, min_samples_leaf=5, min_samples_split=3; total time=  18.9s\n",
      "[CV] END criterion=entropy, max_depth=10, min_samples_leaf=5, min_samples_split=3; total time=  17.6s\n",
      "best_params = {'min_samples_split': 3, 'min_samples_leaf': 5, 'max_depth': 10, 'criterion': 'entropy'}\n"
     ]
    }
   ],
   "source": [
    "dt = DecisionTreeClassifier()\n",
    "dt_random = RandomizedSearchCV(\n",
    "                        estimator=dt,\n",
    "                        param_distributions=random_grid,\n",
    "                        cv=3,\n",
    "                        verbose=2,\n",
    "                        random_state=42,\n",
    "                        n_jobs=-1 \n",
    "                    )\n",
    "dt_random.fit(X_train, y_train)\n",
    "\n",
    "if VERBOSE:\n",
    "        print('best_params =', dt_random.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Il semble bon de préciser malgré tout que, quelque soit la méthode utilisée (manuelle ou « Grid Search »), la recherche du meilleur modèle DT est limitée par nos choix d'hyper paramètres à tester. En effet, le temps de calcul étant déjà suffisamment long, nous avons dû choisir lesquels appliquer. On n'exclue donc pas qu'une combinaison d'hyper paramètres encore meilleure puisse exister."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification (optimisée) des données brutes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle DT optimisé sur les données brutes, avec notre version manuelle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing ['gini', 3, 1, 2] Decision Tree classification…\n",
      "training_time = 3.973532199859619\n",
      "accuracy = 0.4988\n",
      "Computing ['gini', 3, 1, 3] Decision Tree classification…\n",
      "training_time = 3.8572211265563965\n",
      "accuracy = 0.4988\n",
      "Computing ['gini', 3, 1, 4] Decision Tree classification…\n",
      "training_time = 3.8475279808044434\n",
      "accuracy = 0.4988\n",
      "Computing ['gini', 3, 2, 2] Decision Tree classification…\n",
      "training_time = 3.844771146774292\n",
      "accuracy = 0.4988\n",
      "Computing ['gini', 3, 2, 3] Decision Tree classification…\n",
      "training_time = 3.842352867126465\n",
      "accuracy = 0.4988\n",
      "Computing ['gini', 3, 2, 4] Decision Tree classification…\n",
      "training_time = 3.8378307819366455\n",
      "accuracy = 0.4988\n",
      "Computing ['gini', 3, 5, 2] Decision Tree classification…\n",
      "training_time = 3.8376619815826416\n",
      "accuracy = 0.4988\n",
      "Computing ['gini', 3, 5, 3] Decision Tree classification…\n",
      "training_time = 3.854261636734009\n",
      "accuracy = 0.4988\n",
      "Computing ['gini', 3, 5, 4] Decision Tree classification…\n",
      "training_time = 3.903930187225342\n",
      "accuracy = 0.4988\n",
      "Computing ['gini', 5, 1, 2] Decision Tree classification…\n",
      "training_time = 6.231619834899902\n",
      "accuracy = 0.6938\n",
      "Computing ['gini', 5, 1, 3] Decision Tree classification…\n",
      "training_time = 6.448298931121826\n",
      "accuracy = 0.6938\n",
      "Computing ['gini', 5, 1, 4] Decision Tree classification…\n",
      "training_time = 6.376303195953369\n",
      "accuracy = 0.6938\n",
      "Computing ['gini', 5, 2, 2] Decision Tree classification…\n",
      "training_time = 6.276401996612549\n",
      "accuracy = 0.6938\n",
      "Computing ['gini', 5, 2, 3] Decision Tree classification…\n",
      "training_time = 6.210936069488525\n",
      "accuracy = 0.6938\n",
      "Computing ['gini', 5, 2, 4] Decision Tree classification…\n",
      "training_time = 6.309891700744629\n",
      "accuracy = 0.6938\n",
      "Computing ['gini', 5, 5, 2] Decision Tree classification…\n",
      "training_time = 6.254732131958008\n",
      "accuracy = 0.6938\n",
      "Computing ['gini', 5, 5, 3] Decision Tree classification…\n",
      "training_time = 6.236411809921265\n",
      "accuracy = 0.6938\n",
      "Computing ['gini', 5, 5, 4] Decision Tree classification…\n",
      "training_time = 6.264797925949097\n",
      "accuracy = 0.6938\n",
      "Computing ['gini', 10, 1, 2] Decision Tree classification…\n",
      "training_time = 12.032196044921875\n",
      "accuracy = 0.7999\n",
      "Computing ['gini', 10, 1, 3] Decision Tree classification…\n",
      "training_time = 12.055466175079346\n",
      "accuracy = 0.801\n",
      "Computing ['gini', 10, 1, 4] Decision Tree classification…\n",
      "training_time = 12.035308122634888\n",
      "accuracy = 0.801\n",
      "Computing ['gini', 10, 2, 2] Decision Tree classification…\n",
      "training_time = 12.020408630371094\n",
      "accuracy = 0.8012\n",
      "Computing ['gini', 10, 2, 3] Decision Tree classification…\n",
      "training_time = 12.037795782089233\n",
      "accuracy = 0.8022\n",
      "Computing ['gini', 10, 2, 4] Decision Tree classification…\n",
      "training_time = 12.031675100326538\n",
      "accuracy = 0.8005\n",
      "Computing ['gini', 10, 5, 2] Decision Tree classification…\n",
      "training_time = 11.986402988433838\n",
      "accuracy = 0.8013\n",
      "Computing ['gini', 10, 5, 3] Decision Tree classification…\n",
      "training_time = 11.972743034362793\n",
      "accuracy = 0.8014\n",
      "Computing ['gini', 10, 5, 4] Decision Tree classification…\n",
      "training_time = 11.986879110336304\n",
      "accuracy = 0.8009\n",
      "Computing ['entropy', 3, 1, 2] Decision Tree classification…\n",
      "training_time = 3.933255910873413\n",
      "accuracy = 0.5127\n",
      "Computing ['entropy', 3, 1, 3] Decision Tree classification…\n",
      "training_time = 3.9324679374694824\n",
      "accuracy = 0.5127\n",
      "Computing ['entropy', 3, 1, 4] Decision Tree classification…\n",
      "training_time = 3.9352359771728516\n",
      "accuracy = 0.5127\n",
      "Computing ['entropy', 3, 2, 2] Decision Tree classification…\n",
      "training_time = 3.932708263397217\n",
      "accuracy = 0.5127\n",
      "Computing ['entropy', 3, 2, 3] Decision Tree classification…\n",
      "training_time = 3.9236881732940674\n",
      "accuracy = 0.5127\n",
      "Computing ['entropy', 3, 2, 4] Decision Tree classification…\n",
      "training_time = 3.9342830181121826\n",
      "accuracy = 0.5127\n",
      "Computing ['entropy', 3, 5, 2] Decision Tree classification…\n",
      "training_time = 3.92834210395813\n",
      "accuracy = 0.5127\n",
      "Computing ['entropy', 3, 5, 3] Decision Tree classification…\n",
      "training_time = 3.927271842956543\n",
      "accuracy = 0.5127\n",
      "Computing ['entropy', 3, 5, 4] Decision Tree classification…\n",
      "training_time = 3.9353280067443848\n",
      "accuracy = 0.5127\n",
      "Computing ['entropy', 5, 1, 2] Decision Tree classification…\n",
      "training_time = 6.5477869510650635\n",
      "accuracy = 0.7048\n",
      "Computing ['entropy', 5, 1, 3] Decision Tree classification…\n",
      "training_time = 6.544487953186035\n",
      "accuracy = 0.7048\n",
      "Computing ['entropy', 5, 1, 4] Decision Tree classification…\n",
      "training_time = 6.65073299407959\n",
      "accuracy = 0.7048\n",
      "Computing ['entropy', 5, 2, 2] Decision Tree classification…\n",
      "training_time = 7.021758794784546\n",
      "accuracy = 0.7048\n",
      "Computing ['entropy', 5, 2, 3] Decision Tree classification…\n",
      "training_time = 6.6722588539123535\n",
      "accuracy = 0.7048\n",
      "Computing ['entropy', 5, 2, 4] Decision Tree classification…\n",
      "training_time = 6.642971992492676\n",
      "accuracy = 0.7048\n",
      "Computing ['entropy', 5, 5, 2] Decision Tree classification…\n",
      "training_time = 6.551302194595337\n",
      "accuracy = 0.7048\n",
      "Computing ['entropy', 5, 5, 3] Decision Tree classification…\n",
      "training_time = 6.58602499961853\n",
      "accuracy = 0.7048\n",
      "Computing ['entropy', 5, 5, 4] Decision Tree classification…\n",
      "training_time = 6.565788984298706\n",
      "accuracy = 0.7048\n",
      "Computing ['entropy', 10, 1, 2] Decision Tree classification…\n",
      "training_time = 13.080180883407593\n",
      "accuracy = 0.8099\n",
      "Computing ['entropy', 10, 1, 3] Decision Tree classification…\n",
      "training_time = 13.243024826049805\n",
      "accuracy = 0.8105\n",
      "Computing ['entropy', 10, 1, 4] Decision Tree classification…\n",
      "training_time = 13.116933822631836\n",
      "accuracy = 0.8106\n",
      "Computing ['entropy', 10, 2, 2] Decision Tree classification…\n",
      "training_time = 13.0662100315094\n",
      "accuracy = 0.8108\n",
      "Computing ['entropy', 10, 2, 3] Decision Tree classification…\n",
      "training_time = 13.136987209320068\n",
      "accuracy = 0.8108\n",
      "Computing ['entropy', 10, 2, 4] Decision Tree classification…\n",
      "training_time = 13.069911003112793\n",
      "accuracy = 0.8111\n",
      "Computing ['entropy', 10, 5, 2] Decision Tree classification…\n",
      "training_time = 12.988472700119019\n",
      "accuracy = 0.8115\n",
      "Computing ['entropy', 10, 5, 3] Decision Tree classification…\n",
      "training_time = 12.984786987304688\n",
      "accuracy = 0.8115\n",
      "Computing ['entropy', 10, 5, 4] Decision Tree classification…\n",
      "training_time = 12.985044956207275\n",
      "accuracy = 0.8108\n",
      "parameters: ['entropy', 10, 5, 2]\n",
      "training_time = 12.988472700119019\n",
      "accuracy = 0.8115\n",
      "['entropy', 10, 5, 2] Decision Tree:\n",
      "  - Training Time: 12.99s.\n",
      "  - Accuracy: 81.15%.\n"
     ]
    }
   ],
   "source": [
    "best_dt_params, best_dt_training_time, best_dt_accuracy = find_best_dt(X_train, y_train, X_test, y_test)\n",
    "\n",
    "print('%s Decision Tree:' % best_dt_params)\n",
    "print('  - Training Time: %.2fs.' % best_dt_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (best_dt_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tout d'abord, on voit qu'on retombe sur les mêmes hyper paramètres qu'avec la méthode « Grid Search ». Et, on ne peut que remarquer que le temps de recherche de ceux-ci est beaucoup moins long. Sauvegardons-les."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [],
   "source": [
    "BEST_DT_PARAMS = {\n",
    "    'criterion': 'entropy',\n",
    "    'max_depth': 10,\n",
    "    'min_samples_leaf': 2,\n",
    "    'min_samples_split': 3\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Quant aux performances du modèle, on remarque que le modèle DT optimisé sur les données brutes a une accuracy d'environ 81%. En outre, on constate qu'en passant d'une profondeur de 49 à 10, on obtient une accuracy légèrement améliorée ainsi qu'un temps d'entraînement divisé par 2 approximativement. C'est un meilleur modèle que le modèle avec des hyper paramètres par défaut.\n",
    "\n",
    "En regardant un peu tous les modèles avec des hyper paramètres différents (une occurrence a été sauvegardée dans le fichier `find_best_dt.txt` car le temps de recherche est très long), on remarque notamment que :\n",
    "- c'est surtout le paramètre `max_depth` qui a une réelle influence sur l'accuracy et le temps de calcul ;\n",
    "- tous les modèles `max_depth=10` ont des résultats similaires."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Enfin, on sauvegarde également le temps d'entraînement et l'accuracy de la meilleure version, à savoir celle utilisant les meilleurs hyper paramètres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "dt_training_time, dt_accuracy = best_dt_training_time, best_dt_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification après une ACP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle DT initial après réduction de dimension grâce à l'ACP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing Decision Tree classification…\n",
      "training_time = 0.49639892578125\n",
      "accuracy = 0.6952\n",
      "Decision Tree after PCA:\n",
      "  - Training Time: 0.50s.\n",
      "  - Accuracy: 69.52%.\n"
     ]
    }
   ],
   "source": [
    "pca_default_dt_training_time, pca_default_dt_accuracy, _ = find_dt(Xp_train, y_train, Xp_test, y_test)\n",
    "\n",
    "print('Decision Tree after PCA:')\n",
    "print('  - Training Time: %.2fs.' % pca_default_dt_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (pca_default_dt_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle DT initial après réduction de dimension grâce à l'ACP a une accuracy d'environ 70%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On fait de même avec la version optimisée."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing {'criterion': 'entropy', 'max_depth': 10, 'min_samples_leaf': 2, 'min_samples_split': 3} Decision Tree classification…\n",
      "training_time = 0.5024430751800537\n",
      "accuracy = 0.7273\n",
      "Best Decision Tree after PCA:\n",
      "  - Training Time: 0.50s.\n",
      "  - Accuracy: 72.73%.\n"
     ]
    }
   ],
   "source": [
    "pca_best_dt_training_time, pca_best_dt_accuracy, _ = find_dt(Xp_train, y_train, Xp_test, y_test, BEST_DT_PARAMS)\n",
    "\n",
    "print('Best Decision Tree after PCA:')\n",
    "print('  - Training Time: %.2fs.' % pca_best_dt_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (pca_best_dt_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle DT optimisé après réduction de dimension grâce à l'ACP a une accuracy d'environ 73%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Dans les 2 cas, on observe qu'on a considérablement diminué le temps d'entraînement, comparé au même modèle sans réduction de dimension. Néanmoins, l'accuracy a emboîté le pas. En effet, on a perdu 10% d'accuracy, en obtenant désormais environ 70%. Par conséquent, ces 2 modèles semblent plutôt équivalents. Pour décider s'ils sont meilleurs que le même modèle sans réduction de dimension, il faut faire un compromis. Est-ce qu'une dizaine de secondes valent le coup pour gagner 10% d'accuracy ? On peut dire que oui.\n",
    "\n",
    "On va comme précédemment sauvegarder le temps d'entraînement et l'accuracy du modèle avec les meilleurs hyper paramètres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_dt_training_time, pca_dt_accuracy = pca_best_dt_training_time, pca_best_dt_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification après une ALD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle DT initial après une réduction de dimension grâce à l'ALD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing Decision Tree classification…\n",
      "training_time = 0.805649995803833\n",
      "accuracy = 0.7733\n",
      "Decision Tree after LDA:\n",
      "  - Training Time: 0.81s.\n",
      "  - Accuracy: 77.33%.\n"
     ]
    }
   ],
   "source": [
    "lda_default_dt_training_time, lda_default_dt_accuracy, _ = find_dt(Xl_train, y_train, Xl_test, y_test)\n",
    "\n",
    "print('Decision Tree after LDA:')\n",
    "print('  - Training Time: %.2fs.' % lda_default_dt_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (lda_default_dt_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle DT initial après une réduction de dimension grâce à l'ALD a une accuracy d'environ 78%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On observe un temps de calcul similaire au modèle avec ACP, donc beaucoup plus court que le modèle sans réduction de dimension. De plus, l'accuracy se rapproche désormais plus de 80% que de 70% contrairement au modèle avec ACP, pour un même temps d'entraînement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On fait de même avec la version optimisée."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing {'criterion': 'entropy', 'max_depth': 10, 'min_samples_leaf': 2, 'min_samples_split': 3} Decision Tree classification…\n",
      "training_time = 0.7706642150878906\n",
      "accuracy = 0.8139\n",
      "Best Decision Tree after LDA:\n",
      "  - Training Time: 0.77s.\n",
      "  - Accuracy: 81.39%.\n"
     ]
    }
   ],
   "source": [
    "lda_best_dt_training_time, lda_best_dt_accuracy, _ = find_dt(Xl_train, y_train, Xl_test, y_test, BEST_DT_PARAMS)\n",
    "\n",
    "print('Best Decision Tree after LDA:')\n",
    "print('  - Training Time: %.2fs.' % lda_best_dt_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (lda_best_dt_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle DT optimisé après une réduction de dimension grâce à l'ALD a une accuracy d'environ 81%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Le modèle avec les meilleurs hyper paramètres a quant à lui une accuracy qui passe la barre des 80%, cela reste tout de même similaire au modèle précédent. Le temps de calcul est également similaire. Ce modèle est donc meilleur que celui avec ACP en amont, et par transitivité, meilleur que celui sans réduction de dimension. On va donc sauvegarder ses valeurs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_dt_training_time, lda_dt_accuracy = lda_best_dt_training_time, lda_best_dt_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Forêt aléatoire\n",
    "\n",
    "Ici, on utilise un ensemble d'arbres de décision, entraînés sur des sous-ensembles de données légèrement différents."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Mise en place"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut récupérer le temps d'entraînement et l'accuracy grâce à une fonction, qui, de la même manière qu'avec les arbres de décision, utilise dans un premier temps les hyper paramètres par défaut d'un classifieur forêt aléatoire."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_rf(X_train, y_train, X_test, y_test):\n",
    "    if VERBOSE:\n",
    "        print('Computing Random Forest classification…')\n",
    "    \n",
    "    rf = RandomForestClassifier()\n",
    "    start_time = time.time()\n",
    "    rf.fit(X_train, y_train)\n",
    "    rf_training_time = time.time() - start_time\n",
    "    rf_accuracy = rf.score(X_test, y_test)\n",
    "\n",
    "    if VERBOSE:\n",
    "        print('training_time =', rf_training_time)\n",
    "        print('accuracy =', rf_accuracy)\n",
    "    \n",
    "    return rf_training_time, rf_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification (initiale) des données brutes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle RF initial sur les données brutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing Random Forest classification…\n",
      "training_time = 52.043970823287964\n",
      "accuracy = 0.8777\n",
      "Random Forest:\n",
      "  - Training Time: 52.04s.\n",
      "  - Accuracy: 87.77%.\n"
     ]
    }
   ],
   "source": [
    "default_rf_training_time, default_rf_accuracy = find_rf(X_train, y_train, X_test, y_test)\n",
    "\n",
    "print('Random Forest:')\n",
    "print('  - Training Time: %.2fs.' % default_rf_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (default_rf_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle RF initial sur les données brutes a une accuracy d'environ 88%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "C'est le plus long temps de calcul obtenu jusqu'ici, mais aussi la meilleure accuracy. Il est sûrement possible d'améliorer ce modèle en changeant les hyper paramètres."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Optimisation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Nous allons utiliser uniquement la méthode « Grid Search » cette fois."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Par défaut, les hyper paramètres sont définis comme ci-dessous :\n",
    "\n",
    "- `n_estimators=100`\n",
    "- `max_features='auto'`\n",
    "- `max_depth=None`\n",
    "- `min_samples_split=2`\n",
    "- `min_samples_leaf=1`\n",
    "- `bootstrap=True`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On commence par créer la _random grid_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'n_estimators': [200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000], 'max_features': ['auto', 'sqrt'], 'max_depth': [10, 20, 30, 40, 50, 60, 70, 80, 90, 100, 110], 'min_samples_split': [2, 5, 10], 'min_samples_leaf': [1, 2, 4], 'bootstrap': [True, False]}\n"
     ]
    }
   ],
   "source": [
    "n_estimators = [int(x) for x in np.linspace(start = 200, stop = 2000, num = 10)]\n",
    "max_features = ['auto', 'sqrt']\n",
    "max_depth = [int(x) for x in np.linspace(10, 110, num = 11)]\n",
    "min_samples_split = [2, 5, 10]\n",
    "min_samples_leaf = [1, 2, 4]\n",
    "bootstrap = [True, False]\n",
    "\n",
    "random_grid = {\n",
    "    'n_estimators': n_estimators,\n",
    "    'max_features': max_features,\n",
    "    'max_depth': max_depth,\n",
    "    'min_samples_split': min_samples_split,\n",
    "    'min_samples_leaf': min_samples_leaf,\n",
    "    'bootstrap': bootstrap\n",
    "}\n",
    "\n",
    "if VERBOSE:\n",
    "    print(random_grid)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On récupère ensuite les meilleurs hyper paramètres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Fitting 3 folds for each of 10 candidates, totalling 30 fits\n",
      "[CV] END bootstrap=True, max_depth=50, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=200; total time= 2.6min\n",
      "[CV] END bootstrap=True, max_depth=50, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=200; total time= 2.6min\n",
      "[CV] END bootstrap=True, max_depth=50, max_features=sqrt, min_samples_leaf=2, min_samples_split=10, n_estimators=200; total time= 3.1min\n",
      "[CV] END bootstrap=True, max_depth=80, max_features=auto, min_samples_leaf=2, min_samples_split=2, n_estimators=1000; total time=12.2min\n",
      "[CV] END bootstrap=True, max_depth=80, max_features=auto, min_samples_leaf=2, min_samples_split=2, n_estimators=1000; total time=11.3min\n",
      "[CV] END bootstrap=True, max_depth=70, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=200; total time= 2.1min\n",
      "[CV] END bootstrap=False, max_depth=70, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=1000; total time=19.4min\n",
      "[CV] END bootstrap=False, max_depth=70, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=1000; total time=19.5min\n",
      "[CV] END bootstrap=False, max_depth=70, max_features=sqrt, min_samples_leaf=1, min_samples_split=10, n_estimators=1000; total time=19.5min\n",
      "[CV] END bootstrap=True, max_depth=70, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=200; total time= 2.1min\n",
      "[CV] END bootstrap=True, max_depth=70, max_features=auto, min_samples_leaf=2, min_samples_split=10, n_estimators=200; total time= 2.1min\n",
      "[CV] END bootstrap=True, max_depth=80, max_features=auto, min_samples_leaf=2, min_samples_split=2, n_estimators=1000; total time=10.7min\n",
      "[CV] END bootstrap=False, max_depth=90, max_features=auto, min_samples_leaf=4, min_samples_split=10, n_estimators=1600; total time=27.4min\n",
      "[CV] END bootstrap=False, max_depth=90, max_features=auto, min_samples_leaf=4, min_samples_split=10, n_estimators=1600; total time=27.6min\n",
      "[CV] END bootstrap=False, max_depth=90, max_features=auto, min_samples_leaf=4, min_samples_split=10, n_estimators=1600; total time=27.6min\n",
      "[CV] END bootstrap=False, max_depth=100, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=600; total time= 9.4min\n",
      "[CV] END bootstrap=True, max_depth=70, max_features=auto, min_samples_leaf=1, min_samples_split=5, n_estimators=1200; total time=13.0min\n",
      "[CV] END bootstrap=True, max_depth=70, max_features=auto, min_samples_leaf=1, min_samples_split=5, n_estimators=1200; total time=13.0min\n",
      "[CV] END bootstrap=True, max_depth=70, max_features=auto, min_samples_leaf=1, min_samples_split=5, n_estimators=1200; total time=13.0min\n",
      "[CV] END bootstrap=False, max_depth=100, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=600; total time= 9.4min\n",
      "[CV] END bootstrap=False, max_depth=100, max_features=sqrt, min_samples_leaf=4, min_samples_split=10, n_estimators=600; total time= 9.4min\n",
      "[CV] END bootstrap=False, max_depth=70, max_features=auto, min_samples_leaf=2, min_samples_split=2, n_estimators=600; total time= 9.9min\n",
      "[CV] END bootstrap=False, max_depth=70, max_features=auto, min_samples_leaf=2, min_samples_split=2, n_estimators=600; total time= 9.9min\n",
      "[CV] END bootstrap=False, max_depth=70, max_features=auto, min_samples_leaf=2, min_samples_split=2, n_estimators=600; total time= 9.8min\n",
      "[CV] END bootstrap=True, max_depth=100, max_features=auto, min_samples_leaf=1, min_samples_split=5, n_estimators=1800; total time=17.6min\n",
      "[CV] END bootstrap=True, max_depth=100, max_features=auto, min_samples_leaf=1, min_samples_split=5, n_estimators=1800; total time=17.6min\n",
      "[CV] END bootstrap=True, max_depth=100, max_features=auto, min_samples_leaf=1, min_samples_split=5, n_estimators=1800; total time=17.1min\n",
      "[CV] END bootstrap=False, max_depth=20, max_features=auto, min_samples_leaf=1, min_samples_split=2, n_estimators=2000; total time=24.7min\n",
      "[CV] END bootstrap=False, max_depth=20, max_features=auto, min_samples_leaf=1, min_samples_split=2, n_estimators=2000; total time=23.9min\n",
      "[CV] END bootstrap=False, max_depth=20, max_features=auto, min_samples_leaf=1, min_samples_split=2, n_estimators=2000; total time=23.6min\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomizedSearchCV(cv=3, estimator=RandomForestClassifier(), n_jobs=-1,\n",
       "                   param_distributions={'bootstrap': [True, False],\n",
       "                                        'max_depth': [10, 20, 30, 40, 50, 60,\n",
       "                                                      70, 80, 90, 100, 110],\n",
       "                                        'max_features': ['auto', 'sqrt'],\n",
       "                                        'min_samples_leaf': [1, 2, 4],\n",
       "                                        'min_samples_split': [2, 5, 10],\n",
       "                                        'n_estimators': [200, 400, 600, 800,\n",
       "                                                         1000, 1200, 1400, 1600,\n",
       "                                                         1800, 2000]},\n",
       "                   random_state=42, verbose=2)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "best_params = {'n_estimators': 600, 'min_samples_split': 2, 'min_samples_leaf': 2, 'max_features': 'auto', 'max_depth': 70, 'bootstrap': False}\n"
     ]
    }
   ],
   "source": [
    "rf = RandomForestClassifier()\n",
    "rf_random = RandomizedSearchCV(estimator = rf, param_distributions = random_grid, cv=3, verbose=2, random_state=42, n_jobs = -1)\n",
    "rf_random.fit(X_train, y_train)\n",
    "\n",
    "if VERBOSE:\n",
    "    print('best_params =', rf_random.best_params_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Comme précédemment, on rappelle que la recherche du meilleur modèle RF est limitée par nos choix d'hyper paramètres à tester à cause du temps de calcul déjà extrèmement élevé."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On sauvegarde les meilleurs hyper paramètres."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [],
   "source": [
    "BEST_RF_PARAMS = {\n",
    "    'n_estimators': 600,\n",
    "    'min_samples_split': 2,\n",
    "    'min_samples_leaf': 2,\n",
    "    'max_features': 'auto',\n",
    "    'max_depth': 70,\n",
    "    'bootstrap': False\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Élaborons une nouvelle fonction qui permet de récupérer le temps d'entraînement et l'accuracy du meilleur modèle de forêt aléatoire."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_rf_optimized(X_train, y_train, X_test, y_test):\n",
    "    if VERBOSE:\n",
    "        print('Computing Random Forest classification…')\n",
    "\n",
    "    rf = RandomForestClassifier(\n",
    "        n_estimators=BEST_RF_PARAMS.get('n_estimators'),\n",
    "        max_features=BEST_RF_PARAMS.get('max_features'),\n",
    "        max_depth=BEST_RF_PARAMS.get('max_depth'),\n",
    "        min_samples_split=BEST_RF_PARAMS.get('min_samples_split'),\n",
    "        min_samples_leaf=BEST_RF_PARAMS.get('min_samples_leaf'),\n",
    "        bootstrap=BEST_RF_PARAMS.get('bootstrap'),\n",
    "    )\n",
    "    start_time = time.time()\n",
    "    rf.fit(X_train, y_train)\n",
    "    rf_training_time = time.time() - start_time\n",
    "    rf_accuracy = rf.score(X_test, y_test)\n",
    "\n",
    "    if VERBOSE:\n",
    "        print('parameters:', BEST_RF_PARAMS)\n",
    "        print('training_time =', rf_training_time)\n",
    "        print('accuracy =', rf_accuracy)\n",
    "    \n",
    "    return rf_training_time, rf_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification (optimisée) des données brutes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle RF optimisé sur les données brutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing Random Forest classification…\n",
      "parameters: {'n_estimators': 600, 'min_samples_split': 2, 'min_samples_leaf': 2, 'max_features': 'auto', 'max_depth': 70, 'bootstrap': False}\n",
      "training_time = 477.56874322891235\n",
      "accuracy = 0.8825\n",
      "Best Random Forest:\n",
      "  - Training Time: 477.57s.\n",
      "  - Accuracy: 88.25%.\n"
     ]
    }
   ],
   "source": [
    "best_rf_training_time, best_rf_accuracy = find_rf_optimized(X_train, y_train, X_test, y_test)\n",
    "\n",
    "print('Best Random Forest:')\n",
    "print('  - Training Time: %.2fs.' % best_rf_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (best_rf_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle RF optimisé après réduction de dimension grâce à l'ACP a une accuracy d'environ 88%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "La version utilisant les meilleurs hyper paramètres possibles parmi ceux fournis via la _random grid_ semble obtenir un résultat à peine meilleur voire équivalent à ce niveau de différence. Cependant, le temps d'entraînement a été drastiquement augmenté. On peut donc considérer ce modèle comme moins performant que celui utilisant les paramètres par défaut. On sauvegarde donc la version initiale."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_training_time, rf_accuracy = default_rf_training_time, default_rf_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification après une ACP"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle RF initial après réduction de dimension grâce à l'ACP."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing Random Forest classification…\n",
      "training_time = 10.587687969207764\n",
      "accuracy = 0.7788\n",
      "Random Forest after PCA:\n",
      "  - Training Time: 10.59s.\n",
      "  - Accuracy: 77.88%.\n"
     ]
    }
   ],
   "source": [
    "pca_default_rf_training_time, pca_default_rf_accuracy = find_rf(Xp_train, y_train, Xp_test, y_test)\n",
    "\n",
    "print('Random Forest after PCA:')\n",
    "print('  - Training Time: %.2fs.' % pca_default_rf_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (pca_default_rf_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle RF initial après réduction de dimension grâce à l'ACP a une accuracy d'environ 78%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On fait de même avec la version optimisée."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing Random Forest classification…\n",
      "parameters: {'n_estimators': 600, 'min_samples_split': 2, 'min_samples_leaf': 2, 'max_features': 'auto', 'max_depth': 70, 'bootstrap': False}\n",
      "training_time = 100.42728209495544\n",
      "accuracy = 0.7786\n",
      "Best Random Forest after PCA:\n",
      "  - Training Time: 100.43s.\n",
      "  - Accuracy: 77.86%.\n"
     ]
    }
   ],
   "source": [
    "pca_best_rf_training_time, pca_best_rf_accuracy = find_rf_optimized(Xp_train, y_train, Xp_test, y_test)\n",
    "\n",
    "print('Best Random Forest after PCA:')\n",
    "print('  - Training Time: %.2fs.' % pca_best_rf_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (pca_best_rf_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle RF optimisé après réduction de dimension grâce à l'ACP a une accuracy d'environ 78%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ici, on retrouve comme sur les données brutes, une accuracy équivalente à la version par défaut, mais un temps d'entraînement bien plus élevé. Il n'est donc pas non plus rentable d'utiliser un tel modèle."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "pca_rf_training_time, pca_rf_accuracy = pca_default_rf_training_time, pca_default_rf_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Classification après une ALD"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On calcule le temps d'entraînement et l'accuracy du modèle RF initial après réduction de dimension grâce à l'ALD."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing Random Forest classification…\n",
      "training_time = 16.16219997406006\n",
      "accuracy = 0.8396\n",
      "Random Forest after LDA:\n",
      "  - Training Time: 16.16s.\n",
      "  - Accuracy: 83.96%.\n"
     ]
    }
   ],
   "source": [
    "lda_default_rf_training_time, lda_default_rf_accuracy = find_rf(Xl_train, y_train, Xl_test, y_test)\n",
    "\n",
    "print('Random Forest after LDA:')\n",
    "print('  - Training Time: %.2fs.' % lda_default_rf_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (lda_default_rf_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle RF initial après réduction de dimension grâce à l'ALD a une accuracy d'environ 84%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On fait de même avec la version optimisée."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing Random Forest classification…\n",
      "parameters: {'n_estimators': 600, 'min_samples_split': 2, 'min_samples_leaf': 2, 'max_features': 'auto', 'max_depth': 70, 'bootstrap': False}\n",
      "training_time = 157.9050817489624\n",
      "accuracy = 0.8401\n",
      "Best Random Forest after LDA:\n",
      "  - Training Time: 157.91s.\n",
      "  - Accuracy: 84.01%.\n"
     ]
    }
   ],
   "source": [
    "lda_best_rf_training_time, lda_best_rf_accuracy = find_rf_optimized(Xl_train, y_train, Xl_test, y_test)\n",
    "\n",
    "print('Best Random Forest after LDA:')\n",
    "print('  - Training Time: %.2fs.' % lda_best_rf_training_time)\n",
    "print('  - Accuracy: %.2f%%.' % (lda_best_rf_accuracy * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On remarque que le modèle RF optimisé après réduction de dimension grâce à l'ALD a une accuracy d'environ 84%."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Encore une fois, on se retrouve avec moins de 1% d'accuracy de différence, mais un temps d'entraînement beaucoup trop long pour faire de ce modèle un meilleur que celui par défaut."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {},
   "outputs": [],
   "source": [
    "lda_rf_training_time, lda_rf_accuracy = lda_default_rf_training_time, lda_default_rf_accuracy"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Conclusion\n",
    "\n",
    "À partir des accuracy calculées dans la partie précédente, il est possible de déterminer quelles sont les meilleures pour ces données."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut trier toutes les accuracy par ordre décroissant."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "models = {\n",
    "    '%d-Nearest Neighbors' % knn_k: (knn_training_time, knn_accuracy),\n",
    "    '%d-Nearest Neighbors after PCA' % pca_knn_k: (pca_knn_training_time, pca_knn_accuracy),\n",
    "    '%d-Nearest Neighbors after LDA' % lda_knn_k: (lda_knn_training_time, lda_knn_accuracy),\n",
    "    'Naive Bayes': (nb_training_time, nb_accuracy),\n",
    "    'Naive Bayes after PCA': (pca_nb_training_time, pca_nb_accuracy),\n",
    "    'Naive Bayes after LDA': (lda_nb_training_time, lda_nb_accuracy),\n",
    "    'Decision Tree': (dt_training_time, dt_accuracy),\n",
    "    'Decision Tree after PCA': (pca_dt_training_time, pca_dt_accuracy),\n",
    "    'Decision Tree after LDA': (lda_dt_training_time, lda_dt_accuracy),\n",
    "    'Random Forest': (rf_training_time, rf_accuracy),\n",
    "    'Random Forest after PCA': (pca_rf_training_time, pca_rf_accuracy),\n",
    "    'Random Forest after LDA': (lda_rf_training_time, lda_rf_accuracy)\n",
    "}\n",
    "\n",
    "models = dict(sorted(models.items(), key=lambda item: item[1][1], reverse=True))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On peut alors déterminer quels sont les meilleurs modèles de classification, selon leur accuracy, pour ce jeu de données :"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Models:\n",
      "  #1: Random Forest, with a training time of 52.04s and an accuracy of 87.77%.\n",
      "  #2: 4-Nearest Neighbors, with a training time of 0.07s and an accuracy of 85.77%.\n",
      "  #3: 41-Nearest Neighbors after LDA, with a training time of 0.02s and an accuracy of 84.15%.\n",
      "  #4: Random Forest after LDA, with a training time of 16.16s and an accuracy of 83.96%.\n",
      "  #5: Naive Bayes after LDA, with a training time of 0.01s and an accuracy of 81.72%.\n",
      "  #6: Decision Tree after LDA, with a training time of 0.77s and an accuracy of 81.39%.\n",
      "  #7: Decision Tree, with a training time of 12.99s and an accuracy of 81.15%.\n",
      "  #8: Random Forest after PCA, with a training time of 10.59s and an accuracy of 77.88%.\n",
      "  #9: 10-Nearest Neighbors after PCA, with a training time of 0.02s and an accuracy of 76.91%.\n",
      "  #10: Decision Tree after PCA, with a training time of 0.50s and an accuracy of 72.73%.\n",
      "  #11: Naive Bayes after PCA, with a training time of 0.02s and an accuracy of 67.72%.\n",
      "  #12: Naive Bayes, with a training time of 0.26s and an accuracy of 58.56%.\n"
     ]
    }
   ],
   "source": [
    "print('Best Models:')\n",
    "\n",
    "i = 0\n",
    "\n",
    "for model in models:\n",
    "    i += 1\n",
    "    print('  #%d: %s, with a training time of %.2fs and an accuracy of %.2f%%.' % (i, model, models.get(model)[0], models.get(model)[1] * 100))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "On a donc tous nos modèles triés par meilleure accuracy. On remarque que : \n",
    "- RF semble être le meilleur modèle sur les données brutes, mais relativement long;\n",
    "- RF semble être aussi le meilleur après réduction de dimension grâce à l'ACP; \n",
    "- _k_-NN semble être le meilleur après réduction de dimension grâce à l'ALD.\n",
    "\n",
    "Egalement, de manière générale, sur ce jeu de données :\n",
    "- faire une réduction de dimension n'est pas forcément pertinent;\n",
    "- les modèles après ACP sont peu performants par rapport aux autres;\n",
    "-_k_-NN est un très bon modèle quelles que soient les données.\n",
    "\n",
    "Il semble cependant bon de noter que à accuracy relativement similaire, il est préférable de choisir un classifieur ayant un temps d'exécution plus court. En effet, ce qui compte le plus est le rapport accuracy/temps d'entraînement."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "En tenant compte de l'accuracy et du temps d'entraînement, nous trouvons que 4-NN sur les données brutes avec est le meilleur modèle. En effet, RF sur les données brutes donne une accuracy supérieure de 2% mais ce différentiel n'est pas suffisant pour « justifier » la différence de 52 secondes de temps d'entraînement."
   ]
  }
 ],
 "metadata": {
  "interpreter": {
   "hash": "308a216d09664f400cdb2f6c5e0ae139565d7a28869de5b9540c5579b2a69035"
  },
  "kernelspec": {
   "display_name": "Python 3.10.1 64-bit ('3.10.1': pyenv)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.7"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}