32_Evaluacion_Modelos_Clasificacion.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 32 - Evaluacion de Modelos de Clasificacion\n",
    "\n",
    "\n",
    "* Una vez generado nuestro modelo o hipótesis, tenemos que ser capaces de ***cotejar los resultados obtenidos***.\n",
    "\n",
    "\n",
    "* Para los problemas de Clasificación se utilizan (entre otros) las siguientes métricas de evaluación:\n",
    "\n",
    "    - ***Accuracy***\n",
    "    - ***Precision***\n",
    "    - ***Recall***\n",
    "    - ***F1***\n",
    "\n",
    "\n",
    "* Para ver como clasifica el modelo generado usamos la denominada ***“Matriz de Confusión”*** en la cual se muestra el número de elementos que clasifica como verdaderos y falsos positivos y verdaderos y falsos negativos.\n",
    "\n",
    "\n",
    "* Veamos a continuación los detalles de las métricas de evaluación y la matriz de confusión:\n",
    "\n",
    "\n",
    "<hr>\n",
    "\n",
    "\n",
    "## - Matriz de confusión\n",
    "\n",
    "* Cuando hablamos de clasificación tenemos que medir cuanto hemos acertado y como lo hemos acertado, por ello las ***matrices de confusión nos ayudan a ver el acierto o fallo que tiene nuestro modelo a la hora de clasificar***. La matriz de confusión tiene la siguiente forma:\n",
    "\n",
    "    - **True Positive (TP)**: Valores positivos clasificados como positivos\n",
    "    - **True Negative (TN)**: Valores negativos clasificados como negativos\n",
    "    - **False Positive (FP)**: Valores negativos clasificados como positivos\n",
    "    - **False Negative (FN)**: Valore positivos clasificados como negativos\n",
    "    \n",
    "<img src=\"./imgs/31_01_eval.png\" style=\"width: 400px;\"/>\n",
    "\n",
    "\n",
    "## - Accuracy\n",
    "\n",
    "* Indica el porcentaje de acierto:\n",
    "\n",
    "$$Acuracy = \\frac{TP + TN}{TP + FN + FP + TN}$$\n",
    "\n",
    "## - Precision\n",
    "\n",
    "* Los positivos correctamente pronosticados entre todos los ejemplos pronosticados como positivos. \n",
    "\n",
    "\n",
    "* \"***De lo que hemos pronosticado, cuanto hemos acertado***\"\n",
    "\n",
    "\n",
    "$$Precision = \\frac{TP}{TP + FP}$$\n",
    "\n",
    "\n",
    "<img src=\"./imgs/31_02_eval.png\" style=\"width: 400px;\"/>\n",
    "                                                               \n",
    "                                                               \n",
    "\n",
    "## - Recall\n",
    "\n",
    "\n",
    "* Los positivos correctamente pronosticados, entre todos los ejemplos que son positivos. \n",
    "\n",
    "\n",
    "* \"***De lo que hay en realidad, cuanto hemos acertado***\"\n",
    "\n",
    "\n",
    "$$Recall = \\frac{TP}{TP + FN}$$\n",
    "\n",
    "\n",
    "<img src=\"./imgs/31_03_eval.png\" style=\"width: 400px;\"/>\n",
    "\n",
    "\n",
    "## - F1 (Precision-Recall)\n",
    "\n",
    "\n",
    "* Es una media que combina la precisión y el recall. \n",
    "\n",
    "\n",
    "* Si es cercano a ***1*** significa que la clasificación realizada es buena. \n",
    "\n",
    "\n",
    "* Si es cercano a ***0*** la clasificación realizada es mala.\n",
    "\n",
    "\n",
    "$$F1 = 2 \\cdot \\frac{Precision \\cdot Recall}{Precision + Recall}$$\n",
    "\n",
    "\n",
    "<img src=\"./imgs/31_03_eval.png\" style=\"width: 400px;\"/>\n",
    "\n",
    "\n",
    "## - Métricas en Sckit-Learn:\n",
    "\n",
    "* Scikit-Learn dispone del módulo “metrics” donde tiene implementados todos estos métodos de evaluación\n",
    "\n",
    "\n",
    "* Como norma general se le pasa como parámetros un array con la clasificación verdadera y otro array con la predicción:\n",
    "\n",
    "    - **metrics.confusion_matrix(y_true, y_pred[, …])**: Compute confusion matrix to evaluate the accuracy of a classification\n",
    "    - **metrics.accuracy_score(y_true, y_pred[, …])**: Accuracy classification score\n",
    "    - **metrics.precision_score(y_true, y_pred[, …])**: Compute the precision\n",
    "    - **metrics.recall_score(y_true, y_pred[, …])**: Compute the recall\n",
    "    - **metrics.f1_score(y_true, y_pred[, labels, …])**: Compute the F1 score, also known as balanced F-score or F-measure\n",
    "    - **metrics.classification_report(y_true, y_pred)**: Build a text report showing the main classification metrics\n",
    "\n",
    "\n",
    "\n",
    "<hr>\n",
    "\n",
    "\n",
    "# Ejemplo: Clasificación Binaria\n",
    "\n",
    "\n",
    "* Supongamos que tenemos que clasificar 10 elementos de dos clases y la realizadad y la predicción vienen dados por:\n",
    "\n",
    "    - y_real =    [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]\n",
    "    - y_predict = [1, 1, 0, 1, 1, 0, 0, 1, 1, 0]\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_real =    [1, 1, 1, 1, 1, 0, 0, 0, 0, 0]\n",
    "y_predict = [1, 1, 0, 1, 1, 0, 0, 1, 1, 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### - Matriz de confusión\n",
    "\n",
    "* En el ejemplo tenemos:\n",
    "\n",
    "    - 5 Elementos que son de la clase 0\n",
    "    - 5 Elementos que son de la clase 0\n",
    "    - True Positive (TP): De los que son de la clase 1, predigo 4 correctamente.\n",
    "    - False Negative (FN): De los que son de la clase 1, predigo 1 incorrectamente.\n",
    "    - True Negative (TN): De los que son de la clase 0, predigo 3 correctamente.\n",
    "    - False Positive (FP): De los que son de la clase 0, predigo 2 incorrectamente.\n",
    "    \n",
    "    \n",
    "\n",
    "* Podemos ver este resultado en la matriz de confusión:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[4, 1],\n",
       "       [2, 3]])"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import confusion_matrix\n",
    "\n",
    "confusion_matrix(y_true=y_real, y_pred=y_predict, labels=[1, 0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### - Accuracy\n",
    "\n",
    "* Basandonos en el ejemplo, calculamos el accuracy como:\n",
    "\n",
    "$$Acuracy = \\frac{TP + TN}{TP + FN + FP + TN} = \\frac{4 + 3}{4 + 1 + 2 + 3} = \\frac{7}{10} = 0.7$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import accuracy_score\n",
    "\n",
    "accuracy_score(y_true=y_real, y_pred=y_predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### - Precision\n",
    "\n",
    "* Basandonos en el ejemplo, calculamos la precision como:\n",
    "\n",
    "$$Precision = \\frac{TP}{TP + FP} = \\frac{4}{4 + 2} = \\frac{4}{6} = 0.67$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6666666666666666"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import precision_score\n",
    "\n",
    "precision_score(y_true=y_real, y_pred=y_predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### - Recall\n",
    "\n",
    "* Basandonos en el ejemplo, calculamos el recall como:\n",
    "\n",
    "$$Recall = \\frac{TP}{TP + FN} = \\frac{4}{4 + 1} = \\frac{4}{5} = 0.8$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import recall_score\n",
    "\n",
    "recall_score(y_true=y_real, y_pred=y_predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### - F1\n",
    "\n",
    "* Basandonos en el ejemplo, calculamos el F1 como:\n",
    "\n",
    "$$F1 = 2 \\cdot \\frac{Precision \\cdot Recall}{Precision + Recall} = 2 \\cdot \\frac{0.67 \\cdot 0.8}{0.67 + 0.8} = 2 \\cdot \\frac{0.54}{1.47} = 0.73$$"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.7272727272727272"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.metrics import f1_score\n",
    "\n",
    "f1_score(y_true=y_real, y_pred=y_predict)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Para representar de una forma visual más agradable y legible la matriz de confusión, podemos usar el siguiente código que tiene la función \"plot_confusion_matrix()\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVAAAAEmCAYAAAA0k8gFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmYHFW9//H3ZyYLCQmbYYkhEJaAYISEHVllk10QkH2RTbgiIKAsIgKPuFz1pywiBtnVCIhwI+LlIogshkAIARNDhAC5LJEkhDUkkOX7+6POXJpOT3enp3u6Zubzep566K46ferbU/Q3p86pOqWIwMzMll1LswMwM+uqnEDNzGrkBGpmViMnUDOzGjmBmpnVyAnUzKxGTqBWFUlHSvqfOtRzo6Tv1iOmepK0uqSHJL0r6ScdqOcCSb8qsX47SY9LWrljkVqeOIF2YZJekvShpEFF6ydJCknDqqhjWCrbq1y5iPhNROzRsYg7RpnTJU2WNE/SK5Jul/SZOlR/MjAHWCEizq61koj4XkScWLhO0lDge8C+EfFmx8K0PHEC7fpeBA5ve5OSSb967qBScu1ElwNnAKcDqwAbAHcB+9Sh7rWBf0YD7iyJiJcjYqeImFXvuq25nEC7vluAYwreHwvcXFhA0j6SnpL0jqSXJV1csPmh9N+3JL0naVtJx0l6VNJPJc0FLk7rHkn1fTOVbVsWSrqxVHCSRkmamE6NbwWWK9q+b2oxvyXp75I2aaee4cBXgcMj4oGI+CAi3k8t4x+kMitKulnSbEkzJF0oqSVtO07SI5J+LOlNSS9K2ittuzH93dq+127FXQ2Sdpb0SsH7cyW9mr7XNEm7pvUXS/p1Qbn9JU1J3+9BSRsVbHtJ0jmSnpH0tqRbJX3s72P55gTa9T0GrCBpI0mtwKHAr4vKzCNLsiuRtdZOlXRA2rZj+u9KETEgIsal91sDLwCrAZcVVhYR/5nKDgA2AmYDtxUHJqkPWQvxFrIW4+3AQQXbNwOuB74CfAL4JTBWUt8S33NX4JWIeLzM3+JKYEVgXWCn9J2/XLB9a2AaMAj4T+A6SYqI44DfAG3f6y9l9oGkDYHTgC0jYiDweeClEuU2AMYAZwKrAvcAf0x/lzZfAvYE1gE2AY4rt2/LFyfQ7qGtFbo78CzwauHGiHgwIv4REUsi4hmyH/VOFep8LSKujIhFETG/VAFJ/cgS5OURcU+JItsAvYGfRcTCiPg98ETB9pOAX0bE+IhYHBE3AR+kzxX7BDCzvWAL/vE4PyLejYiXgJ8ARxcUmxER10bEYuAmYDCwent1lrEY6AtsLKl3RLwUEdNLlDsU+FNE3BcRC4Efk3WvfLagzBUR8VpEzAX+CIysIR5rEifQ7uEW4Aiy1svNxRslbS3pr+nU9m3gFLJWWDkvV7Hf64BpEfHDdrZ/Eni1qF9xRsHrtYGz0+ntW5LeAoamzxV7gyzhtWcQ0Keo/hnAkIL3/257ERHvp5cDytRZUkQ8T9aqvBiYJel3kkrF/MnCeCJiCdnftWRMwPu1xGPN4wTaDUTEDLLBpL2BP5Qo8ltgLDA0IlYErgHU9vH2qi23T0nnARsCJ5QpNhMYIkkF69YqeP0ycFlErFSw9I+IMSXquh9YU9IW7exrDrCQLCkX7uvV0sUrmgf0L3i/RuHGiPhtRGyf9hdAqX9EXiuMJ/0dhnYgJssZJ9Du4wRgl4iYV2LbQGBuRCyQtBVZa7XNbGAJWb9hVdLgy+nAAe2d3ifjgEXA6ZJ6SfoisFXB9muBU1ILWZKWTwNeA4sriojngKuBMWlAp4+k5SQdJum8dFp+G3CZpIGS1gbOYun+4GpNAvaWtIqkNchanG3ff0NJu6S+2gXAfLLT+mK3AftI2lVSb+Bssi6Kv9cYk+WME2g3ERHTI2JCO5v/A7hU0rvARRQM+KRT2cuAR9NpdKn+x2KHkg2KTC0Yib+mREwfAl8k61p4M33uDwXbJ5D1g16Vtj9P+UGU01PZnwNvAdOBA8n6DgG+RtZyfAF4hKzlfX0V36eUW4CnyQaH/ge4tWBbX+AHZK3ef5MNtF1QXEFETAOOIhvcmgPsB+yX/i7WDcgTKpuZ1cYtUDOzGjmBmlmPIak13VRyd4ltfdPNDM9LGq8qboV2AjWznuQMYGo7204A3oyI9YGfUvrKio9xAjWzHkHSmmR34i01W1byBbIbLAB+D+xadAneUvIySURTqVe/UJ+lrpyxHBm10VqVC1lTzZjxEnPmzCmbcJZV6wprRywqd6VcJubPnkJ2SVmb0RExuqjYz4Bvkl3WV8oQ0g0kEbEo3XTyCbIrKEpyAgXUZyB9N/xSs8OwMh4df1WzQ7AKttu6vXscaheL5lf121ww6ecLIqLdACTtC8yKiCcl7dxesVIhlNuvT+HNLMcEaqm8VLYdsL+kl4DfAbsUzpqVvEJ2p1jbFI4rAnPLVeoEamb5JaCltfJSQUScHxFrRsQw4DDggYg4qqjYWLJpDQEOTmXKtkB9Cm9m+VZ+HKeDVetSYEJEjCWbHOcWSc+TtTwPq/R5J1AzyzFVe4petYh4EHgwvb6oYP0C4JBlqcsJ1MzyrYEt0I5yAjWz/JKq6uNsFidQM8u3Op/C15MTqJnlm0/hzcxqUf9BpHpyAjWz/BJugZqZ1UbQkt80ld/IzMwAWtwCNTNbdsJ9oGZmNXMfqJlZLTwKb2ZWO9+JZGZWA8mn8GZmNfMpvJlZjdwCNTOrhWdjMjOrja8DNTOrlS9jMjOrnftAzcxq5BaomVkN/EgPM7MO8Cm8mVlt5ARqZrbssgnpnUDNzJadhDyhsplZbfLcAs3v9QFmZmQJtNJSRR3LSXpc0tOSpki6pESZ4yTNljQpLSdWqtctUDPLtTq1QD8AdomI9yT1Bh6R9OeIeKyo3K0RcVq1lTqBmll+KS0dFBEBvJfe9k5LdLRen8KbWW4J0dLSUnEBBkmaULCcvFRdUqukScAs4L6IGF9ilwdJekbS7yUNrRSfW6BmlmtVnsLPiYgtyhWIiMXASEkrAXdKGhERkwuK/BEYExEfSDoFuAnYpVydboGaWa7VYxCpUES8BTwI7Fm0/o2I+CC9vRbYvFJdTqBmll+qcqlUjbRqankiqR+wG/BsUZnBBW/3B6ZWqten8GaWa3UahR8M3CSplazheFtE3C3pUmBCRIwFTpe0P7AImAscV6lSJ1Azy622QaSOiohngFEl1l9U8Pp84PxlqdcJ1MzyLb83IjmBmlmOKd+3cjqBmlmuOYGamdWgXn2gjeIEamb5lt8GqK8D7Y5aWsS4Medyx+WnNDsUK+ErJx7PWp9cjc1Hjmh2KPmn+l9IX09OoN3QaUd8jmkvvt7sMKwdRx97HP919383O4wuwwnUOs2Q1VZiz+0/zQ13/r3ZoVg7tt9hR1ZZZZVmh9Fl5DmBug+0m/nRNw7iW5ffxYD+yzU7FLO6yPMjPRrWApW0hqTfSZou6Z+S7pG0gaRhkiZXrqHm/R6SZpxeIqns7CzdzV47jGDW3Hd5aurLzQ7FrC6qaX12uxaosm90J3BTRByW1o0EVgca/eueDHwR+GWD95M7245cl313+gx7bv9p+vbpzQrLL8f13z2G4y+8udmhmdWsJ14H+jlgYURc07YiIiYBSBrWti69vgVYPq06LSL+nmZFuRVYIcV4akQ8LGkP4BKgLzAd+HJEtM0y3bafqanuRnyvXLvoyrFcdOVYAHbYfDhnHrOrk6d1eXn+LTfqFH4E8GQV5WYBu0fEZsChwBVp/RHAvRExEtgUmCRpEHAhsFsqPwE4q9YAJZ3cNnt1LJpfazVmy+yYow5n5x225V/TprHesDW58frrmh1SrqlFFZdmafYgUm/gqnR6vxjYIK1/Arg+PfzproiYJGknYGPg0fQvUh9gXK07jojRwGiAlv6rdfjZKHnz8JPP8fCTzzU7DCvh5l+PaXYIXUcPvRd+CnBwFeW+DrxO1spsARYARMRDknYE9gFukfQj4E2y55gc3piQzSxvBOQ4fzbsFP4BoK+kk9pWSNoytSILrQjMjIglwNFAayq7NjArIq4FrgM2Ax4DtpO0firTX9IGmFk3lu9R+IYk0PQI0QOB3dNlTFOAi4HXiopeDRwr6TGy0/d5af3OZP2eTwEHAZdHxGyyGaLHSHqGLKF+qnjfkg6U9AqwLfAnSffW+euZWSeSKi/N0rA+0Ih4DfhSO5tHpDLPAZsUrD8/rb+J7Il4xXU+AGxZYb93kl1CZWZdnbK5HfKq2YNIZmbtEk6gZmY1y/MgkhOomeVaT7yMycys45o8SFSJE6iZ5ZYf6WFm1gFugZqZ1ch9oGZmtch5H2h+OxfMrMdruw600lKxHmk5SY9LejpNuH5JiTJ9Jd0q6XlJ4wun3myPE6iZ5Vqd7oX/ANglIjYFRgJ7StqmqMwJwJsRsT7wU+CHlSp1AjWzXKvHvfCRaZt8vXdaiqex/AIf3UL+e2BXVcjOTqBmll91fC68pFZJk8gmcr8vIsYXFRlCeuRQRCwC3gY+Ua5OJ1Azy622+UCraIEOanvCRFpOLq4rIhanp1ysCWwlaUSJ3S31sXLxeRTezHKsukEiYE5EVPUU3oh4S9KDwJ5kD6Fs8wowFHhFUi+y+YrnlqvLLVAzy7V6nMJLWlXSSul1P2A34NmiYmOBY9Prg4EH0tzG7XIL1Mzyq37XgQ4GbpLUStZwvC0i7pZ0KTAhIsaSPf3iFknPk7U8D6tUqROomeVW1gfa8QwaEc8Ao0qsv6jg9QLgkGWp1wnUzHLNEyqbmdXI98KbmdUi5/fCO4GaWW6J5j62uBInUDPLtRznTydQM8u31q44iCRphXIfjIh36h+OmdlHpK47iDSF7D7Qwujb3gewVgPjMjMDIMcN0PYTaEQM7cxAzMxKyXMLtKp74SUdJumC9HpNSZs3Niwzs0w95gNtlIoJVNJVwOeAo9Oq94FrGhmUmRlk/YWtUsWlWaoZhf9sRGwm6SmAiJgrqU+D4zIzg2WYMLkZqkmgCyW1kCYWlfQJYElDozIzS3KcP6vqA/05cAewanqS3SNU8bAlM7OOEtAiVVyapWILNCJulvQk2QSkAIdExORynzEzq5fuMBtTK7CQ7DTes9ibWado9ih7JdWMwn8LGAN8kuxhTL+VdH6jAzMzgy5+Cg8cBWweEe8DSLoMeBL4fiMDMzOD0o/KzItqEuiMonK9gBcaE46Z2cd1ycuYJP2UrM/zfWCKpHvT+z3IRuLNzBpKUtecjYmPnpc8BfhTwfrHGheOmdnH5bgBWnYykes6MxAzs1K65Cl8G0nrAZcBGwPLta2PiA0aGJeZWbqQvtlRtK+aazpvBG4g+y57AbcBv2tgTGZm/yfPlzFVk0D7R8S9ABExPSIuJJudycysoaR8J9BqLmP6QFknxHRJpwCvAqs1Niwzs0yOu0CraoF+HRgAnA5sB5wEHN/IoMzM2ihNaVduqaKOoZL+KmmqpCmSzihRZmdJb0ualJaLKtVbzWQi49PLd/loUmUzs05RpxboIuDsiJgoaSDwpKT7IuKfReUejoh9q6203IX0d5LmAC0lIr5Y7U7MzGpRrwvpI2ImMDO9flfSVGAIUJxAl0m5FuhVHam4K1ln2GB+eMMFzQ7Dytj5x39rdghWwbTX321IvfW+DlTSMGAUML7E5m0lPQ28BpwTEVPK1VXuQvr7OxCjmVldVDl/5iBJEwrej46I0cWFJA0gmyD+zIh4p2jzRGDtiHhP0t7AXcDwcjutdj5QM7NOJ6pugc6JiC3K1iX1Jkuev4mIPxRvL0yoEXGPpKslDYqIOe3V6QRqZrlWjzuR0qWY1wFTI+L/tVNmDeD1iAhJW5E1ft8oV2/VCVRS34j4YBliNjPrEIl6zca0HdlVRP+QNCmtuwBYCyAirgEOBk6VtAiYDxwWEe0OpEN198JvRZa5VwTWkrQpcGJEfK3Wb2JmVq165M+IeIQKczNHxFUs4+B5Nf2zVwD7kpqyEfE0vpXTzDpJ23ORyi3NUs0pfEtEzCjqyF3coHjMzP5P22ON86qaBPpyOo0PSa3A14B/NTYsM7NMa37zZ1UJ9FSy0/i1gNeBv6R1ZmYNpSbPtlRJNffCzwIO64RYzMyWkuP8WdUo/LWUuCc+Ik5uSERmZgXyPCN9Nafwfyl4vRxwIPByY8IxM/tIlx9EiohbC99LugW4r2ERmZm1EbRWeTN8M9RyK+c6wNr1DsTMrBSVv/69qarpA32Tj/pAW4C5wHmNDMrMDPL/VM6yCTTdgL8p2XOQAJZUujfUzKye8pxAy/YupGR5Z0QsTouTp5l1qno8E6lRqumefVzSZg2PxMysiNIgUqWlWco9E6lXRCwCtgdOkjQdmEfWLRER4aRqZg3XVS9jehzYDDigk2IxM/uYrjyIJICImN5JsZiZLSXHDdCyCXRVSWe1t7G9afHNzOpFiNYcZ9ByCbQVGECFWZzNzBpGXfcUfmZEXNppkZiZldBVB5HyG7WZ9QjZY42bHUX7yiXQXTstCjOzdnTJFmhEzO3MQMzMiomu/0gPM7PmEE29VbMSJ1Azy7X8pk8nUDPLsS4/I72ZWTPl+TrQHE+Wb2ZWeSq7avpIJQ2V9FdJUyVNkXRGiTKSdIWk5yU9U80sdG6Bmlluibq18hYBZ0fEREkDgScl3RcR/ywosxcwPC1bA79I/22XW6Bmlmv1aIFGxMyImJhevwtMBYYUFfsCcHNkHgNWkjS4XL1OoGaWa6piWab6pGHAKGB80aYhfPyR7a+wdJL9GJ/Cm1luSVQ7G9MgSRMK3o+OiNFL16cBwB3AmRHxTvHmEvWWfYyRE6iZ5VqVF9LPiYgtKtTTmyx5/iYi/lCiyCvA0IL3awKvlavTp/Bmlmv1OIVPTxi+DphaZi7jscAxaTR+G+DtiJhZrl63QM0s1+p0Hf12wNHAPyRNSusuANYCiIhrgHuAvYHngfeBL1eq1AnUzHIru4yp4xk0Ih6hQmM1Pbb9q8tSrxOomeWYfCunmVmtcpw/nUDNLL/qdQrfKE6gZpZfcgvUzKxm7gO1TjHn369y1bfP4K03ZiO1sNtBR7LPESc2Oywr0KdV/OLIkfTp1UKrxAPTZvOrR2Y0O6zcyuYDbXYU7XMC7UZaW3txzFnfYd2NPsP8ee9x7hF7ssnWOzJ0vQ2aHZolHy4OThvzNPMXLqG1RYw+aiTjXpjLlNfebXZouaUc94H6TqRuZOVVV2fdjT4DQL/lBzBkneHMnf3vJkdlxeYvXAJArxbRq0UV7rY2qfLSLG6BdlOzXnuZF6dNZviIUc0OxYq0CG48bnPWXLkfd0x8lSkz3fosp0e2QCWtIel3kqZL+qekeyRtIGmYpMkN3O8qku6T9Fz678qN2ldezX9/Hj8+5yS+fM4l9B8wsNnhWJElAcfc8CT7/3wcGw9egXUH9W92SLklRKsqL83SkASabty/E3gwItaLiI3J7jtdvRH7K3IecH9EDAfuT+97jEULF/KTc05ih70OZOtd9252OFbGex8sZuL/vsU2667S7FDyq4rT92aewjeqBfo5YGG6QR+AiJgUEQ8XFkqt0YclTUzLZ9P6wZIekjRJ0mRJO6T1e0gal8renub2K/YF4Kb0+ibggIZ8wxyKCH5xydkMWWd99jv6K80Ox0pYqV9vBvRtBaBvrxa2HLYyM954v8lR5Vu9J1Sup0b1gY4Anqyi3Cxg94hYIGk4MAbYAjgCuDciLpPUCvSXNAi4ENgtIuZJOhc4C7i0qM7V26agioiZklYrtWNJJwMnAwwaXHbS6S7j2UlP8NCf7mCt4RtxzqG7A3DEaeex2Q67NjkyazNoQB++ve+GtKZHUdz/7GwenT632WHllh9rXF5v4CpJI4HFQNv1Nk8A16cJUO+KiEmSdgI2Bh5NE6z2AcbVuuM0W/VogPU23rRbjINuNGorbn/q1WaHYWU8P3sex94wsdlhdCk5zp8NS6BTgIOrKPd14HVgU7LuhAUAEfGQpB2BfYBbJP0IeBO4LyIOr1Dn65IGp9bnYLJWrpl1UT1xFP4BoK+kk9pWSNoytSILrQjMjIglZJOdtqayawOzIuJaslmkNwMeA7aTtH4q019SqSvExwLHptfHAv9Vv69lZp2txw0ipYlJDwR2T5cxTQEuZunni1wNHCvpMbLT93lp/c7AJElPAQcBl0fEbOA4YIykZ8gS6qdK7P4Hab/PAbun92bWRfXEQSQi4jXgS+1sHpHKPAdsUrD+/LT+Jj4aSS+s8wFgywr7fQPwqIlZd5HfM/imDyKZmbVL8ii8mVnN8ps+nUDNLO9ynEGdQM0sx5Try5icQM0s13LcBeoEamb5JZxAzcxq5lN4M7Ma5bkF6kd6mFmu1eNOJEnXS5rV3mTuknaW9HaaQnOSpIuqic0tUDPLL4Hq0wS9EbgKuLlMmYcjYt9lqdQtUDPLrbZBpI5OJhIRDwF1n3jVCdTMcq0TJxPZVtLTkv4s6dPVfMCn8GaWb9VlyEGSJhS8H50mTa/WRGDtiHhP0t7AXcDwSh9yAjWzXKvyMqY5EbFFrfuIiHcKXt8j6WpJgyJiTrnPOYGaWa61dMJlTJLWAF6PiJC0FVn35huVPucEamb5VocEKmkM2UTtgyS9AnyH7JlspKcHHwycKmkRMB84LE0MX5YTqJnlVjZI1PEMWulZahFxFdllTsvECdTM8qvJzzyqxAnUzHLNCdTMrCaeD9TMrGZugZqZ1aDZjy2uxAnUzPItxxnUCdTMcs2PNTYzq1F+06cTqJnlma8DNTPriPxmUCdQM8stP5XTzKwDOmM2plo5gZpZrvlOJDOzWuU3fzqBmlm+5Th/OoGaWX5JvpDezKx2+c2fTqBmlm85zp9OoGaWbzk+g3cCNbM884TKZmY18Z1IZmYd4ARqZlYjn8KbmdXC09mZmdXGfaBmZh3gU3gzsxrluQXa0uwAzMzKURVLxTqk6yXNkjS5ne2SdIWk5yU9I2mzamJzAjWzfKtHBoUbgT3LbN8LGJ6Wk4FfVFOpE6iZ5ZbIZmOqtFQSEQ8Bc8sU+QJwc2QeA1aSNLhSve4DBV6Y+sycQ0YNmdHsOOpsEDCn2UFYWd3tGK1d7wonTnzy3n69NaiKostJmlDwfnREjF6GXQ0BXi54/0paN7Pch5xAgYhYtdkx1JukCRGxRbPjsPb5GFUWEeVOu+upVDM2Kn3Ip/BmZlmLc2jB+zWB1yp9yAnUzAzGAsek0fhtgLcjouzpO/gUvjtblv4faw4fo04iaQywMzBI0ivAd4DeABFxDXAPsDfwPPA+8OWq6o2oeJpvZmYl+BTezKxGTqBmOSTl+QZGa+MEapZPTqBdgPtAexhJo4A3IuJ/mx2LlSZpT+Bo4CngxYi4I61X+AebK26B9iCSPg88CXxf0rDmRmOlpEtorgTuA94AvivpPAAnz/xxAu0hJC0PbEU2UcKbwLck1f3WO+uwAcADEXFjRNwA7At8TdK5TY7LSnAC7SEiYh4wJiJ+BZxBdg3ctySt29zIrMg8YNX0Dx4RMR3YEThK0n5NjcyW4gTag0TE86kfbTFwItAHOE9SH0mHp1N8a6KIGAe8B9xdsG462fRqQ5oVl5XmBNrDRERIao2IRWRJdD4wDvg+4IGlJpLUAhARxwDvSvqrpJXT5lWAzZsWnJXkUfhuqm3EtnDktuh1a0QslnQK8D1gh4iY0syYe5p2jlHviFiYXl8PDAQWASOBg32M8sX3wndDRZe7rEQ2aEThjzUlz5WBNYDP+YfZucoco4WS+kTEhxFxvKTPACsAr0XEi82K10pzC7QbS63LzwP/AP4dEVeXKNMnIj7s9OAMaP8YSWqJiCVNDc4qch9oNyXpKOBw4JvAlsDGRdvb+tucPJuk3DFy8uwanEC7iRL3Tvcju1xpe6AVODOVGw7+gTbDsh4jyz8n0G6gaBCi7VKX+WR3sxwVEXtExKJ0uniwpN7NirWn8jHqnjyI1A0U/DDPBraSdBrZdYTbAn0krUH2SNdTgSPaRnmt8/gYdU8eROomJB1PNov2ARHxhqQVgRFkp4e7kbV2LoiIyU0Ms0fzMep+nEC7idSyWQhMJrvnfbf0+gJgCbDEA0bN5WPU/bgPtAtqZ7LdycA2wCVkz7e+hux+94ERscA/zM7lY9QzuA+0iykajPgqsDLQOyK+I+kvwPIR8Y6k/YHP4n8kO52PUc/hA9dFSToD+BJwO/AVSXdExOL0wzyW7N72o6p5NKs1ho9R9+cE2kVI2k7S7ul2zDXIJpY4ANgPeBgYLOneVPwZYB/fntm5fIx6Hg8idRGSjgS+C5wYEfdLWoVsBPeHEbFtmhz5ReCmiKjqmdZWXz5GPY/7QHNO0uZkd6ncA3wI/EzSmekHugCYLGkgsAXwbeC25kXbM/kY9VxOoDkmaR+yfrKfAHMj4nZJfUk/UGAi0B/4Jdno7u5p8l3rJD5GPZtP4XNK0k7AdcCRETG+aNtRwLnAsWSz+GwEvBMRL3V2nD2Zj5G5BZpfmwNXRsR4Sb3SfdJtc3n+WtKHwFiyUdwHmxtqj+Vj1MM5geZMwTWE6wBvp9WL4WP3U28K/DcQ+DEcnc7HyNr4MqacKZil/E5gG0mbp8tiWtrm8CS7BXCdiLg9Il5oTqQ9l4+RtXECza/xwCPAoekHuiQilkg6FDgMmNvc8Awfox7Pg0g5luaNPAHYFXgCWAAcTPZwMc/YkwM+Rj2bE2jOSepHNlixGzAT+GtE/Ku5UVkhH6OeywnUzKxG7gM1M6uRE6iZWY2cQM3MauQEamZWIydQM7MaOYGamdXICdRKkrRY0iRJkyXdLql/B+raWdLd6fX+ks4rU3YlSf9Rwz4ulnROteuLytwo6eBl2NcwSb5I3pxArV3zI2JkRIwgmyT4lMKNyizz/z8RMTYiflCmyErAMidQs2ZwArVqPAysn1peUyVdTTZR8FBJe0gaJ2liaqkOAJC0p6RnJT0CfLGtIknHSboqvV5d0p2Snk7LZ4EfAOul1u+PUrlvSHpC0jOSLimo61uSpqUnXW5Y6UtIOinV87SkO4pa1btJeljSvyTtm8q3SvpRwb6/0tE/pHUvTqDKlqMvAAACPklEQVRWlqRewF5kkwJDlqhujohRwDzgQmC3iNgMmACcJWk54Fqyh6ntAKzRTvVXAH+LiE2BzYApwHnA9NT6/YakPYDhwFbASGBzSTumx2gcBowiS9BbVvF1/hARW6b9TSW7h73NMGAnYB/gmvQdTgDejogtU/0nSVqniv1YD+H5QK09/SRNSq8fJpt5/ZPAjIh4LK3fBtgYeFQSQB9gHPAp4MWIeA5A0q+Bk0vsYxfgGICIWAy8LWnlojJ7pOWp9H4AWUIdCNwZEe+nfYyt4juNkPRdsm6CAcC9Bdtui4glwHOSXkjfYQ9gk4L+0RXTvn2fuwFOoNa++RExsnBFSpLzClcB90XE4UXlRpJNJFwPAr4fEb8s2seZNezjRuCAiHha0nHAzgXbiuuKtO+vRURhokXSsGXcr3VTPoW3jngM2E7S+gCS+kvaAHgWWEfSeqnc4e18/n7g1PTZVkkrAO+StS7b3AscX9C3OkTSasBDwIGS+il74uV+VcQ7EJgpqTdwZNG2Q9KEyOsB6wLT0r5PTeWRtIGk5avYj/UQboFazSJidmrJjVH2JEqACyPiX5JOBv4kaQ7ZpMMjSlRxBjBa0glkj8Q4NSLGSXo0XSb059QPuhEwLrWA3yN7xtBESbcCk4AZZN0MlXybbBLkGWR9uoWJehrwN2B14JSIWCDpV2R9oxOV7Xw2cEB1fx3rCTydnZlZjXwKb2ZWIydQM7MaOYGamdXICdTMrEZOoGZmNXICNTOrkROomVmN/j8uyboegQJcugAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import matplotlib.pyplot as plt\n",
    "import itertools\n",
    "%matplotlib inline\n",
    "\n",
    "def plot_confusion_matrix(cm, classes, title, cmap=plt.cm.Blues):\n",
    "    \"\"\"\n",
    "    This function prints and plots the confusion matrix.\n",
    "    \"\"\"\n",
    "    plt.imshow(cm, interpolation='nearest', cmap=cmap)\n",
    "    plt.title(title)\n",
    "    plt.colorbar()\n",
    "    tick_marks = np.arange(len(classes))\n",
    "    plt.xticks(tick_marks, classes, rotation=45)\n",
    "    plt.yticks(tick_marks, classes)\n",
    "\n",
    "    thresh = cm.max() / 2.\n",
    "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
    "        plt.text(j, i, format(cm[i, j], 'd'), horizontalalignment=\"center\",\n",
    "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
    "\n",
    "    plt.tight_layout()\n",
    "    plt.ylabel('True label')\n",
    "    plt.xlabel('Predicted label')\n",
    "\n",
    "plt.figure()\n",
    "matriz_confusion = confusion_matrix(y_true=y_real, y_pred=y_predict, labels=[1, 0])\n",
    "plot_confusion_matrix(matriz_confusion, classes=['Clase 1', 'Clase 0'], title='Matriz de Confusión')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "\n",
    "# Ejemplo: Clasificación Múltiple\n",
    "\n",
    "\n",
    "* Para el caso de la clasificación múltiple las métricas son las mismas, pero el procedimiento para calcularlo es diferente.\n",
    "\n",
    "\n",
    "* Todas estas métricas se calculan para cada clase, considerando una \"Clase A\" como la clase a clasificar correctamente y el resto de clases se consideran como una sola clase.\n",
    "\n",
    "\n",
    "* Veamos el siguiente ejemplo en el que tenemos:\n",
    "\n",
    "    - y_real =    [1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 2, 2, 2, 2, 2]\n",
    "    - y_predict = [1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 2, 1, 1, 0, 2]\n",
    "    \n",
    "    \n",
    "* Tendríamos que calcular las metricas para las clases 0, 1 y 2 considerando:\n",
    "<span></span><br><br>\n",
    "    - ***Clase 0***: Consideramos la Clase 0 como la que es y el resto de clases como una misma Clase X\n",
    "        + y_real =    [X, X, X, X, X, 0, 0, 0, 0, 0, X, X, X, X, X]\n",
    "        + y_predict = [X, X, 0, X, X, 0, 0, X, X, 0, X, X, X, 0, X]\n",
    "<span></span><br><br>        \n",
    "    - ***Clase 1***: Consideramos la Clase 1 como la que es y el resto de clases como una misma Clase X\n",
    "        + y_real =    [1, 1, 1, 1, 1, X, X, X, X, X, X, X, X, X, X]\n",
    "        + y_predict = [1, 1, X, 1, 1, X, X, 1, 1, X, X, 1, 1, X, X]\n",
    "<span></span><br><br>        \n",
    "    - ***Clase 2***: Consideramos la Clase 2 como la que es y el resto de clases como una misma Clase X\n",
    "        + y_real =    [X, X, X, X, X, X, X, X, X, X, 2, 2, 2, 2, 2]\n",
    "        + y_predict = [X, X, X, X, X, X, X, X, X, X, 2, X, X, X, 2]\n",
    "\n",
    "\n",
    "* Veamos a continuación un ejemplo, en el que usaremos la función \"classification_report()\" que nos devolverá las métricas vistas anteriormente por clase:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "           0       0.60      0.60      0.60         5\n",
      "           1       0.50      0.80      0.62         5\n",
      "           2       1.00      0.40      0.57         5\n",
      "\n",
      "   micro avg       0.60      0.60      0.60        15\n",
      "   macro avg       0.70      0.60      0.60        15\n",
      "weighted avg       0.70      0.60      0.60        15\n",
      "\n"
     ]
    }
   ],
   "source": [
    "y_real =    [1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 2, 2, 2, 2, 2]\n",
    "y_predict = [1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 2, 1, 1, 0, 2]\n",
    "\n",
    "from sklearn.metrics import classification_report\n",
    "\n",
    "print(classification_report(y_true=y_real, y_pred=y_predict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "* Veamos a continuación la matriz de confusión:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAVAAAAEmCAYAAAA0k8gFAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDMuMC4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvOIA7rQAAIABJREFUeJzt3XmcHFW5//HPdyYLewgmLFkkBBI1RAhLEAgIshkIRESQwEWIbBcUIcpVARGBn+tFL4LIRRAuEJUlIhoQjMgii2EJISAhIIkhlywSQlgDIcnk+f1RNZem6enu9HRP1WS+b1/1orvqzKlnzst5cupU1TmKCMzMbM01ZR2AmVln5QRqZlYjJ1Azsxo5gZqZ1cgJ1MysRk6gZmY1cgK1qkj6N0l/rkM910r6bj1iqidJm0m6X9Kbkn7SjnrOkfTLEvtHSXpUUu/2RWp54gTaiUl6QdIKSX2K9s+QFJIGVVHHoLRst3LlIuLXEXFA+yJuHyVOl/S0pGWS5kuaJOnjdaj+ZGAJsFFEnFlrJRHx/Yg4sXCfpIHA94GDI+LV9oVpeeIE2vnNBY5q/ZImk3XreYJKybUDXQKcAZwObAIMBX4PjKlD3VsCz0QD3iyJiBcjYq+IWFzvui1bTqCd30Tg2ILvxwHXFxaQNEbSE5LekPSipPMLDt+f/vc1SW9J2k3SeEkPSbpY0lLg/HTfg2l930jLtm4rJV1bKjhJO0ianl4a3wSsU3T84LTH/Jqkv0naro16hgBfBo6KiHsi4t2IeDvtGf8wLdNL0vWSXpY0T9K5kprSY+MlPSjpx5JelTRX0oHpsWvTdmv9vfYrHmqQtLek+QXfvylpQfp7PSdp33T/+ZJ+VVBurKSZ6e93n6SPFRx7QdJ/SHpK0uuSbpL0vvaxfHMC7fweBjaS9DFJzcCRwK+KyiwjSbIbk/TWTpV0aHrsk+l/N46IDSJiavr9E8A/gU2B7xVWFhH/mZbdAPgY8DJwc3FgknqQ9BAnkvQYJwGfKzi+I3AN8O/Ah4BfAJMl9Szxe+4LzI+IR8u0xc+AXsBgYK/0d/5iwfFPAM8BfYD/BK6WpIgYD/waaP29/lLmHEj6CHAaMDIiNgQ+DbxQotxQ4AZgAtAXuAO4LW2XVp8HRgNbAdsB48ud2/LFCXTt0NoL3R94FlhQeDAi7ouIv0fE6oh4iuSPeq8KdS6MiJ9FxKqIeKdUAUnrkiTISyLijhJFdgW6Az+NiJUR8VvgsYLjJwG/iIhHIqIlIq4D3k1/rtiHgEVtBVvwj8fZEfFmRLwA/AT4QkGxeRFxVUS0ANcBWwCbtVVnGS1AT2CYpO4R8UJEzClR7kjgjxFxV0SsBH5MMryye0GZSyNiYUQsBW4DRtQQj2XECXTtMBE4mqT3cn3xQUmfkHRvemn7OnAKSS+snBerOO/VwHMR8aM2jvcDFhSNK84r+LwlcGZ6efuapNeAgenPFXuFJOG1pQ/Qo6j+eUD/gu//av0QEW+nHzcoU2dJETGbpFd5PrBY0o2SSsXcrzCeiFhN0q4lYwLeriUey44T6FogIuaR3Ew6CPhdiSK/ASYDAyOiF3AFoNYfb6vacueUdBbwEeCEMsUWAf0lqWDfhws+vwh8LyI2LtjWi4gbStR1NzBA0s5tnGsJsJIkKReea0Hp4hUtA9Yr+L554cGI+E1E7JGeL4BS/4gsLIwnbYeB7YjJcsYJdO1xArBPRCwrcWxDYGlELJe0C0lvtdXLwGqSccOqpDdfTgcObevyPjUVWAWcLqmbpMOAXQqOXwWckvaQJWn99IbXhsUVRcTzwOXADekNnR6S1pE0TtJZ6WX5zcD3JG0oaUvga3xwPLhaM4CDJG0iaXOSHmfr7/8RSfukY7XLgXdILuuL3QyMkbSvpO7AmSRDFH+rMSbLGSfQtUREzImIaW0c/hJwoaQ3gfMouOGTXsp+D3govYwuNf5Y7EiSmyKzCu7EX1EiphXAYSRDC6+mP/e7guPTSMZBL0uPz6b8TZTT07I/B14D5gCfJRk7BPgKSc/xn8CDJD3va6r4fUqZCDxJcnPoz8BNBcd6Aj8k6fX+i+RG2znFFUTEc8AxJDe3lgCHAIek7WJrAXlCZTOz2rgHamZWIydQM+syJDWnL5XcXuJYz/RlhtmSHlEVr0I7gZpZV3IGMKuNYycAr0bENsDFlH6y4n2cQM2sS5A0gORNvA/MlpX6DMkLFgC/BfYtegTvA/IySUSmmtftFd022jTrMHKn14al3qi0gb38unop8+a9wJIlS8omnDXVvNGWEavKPSmXiHdenknySFmrKyPiyqJiPwW+QfJYXyn9SV8giYhV6UsnHyJ5gqIkJ1Cg20ab0u/on2YdRu4ctOdWWYeQSz8ZOyzrEHJp1CfaesehdrHqHXp+5PMVyy2f8fPlEdFmAJIOBhZHxOOS9m6rWKkQyp3Xl/BmlmMCNVXeKhsFjJX0AnAjsE/hrFmp+SRvirVO4dgLWFquUidQM8svAU3NlbcKIuLsiBgQEYOAccA9EXFMUbHJJNMaAhyelinbA/UlvJnlW/n7OO2sWhcC0yJiMsnkOBMlzSbpeY6r9PNOoGaWY6r2Er1qEXEfcF/6+byC/cuBI9akLidQM8u3BvZA28sJ1MzyS6pqjDMrTqBmlm91voSvJydQM8s3X8KbmdWi/jeR6skJ1MzyS7gHamZWG0FTftNUfiMzMwNocg/UzGzNCY+BmpnVzGOgZma18F14M7Pa+U0kM7MaSL6ENzOrmS/hzcxq5B6omVktPBuTmVltcv4caH4jMzOr06JyktaR9KikJyXNlHRBiTLjJb0saUa6nVipXvdAzSzf6jMG+i6wT0S8Jak78KCkOyPi4aJyN0XEadVW6gRqZvlWh0v4dHXNt9Kv3dOt7Iqb1fAlvJnlV+uSHu1c1jipSs2SZgCLgbsi4pESxT4n6SlJv5U0sFKdTqBmlm+tD9OX26CPpGkF28nF1URES0SMAAYAu0gaXlTkNmBQRGwH/AW4rlJovoTPiR7dmrj5tN3o0a2J5mZx55OL+Omfns86rMy9uWQRd196Nm+/9gqSGLb/EWx/8BeyDisX/jzlT/zH186gpaWF8cefyNe/cVbWITWEqhsDXRIRO1dTMCJek3QfMBp4umD/KwXFrgJ+VKkuJ9CcWLFqNUdf/jBvr2ihW5OYdPpu3DfrZWbMey3r0DLV1NyNUeO/Qd/Bw1jxzjImff0IBm6/G5sM3Cbr0DLV0tLChNO/zB/vvIv+Awawx64jOfjgsXxs2LCsQ6urZEL69t9EktQXWJkmz3WB/ShKkJK2iIhF6dexwKxK9foSPkfeXtECQLdm0a25qQ5D3J3f+r370ndwkhR6rLs+vQcMZtnSxRlHlb3HHn2Urbfehq0GD6ZHjx4cceQ4br/tD1mHVX8Saqq8VWEL4F5JTwGPkYyB3i7pQklj0zKnp484PQmcDoyvVKl7oDnSJLjtzD3Yss/6THxwHjP+t2v3Pou9sXgBS+bOYrMh22UdSuYWLlzAgAHv3ePo338Ajz5a6p5I51ePHmhEPAXsUGL/eQWfzwbOXpN6G9YDlbS5pBslzZH0jKQ7JA2VNEjS05VrqPm8m0i6S9Lz6X97N+pc9bY6YMyPH2S38+9m+w9vzNDNN8g6pNxY+c4yplw0gVFfPIse67ldkqdy3q8eiSaPJFXcstKQBKrkN7oVuC8ito6IYcA5wGaNOF+Rs4C7I2IIcHf6vVN5c/kqHp7zCnt9dNOsQ8mFllUr+dNFExiy5xi23nX/rMPJhf79BzB//ov/933Bgvn069cvw4gap8slUOBTJAO2V7TuiIgZEfFAYaG0N/qApOnptnu6fwtJ96evUz0tac90/wGSpqZlJ0kq1RX5DO89fnAdcGhDfsM622T9Hmy4TjKi0rN7E3sM7cOcxW9V+Km1X0Rw7+Xn0XvAYEaMHZ91OLmx88iRzJ79PC/MncuKFSuYdNONjDl4bOUf7GxU5ZaRRo2BDgcer6LcYmD/iFguaQhwA7AzcDQwJSK+J6kZWE9SH+BcYL+IWCbpm8DXgAuL6tys9U5aRCySVLIblz4ndjJA84Z91/w3rLNNN+rJj4/enuam5F/UP85YyD3P+GbJv56dzj/+OplNPjyUm848DIBdj57Aljt9MuPIstWtWzcuvuQyDhnzaVpaWjhu/PEM23bbrMOqOyGamvJ7rzvrm0jdgcskjQBagKHp/seAa9J3Vn8fETMk7QUMAx5Ku+w9gKm1njgirgSuBOi52ZDM73c/u+hNDv7Jg1mHkTtbfGwnvnTLzKzDyKXRBx7E6AMPyjqMhsvz2G6jEuhM4PAqyn0VeAnYnmQ4YTlARNwv6ZPAGGCipIuAV0kePTiqQp0vtT7PJWkLkl6umXVSeU6gjeob3wP0lHRS6w5JI9NeZKFewKKIWA18AWhOy24JLI6Iq4CrgR2Bh4FRkrZJy6wnaSgfNBk4Lv18HLAWPhxn1kXkfAy0IQk0nfnks8D+6WNMM4HzgYVFRS8HjpP0MMnl+7J0/97ADElPAJ8DLomIl0kebL0hfRj2YeCjJU7/w/S8zwP7p9/NrJPK8134ho2BRsRC4PNtHB6elnkeKHwq+ux0/3WUeJE/Iu4BRlY47yvAvjWEbGY545tIZmbtkd8hUCdQM8sx5fsmkhOomeWaE6iZWQ08Bmpm1h757YA6gZpZjnkM1Mysdk6gZmY1cgI1M6tRlUt2ZCK/t7fMrMur5jXOanqoktaR9KikJ9N1jy4oUaanpJskzZb0iKRBlep1AjWzXKvTu/DvAvtExPbACGC0pF2LypwAvBoR2wAXU8Wyxk6gZpZr9UigkWhd4qF7uhXPA1y4msVvgX1VoXInUDPLtSqXNe4jaVrBdvIH6pGaJc0gmSP4rogoXsa0P/AiQESsAl4HPlQuNt9EMrP8qv450CURsXO5AhHRAoyQtDFwq6ThEVG4QnCpE5VdrcI9UDPLLQFS5W1NRMRrwH3A6KJD84GBAJK6kUz4vrRcXU6gZpZjdbsL3zfteSJpXWA/4NmiYoWrWRwO3JNODt8mX8KbWa7V6Tn6LYDr0lV+m4CbI+J2SRcC0yJiMsnyQRMlzSbpeY6rVKkTqJnll6CpDg/SR8RTwA4l9p9X8Hk5cMSa1OsEama5JeqTQBvFCdTMci3Hr8I7gZpZvnkyETOzWtTwmFJHcgI1s9zykh5mZu3gHqiZWY08BmpmVguPgZqZ1cbPgZqZtYMv4c3MapTj/OkECrDVphtwzZd2zzqM3Nn389/OOoRc+snYy7IOoevwuvBmZrVpnQ80r5xAzSzH5JtIZma18iW8mVkt/ByomVltkjHQ/GbQ/L6lb2ZG8iB9pa0SSQMl3StplqSZks4oUWZvSa9LmpFu55Wqq5B7oGaWa3Xqga4CzoyI6ZI2BB6XdFdEPFNU7oGIOLjaSt0DNbP8qmJJ42rya0Qsiojp6ec3gVlA//aG5wRqZrml6pc17iNpWsF2cpt1SoNIFph7pMTh3SQ9KelOSdtWis+X8GaWa1VewS+JiJ0r16UNgFuACRHxRtHh6cCWEfGWpIOA3wNDytXnHqiZ5Vpzkypu1ZDUnSR5/joifld8PCLeiIi30s93AN0l9SlXZ5s9UEkblfvBEtnbzKyuVKd34ZVUcjUwKyL+q40ymwMvRURI2oWkg/lKuXrLXcLPBILkUaxWrd8D+HD14ZuZ1aZOb3KOAr4A/F3SjHTfOaR5LCKuAA4HTpW0CngHGBcRUa7SNhNoRAysR9RmZu1Rjx5oRDzI+zuDpcpcBqzRVFtVjYFKGifpnPTzAEk7rclJzMxqVY/HmBqlYgKVdBnwKZLuL8DbwBWNDMrMDJIuY7NUcctKNY8x7R4RO0p6AiAilkrq0eC4zMzgvec8c6maBLpSUhPJjSMkfQhY3dCozMxSOc6fVY2B/pzk2am+ki4AHgR+1NCozMxIV+WUKm5ZqdgDjYjrJT0O7JfuOiIinm5sWGZmibVhRvpmYCXJZbzfXjKzDpH1XfZKqrkL/y3gBqAfMAD4jaSzGx2YmRl08kt44Bhgp4h4G0DS94DHgR80MjAzM6jw9HvGqkmg84rKdQP+2ZhwzMzer1M+xiTpYpIxz7eBmZKmpN8PILkTb2bWUFL1sy1loVwPtPVO+0zgjwX7H25cOGZm75fjDmjZyUSu7shAzMxKyfMlfDV34beWdKOkpyT9o3XriOC6ku+ffRpjdh3KMWN2zzqU3GlqElNv+Ca3XHJK1qHkyp+n/Inttv0I2350Gy76zx9mHU5DJA/SV96yUs0zndcC/0PyuxwI3Azc2MCYuqSDDjua/7p6UtZh5NJpR3+K5+a+lHUYudLS0sKE07/MH267kyeeeoZJN97ArGeKF5hcO+T5MaZqEuh6ETEFICLmRMS5JLMzWR2NGLk7G/XqnXUYudN/040Zvce2/M+tf8s6lFx57NFH2Xrrbdhq8GB69OjBEUeO4/bb/pB1WHUndf4E+m46Hf4cSadIOgTYtMFxmQFw0dc/x7cu+T2rV5edGLzLWbhwAQMGvDfnef/+A1iwYEGGETVOp54PFPgqsAFwOsm0+CcBx1f6IUmbp2OncyQ9I+kOSUMlDZLUsHfpJR0haaak1ZIqrtJn+XXgnsNZvPRNnpj1Ytah5E6plSbyfLOlPapc1rhSHQMl3StpVpofzihRRpIulTQ7veezY6V6q5lMpHXt5Dd5b1LlSsEKuBW4LiLGpftGAJsBjf5reBo4DPhFg89jDbbbiMEcvNfHGb3HtvTs0Z2N1l+Ha757LMefe33WoWWuf/8BzJ//3p/SggXz6devX4YRNU6d/l1YBZwZEdMlbQg8LumuiCgcOD6QZBnjIcAngP9O/9umcg/S30o6B2gpEXFYmXo/BaxMF2pqLT8jrXdQwTkGAROB9dNdp0XE3yRtAdwEbJTGeGpEPCDpAOACoCcwB/hi6zKkBeeZldZdJjzrDM772WTO+9lkAPbcaQgTjt3XyTO188iRzJ79PC/MnUu//v2ZdNONXDvxN1mHVXf1epA+IhYBi9LPb0qaBfQHChPoZ4Dr04XkHpa0saQt0p8tqVwPdI0WVyoynOR9+UoWA/tHxHJJQ0gmLdkZOBqYEhHfk9QMrJeuz3wusF9ELJP0TeBrwIXtiDM3vvPVE3ni0Yd47dVXOHTPbTnh9LM45IiqOvzWBXXr1o2LL7mMQ8Z8mpaWFo4bfzzDtt0267Aaot6dobTjtgPwSNGh/rz/Cnl+um/NE2hE3F1zhNXrDlyWXt63AEPT/Y8B10jqDvw+ImZI2gsYBjyUNmgPYGqtJ5Z0MnAywGb9BtT+G9TJBRf/MusQcu2Bx5/ngcefzzqMXBl94EGMPvCgrMNouCrnz+wjaVrB9ysj4sriQpI2IJkgfkJEvFF8uES9tS1r3E4zSdZYruSrwEvA9iTttBwgIu6X9ElgDDBR0kXAq8BdEXFUPQJMG/dKgI9+fAff4jXLIVF1D3RJRJS9aZx2yG4Bfh0RvytRZD5QuJz7AGBhuTobNTnyPUBPSSe17pA0Mu1FFuoFLIqI1SQ3qJrTslsCiyPiKuBqYEeSd/BHSdomLbOepKGY2VqtHm8ipTe2rwZmRcR/tVFsMnBsejd+V+D1cuOfsAY9UEk9I+LdaspGREj6LPBTSWeR9CxfACYUFb0cuEXSEcC9wLJ0/97A1yWtBN4Cjo2IlyWNB26Q1DMtdy7wvtdK0/P+DOgL/FHSjIj4dLW/p5nlh0S9ZmMaRdJJ+7ukGem+c4APA6Q3vO8ADgJmk8xC98VKlVZMoJJ2IcncvYAPS9oeODEivlLu5yJiIfD5Ng4PT8s8D2xXsP/sdP91wHUl6rwHGFnhvLeSPEJlZmuBeuTPiHiQCnMzp3ffv7wm9VZzCX8pcDDwSnqSJ/GrnGbWQfL8JlI1l/BNETGvaCC3pUHxmJn9n9ZljfOqmgT6YnoZH+kzmV+haNzRzKxRmvObP6tKoKeSXMZ/mOSRo7+k+8zMGkoZz7ZUSTXvwi8GxnVALGZmH5Dj/FnVXfirKPE0fkSc3JCIzMwK5HhNuaou4f9S8Hkd4LM0fkYlM7POfxMpIm4q/C5pInBXwyIyM2slaG7U+5J1UMu78FsBW9Y7EDOzUlT++fdMVTMG+irvjYE2AUuBsxoZlJkZvLcqZ16VTaDpC/jbA62LrayOUmsJmJk1SJ4TaNnRhTRZ3hoRLenm5GlmHaoeayI1SjXDs49Ws7iSmVm9Kb2JVGnLSrk1kbpFxCpgD+AkSXNIppsTSefUSdXMGq6zPsb0KMlExod2UCxmZu/TmW8iCSAi5nRQLGZmH5DjDmjZBNpX0tfaOlhmWnwzs7oQojnHGbRcAm0GNqDCLM5mZg1T5ZpHWSmXQBdFxFqx5rqZdV71uIkk6RqSlTUWR8TwEsf3Bv4AzE13/a6a/FdxDNTMLCvJssZ1qepa4DLg+jJlHoiIg9ek0nIJdN81qcjMrBHq0QONiPslDWp3RUXafAQ1IpbW+2RmZmtCJEt6VNqAPpKmFWy1zFe8m6QnJd0padtqfqCW2ZjMzDqGqPZVzSURsXM7zjQd2DIi3pJ0EPB7YEilH8rxTHtmZuk4aIWtvSLijYh4K/18B9BdUp9KP+ceqJnlVkfNSC9pc+CliIh0FeIm4JVKP+cEama5Vo/nQCXdAOxNMlY6H/gO0B0gIq4ADgdOlbQKeAcYV83sc06gZpZj9ZmuLiKOqnD8MpLHnNaIE6iZ5ZbI940aJ1Azy7UsJ0yuxAnUzHItv+nTCdTKOP7bX8o6hFw6c/IzWYeQSy++vrzudUp02tmYzMwy50t4M7Ma5Td9OoGaWc7luAPqBGpm+ZU8xpTfDOoEamY5pk67KqeZWeZynD+dQM0sv3wJb2ZWK7kHamZWM4+BmpnVIJkPNOso2uYEama5Jo+BmpnVJsdX8E6gZpZvee6B5nmuUjPr4oRoVuWtYj3SNZIWS3q6jeOSdKmk2ZKekrRjNfE5gZpZfqWPMVXaqnAtMLrM8QNJljEeApwM/Hc1lTqBmlmu1WNZ44i4H1hapshngOsj8TCwsaQtKtXrMVAzy601WNa4j6RpBd+vjIgr1+BU/YEXC77PT/ctKvdDTqBmlmtVXqIviYid23OaEvu8rLGZdW4ddBd+PjCw4PsAYGGlH/IYqJnlWp1uIlUyGTg2vRu/K/B6RJS9fAf3QM0s5+qRHyXdAOxNMlY6H/gO0B0gIq4A7gAOAmYDbwNfrKZeJ1Azy7c6ZNCIOKrC8QC+vKb1OoGaWW5Jno3JzKxm+U2fTqBmlnc5zqBOoGaWY8r1ZCJOoDnx/bNP46F7/0zvD/XhV3/8W9bh5MabSxZx96Vn8/ZrryCJYfsfwfYHfyHrsDLXldolx0OgTqB5cdBhR/O5Y07i/33j1KxDyZWm5m6MGv8N+g4exop3ljHp60cwcPvd2GTgNlmHlqmu0i4i3wnUD9LnxIiRu7NRr95Zh5E76/fuS9/BwwDose769B4wmGVLF2ccVfa6Uruoiv9lxT1Q6zTeWLyAJXNnsdmQ7bIOJVfW9nbpkj1QSZtLulHSHEnPSLpD0lBJg9qa1LRO571I0rPppKi3Stq4UeeyjrPynWVMuWgCo754Fj3W2yDrcHKjK7RLPaaza5SGJFBJAm4F7ouIrSNiGHAOsFkjzlfkLmB4RGwH/AM4uwPOaQ3Usmolf7poAkP2HMPWu+6fdTi50SXaRSCp4paVRvVAPwWsTN8xBSAiZkTEA4WF0t7oA5Kmp9vu6f4tJN0vaYakpyXtme4/QNLUtOwkSR/4Jzci/hwRq9KvD5PMqmKdVERw7+Xn0XvAYEaMHZ91OLnRVdql9SZSB0wmUpNGJdDhwONVlFsM7B8ROwJHApem+48GpkTECGB7YIakPsC5wH5p+WnA1yrUfzxwZw3xd7jvfPVE/v3IT/O/c2dz6J7bctukiVmHlAv/enY6//jrZOb//VFuOvMwbjrzMOY9fn/WYWWuK7VLni/hs76J1B24TNIIoAUYmu5/DLhGUnfg9xExQ9JewDDgobTL3gOY2lbFkr4FrAJ+3cbxk0nWPmGzftl3Ui+4+JdZh5BLW3xsJ750y8ysw8idLtUuOb6J1KgEOhM4vIpyXwVeIullNgHLIVm/RNIngTHAREkXAa8Cd1WaVQVA0nHAwcC+6SwrH5BO938lwEc/vkPFmafNLBt5fhOpUZfw9wA9JZ3UukPSyLQXWagXsCgiVgNfAJrTslsCiyPiKuBqYEeS8cxRkrZJy6wnaWhRfUgaDXwTGBsRb9f/VzOzjtSkyltmsTWi0rTX91lg//QxppnA+XxwivzLgeMkPUxy+b4s3b83ybjnE8DngEsi4mVgPHCDpKdIEupHS5z+MmBD4K70JtQVJcqYWWeR40HQho2BRsRC4PNtHB6elnkeKHz69+x0/3XAdSXqvAcYWeG8a9e7bGZdWJIfu94lvJlZ+1XxCFO1jzFJGi3pOUmzJZ1V4vh4SS+nV64zJJ1Yqc6s78KbmZVVj+c8JTUDPwf2J1mB8zFJkyPimaKiN0XEadXW6x6omeVYNVOJVJVhdwFmR8Q/I2IFcCPwmfZG5wRqZrlW5SV8H0nTCraTi6rpD7xY8H1+uq/Y59J5NH4raWCJ4+/jS3gzy601uMm+JCJ2rlBVseLnv28DboiIdyWdQnIje59yJ3UP1MzyrT6PMc0HCnuUAyh6rDIiXomId9OvVwE7VarUCdTMcq1JqrhV4TFgiKStJPUAxgGTCwtI2qLg61hgVqVKfQlvZrlWj6dAI2KVpNOAKSRvPF4TETMlXQhMi4jJwOmSxpLMobGU5MWdspxAzSy/6jhdXUTcAdxRtO+8gs9ns4bzBzuBmlnO5fdNJCdQM8utvK/K6QRqZrmW5WxLlTiBmlmu5XkyESdQM8u3/OZPJ1Azy7cc508nUDPLL4lqH5TPhBOomeVbfvOnE6iZ5VuO86cTqJnlW46v4J1AzSzPqp4wORNOoGaWW34TycysHZxAzcxq5Et4M7Na1HE6u0ZwAjWz3PIYqJlZO/gS3sysRnnugXpROTPLtfosygmSRkt6TtJsSWeVON5T0k3p8UdhU6jUAAAJi0lEQVQkDapUpxOomeVbHTKopGbg58CBwDDgKEnDioqdALwaEdsAFwM/qlSvE6iZ5Zao27LGuwCzI+KfEbECuBH4TFGZzwDXpZ9/C+wrla/cY6DAc0/PWDJq6Cbzso4j1QdYknUQOeR2KS1P7bJlvSucPv3xKet2V58qiq4jaVrB9ysj4sqC7/2BFwu+zwc+UVTH/5VJl0F+HfgQZdrXCRSIiL5Zx9BK0rSI2DnrOPLG7VLa2t4uETG6TlWV6klGDWXex5fwZtYVzAcGFnwfACxsq4ykbkAvYGm5Sp1AzawreAwYImkrST2AccDkojKTgePSz4cD90RE2R6oL+Hz58rKRbokt0tpbpcqpGOapwFTgGbgmoiYKelCYFpETAauBiZKmk3S8xxXqV5VSLBmZtYGX8KbmdXICbQTqfRMmlkh//+l8ZxAOxf/QVjVKt0AsfbzGGgnIWk08AXgCWBuRNyS7ldX/kORtAPwSkT8b9ax5ImkfYCjgNnAUxFxZ8YhrZXcA+0EJO0K/Ay4C3gF+G7rZAhdPHl+Gngc+EE1Ez90FZIOAi4FngJWA/8maZtso1o7+TGmzmEDkmfSrgWQdD9wf9r5rDjhwdpI0vok7zefDIwAviXpuxGRl1dyMyFpU+Ak4CsRca+k/iTt1JekN2p15B5o57AM6JsmDSJiDvBJ4BhJh2QaWUYiYhlwQ0T8EjgD6E6SRAdnG1m2ImIxcBkwIx3eWUDyfvfu2Ua2dnIC7QQiYirwFnB7wb45wH+TTIDQJUXE7DRJtAAnAj2AsyT1kHRUeonfZbTedY+IuyPi1YJDb5K8loikMZJ2yyK+tZETaM5JagKIiGOBNyXdK6l3engTYKfMgsuBiAhJzRGxiiSJvgNMBX4AdKkbSyXGw1v/vucAcyUdAJwHvNShga3FPAaaM6131Vv/GxGrJXWPiJURMVbSNcCVklaRjP0dnnHIHaK4XQr3RURLaxKVNIvkaYU9I2JWtlE3XqV2SYu9DVwLPAMcHxH/zCjctY4TaI4UPZK0MfAqQESslNQjIlZExPGSPg5sBCyMiLlZxdtRyrRLFCXR3sDmwKciYmZW8XaUatolPdZCMo5+dET4RlId+TnQHJJ0CvBp4O/AvyLi8nR/U0SszjS4DLXVLkVleqQzjncZVbZLv4gonr7N2sljoDkj6RiSB6C/AYwkWb8FgC6ePNtsl/R461hxV0ueldqlG4CTZ2M4gWasxPvK65I8lrMHybRbE9JyQzo4tEytabt0lX9camiXVR0aYBfjBJqhooH/1seR3iF54+iYiDggvTFyCnC4pO5ZxdqR3C6luV3yxzeRMlTwx3AmsEs64evtwG5AD0mbA6OBU0luAKzMLNgO5HYpze2SP76JlDFJxwNfBA6NiFck9QKGk1yS7UfSwzgnIp7OMMwO53Ypze2SL06gGUt7EyuBp0neWd4v/XwOyUQQq7vajRFwu7TF7ZIvHgPtQG1McPs0sCtwAck7y1eQvNe9YUQs7wp/DG6X0twu+ecx0A5SdAPgy0BvoHtEfEfSX4D1I+INSWNJJn7oEv+4uV1Kc7t0EhHhrQM23hsuOQP4K/AR4F/ALQVljgNmAttmHa/bxe3irfLmf7UaTNIoSftHRKR3SXcCDgUOAR4AtpA0JS3+FDAmusZriG6XEtwunUzWGXxt34B/A+YC+6bfNyGZy3Nq+n1LksH//8k6VrdL9pvbpXNtHgNtEEk7kbwZcgewAvippAkRcbek5cDTkjYEdga+DdycXbQdx+1Smtulc3ICbQBJY0jmo/wJsDQiJknqSfpHAUwH1gN+QXJHdf9IJkheq7ldSnO7dGJZd4HXtg3Yi2TtmU+UOHYMyYw5O5I8erIdMCjrmN0ubhdvtW3ugdbfTsDPIuIRSd0ieTe5dc7KX0laAUwmeXf5vmxD7VBul9LcLp2YE2idFDy3txXwerq7Bd73DvP2wJ+AoIssN+F2Kc3tsnbwY0x10vp/euBWYFdJO0VESGpqnauS5LW7rSJiUnSRZRXcLqW5XdYOTqD19wjwIHBk+kexOpJ1jY4ExgFLsw0vM26X0twunZgnE2mAdK7GE4B9gceA5SSLvx0eXXiWHLdLaW6XzssJtEEkrUtyg2A/YBFwb0T8I9uosud2Kc3t0jk5gZqZ1chjoGZmNXICNTOrkROomVmNnEDNzGrkBGpmViMnUDOzGjmBWkmSWiTNkPS0pEmS1mtHXXtLuj39PFbSWWXKbizpSzWc43xJ/1Ht/qIy10o6fA3ONUiSH3A3J1Br0zsRMSIihpNM8HtK4UEl1vj/PxExOSJ+WKbIxsAaJ1CzLDiBWjUeALZJe16zJF1OMsnvQEkHSJoqaXraU90AQNJoSc9KehA4rLUiSeMlXZZ+3kzSrZKeTLfdgR8CW6e934vScl+X9JikpyRdUFDXtyQ9l65S+ZFKv4Skk9J6npR0S1Gvej9JD0j6h6SD0/LNki4qOPe/t7chbe3iBGplSeoGHEgysS8kier6iNgBWAacC+wXETsC04CvSVoHuIpkIbQ9gc3bqP5S4K8RsT3JpMEzgbOAOWnv9+uSDgCGALsAI4CdJH0yXQJjHLADSYIeWcWv87uIGJmebxbJ++etBpFMbjwGuCL9HU4AXo+IkWn9J0naqorzWBfh+UCtLetKmpF+fgC4GugHzIuIh9P9uwLDgIckAfQApgIfBeZGxPMAkn4FnFziHPsAxwJERAvwuqTeRWUOSLcn0u8bkCTUDYFbI+Lt9ByTq/idhkv6LskwwQbAlIJjN0fEauB5Sf9Mf4cDgO0Kxkd7pef2O+oGOIFa296JiBGFO9IkuaxwF3BXRBxVVG4EySTA9SDgBxHxi6JzTKjhHNcCh0bEk5LGA3sXHCuuK9JzfyUiChMtkgat4XltLeVLeGuPh4FRkrYBkLSepKHAs8BWkrZOyx3Vxs/fDZya/myzpI2AN0l6l62mAMcXjK32l7QpcD/wWUnrKlmt8pAq4t0QWCSpO8nywYWOSCcz3hoYDDyXnvvUtDyShkpav4rzWBfhHqjVLCJeTntyNyhZRRLg3Ij4h6STgT9KWkIyYfDwElWcAVwp6QSS5SxOjYipkh5KHxO6Mx0H/RgwNe0Bv0WyPtB0STcBM4B5JMMMlXybZALjeSRjuoWJ+jngr8BmwCkRsVzSL0nGRqcrOfnLwKHVtY51BZ7OzsysRr6ENzOrkROomVmNnEDNzGrkBGpmViMnUDOzGjmBmpnVyAnUzKxG/x9MjN9SDQN3mAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 2 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.figure()\n",
    "matriz_confusion = confusion_matrix(y_true=y_real, y_pred=y_predict, labels=[0, 1, 2])\n",
    "plot_confusion_matrix(matriz_confusion, classes=['Clase 0', 'Clase 1', 'Clase 2'], title='Matriz de Confusión')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>\n",
    "\n",
    "*Este Notebook ha sido desarrollado por **Ricardo Moya García** y registrado en Safe Creative como ***Atribución-NoComercial-CompartirIgual***.*\n",
    "\n",
    "<img src=\"./imgs/CC_BY-NC-SA.png\" alt=\"CC BY-NC\">"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}