Assignment _3_Solutions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Assignment 3: Classification with Logistic Regression\n",
    "\n",
    "# Total: /100\n",
    "\n",
    "## Instructions\n",
    "\n",
    "* Complete the assignment\n",
    "\n",
    "* Once the notebook is complete, restart your kernel and rerun your cells\n",
    "\n",
    "* Submit this notebook to owl by the deadline\n",
    "\n",
    "* You may use any python library functions you wish to complete the assignment.\n",
    "\n",
    "## The Dataset\n",
    "\n",
    "The dataset contains transactions made by credit cards in September 2013 by European cardholders. \n",
    "This dataset presents transactions that occurred in two days, where we have 492 frauds out of 284,807 transactions. The dataset is highly unbalanced, the positive class (frauds) account for 0.172% of all transactions.\n",
    "\n",
    "It contains only numerical input variables which are the result of a PCA transformation. Unfortunately, due to confidentiality issues, we cannot provide the original features and more background information about the data. Features V1, V2, … V28 are the principal components obtained with PCA, the only features which have not been transformed with PCA are 'Time' and 'Amount'. \\[You will learn about PCA in a later Lesson.\\] Feature 'Time' contains the seconds elapsed between each transaction and the first transaction in the dataset. The feature 'Amount' is the transaction Amount. Feature 'Class' is the response variable and it takes value 1 in case of fraud and 0 otherwise.\n",
    "\n",
    "You can read more about the data and the variables [here](https://www.kaggle.com/mlg-ulb/creditcardfraud)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You may need these\n",
    "import pandas as pd\n",
    "import seaborn as sns\n",
    "import numpy as np\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import confusion_matrix, roc_curve, precision_recall_curve, auc\n",
    "\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 1: /5 pts\n",
    "\n",
    "Read in the `creditcard.csv` dataset and display the first 5 rows.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Time</th>\n",
       "      <th>V1</th>\n",
       "      <th>V2</th>\n",
       "      <th>V3</th>\n",
       "      <th>V4</th>\n",
       "      <th>V5</th>\n",
       "      <th>V6</th>\n",
       "      <th>V7</th>\n",
       "      <th>V8</th>\n",
       "      <th>V9</th>\n",
       "      <th>...</th>\n",
       "      <th>V21</th>\n",
       "      <th>V22</th>\n",
       "      <th>V23</th>\n",
       "      <th>V24</th>\n",
       "      <th>V25</th>\n",
       "      <th>V26</th>\n",
       "      <th>V27</th>\n",
       "      <th>V28</th>\n",
       "      <th>Amount</th>\n",
       "      <th>Class</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>-1.359807</td>\n",
       "      <td>-0.072781</td>\n",
       "      <td>2.536347</td>\n",
       "      <td>1.378155</td>\n",
       "      <td>-0.338321</td>\n",
       "      <td>0.462388</td>\n",
       "      <td>0.239599</td>\n",
       "      <td>0.098698</td>\n",
       "      <td>0.363787</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.018307</td>\n",
       "      <td>0.277838</td>\n",
       "      <td>-0.110474</td>\n",
       "      <td>0.066928</td>\n",
       "      <td>0.128539</td>\n",
       "      <td>-0.189115</td>\n",
       "      <td>0.133558</td>\n",
       "      <td>-0.021053</td>\n",
       "      <td>149.62</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.191857</td>\n",
       "      <td>0.266151</td>\n",
       "      <td>0.166480</td>\n",
       "      <td>0.448154</td>\n",
       "      <td>0.060018</td>\n",
       "      <td>-0.082361</td>\n",
       "      <td>-0.078803</td>\n",
       "      <td>0.085102</td>\n",
       "      <td>-0.255425</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.225775</td>\n",
       "      <td>-0.638672</td>\n",
       "      <td>0.101288</td>\n",
       "      <td>-0.339846</td>\n",
       "      <td>0.167170</td>\n",
       "      <td>0.125895</td>\n",
       "      <td>-0.008983</td>\n",
       "      <td>0.014724</td>\n",
       "      <td>2.69</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.0</td>\n",
       "      <td>-1.358354</td>\n",
       "      <td>-1.340163</td>\n",
       "      <td>1.773209</td>\n",
       "      <td>0.379780</td>\n",
       "      <td>-0.503198</td>\n",
       "      <td>1.800499</td>\n",
       "      <td>0.791461</td>\n",
       "      <td>0.247676</td>\n",
       "      <td>-1.514654</td>\n",
       "      <td>...</td>\n",
       "      <td>0.247998</td>\n",
       "      <td>0.771679</td>\n",
       "      <td>0.909412</td>\n",
       "      <td>-0.689281</td>\n",
       "      <td>-0.327642</td>\n",
       "      <td>-0.139097</td>\n",
       "      <td>-0.055353</td>\n",
       "      <td>-0.059752</td>\n",
       "      <td>378.66</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.0</td>\n",
       "      <td>-0.966272</td>\n",
       "      <td>-0.185226</td>\n",
       "      <td>1.792993</td>\n",
       "      <td>-0.863291</td>\n",
       "      <td>-0.010309</td>\n",
       "      <td>1.247203</td>\n",
       "      <td>0.237609</td>\n",
       "      <td>0.377436</td>\n",
       "      <td>-1.387024</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.108300</td>\n",
       "      <td>0.005274</td>\n",
       "      <td>-0.190321</td>\n",
       "      <td>-1.175575</td>\n",
       "      <td>0.647376</td>\n",
       "      <td>-0.221929</td>\n",
       "      <td>0.062723</td>\n",
       "      <td>0.061458</td>\n",
       "      <td>123.50</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2.0</td>\n",
       "      <td>-1.158233</td>\n",
       "      <td>0.877737</td>\n",
       "      <td>1.548718</td>\n",
       "      <td>0.403034</td>\n",
       "      <td>-0.407193</td>\n",
       "      <td>0.095921</td>\n",
       "      <td>0.592941</td>\n",
       "      <td>-0.270533</td>\n",
       "      <td>0.817739</td>\n",
       "      <td>...</td>\n",
       "      <td>-0.009431</td>\n",
       "      <td>0.798278</td>\n",
       "      <td>-0.137458</td>\n",
       "      <td>0.141267</td>\n",
       "      <td>-0.206010</td>\n",
       "      <td>0.502292</td>\n",
       "      <td>0.219422</td>\n",
       "      <td>0.215153</td>\n",
       "      <td>69.99</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 31 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   Time        V1        V2        V3        V4        V5        V6        V7  \\\n",
       "0   0.0 -1.359807 -0.072781  2.536347  1.378155 -0.338321  0.462388  0.239599   \n",
       "1   0.0  1.191857  0.266151  0.166480  0.448154  0.060018 -0.082361 -0.078803   \n",
       "2   1.0 -1.358354 -1.340163  1.773209  0.379780 -0.503198  1.800499  0.791461   \n",
       "3   1.0 -0.966272 -0.185226  1.792993 -0.863291 -0.010309  1.247203  0.237609   \n",
       "4   2.0 -1.158233  0.877737  1.548718  0.403034 -0.407193  0.095921  0.592941   \n",
       "\n",
       "         V8        V9  ...       V21       V22       V23       V24       V25  \\\n",
       "0  0.098698  0.363787  ... -0.018307  0.277838 -0.110474  0.066928  0.128539   \n",
       "1  0.085102 -0.255425  ... -0.225775 -0.638672  0.101288 -0.339846  0.167170   \n",
       "2  0.247676 -1.514654  ...  0.247998  0.771679  0.909412 -0.689281 -0.327642   \n",
       "3  0.377436 -1.387024  ... -0.108300  0.005274 -0.190321 -1.175575  0.647376   \n",
       "4 -0.270533  0.817739  ... -0.009431  0.798278 -0.137458  0.141267 -0.206010   \n",
       "\n",
       "        V26       V27       V28  Amount  Class  \n",
       "0 -0.189115  0.133558 -0.021053  149.62      0  \n",
       "1  0.125895 -0.008983  0.014724    2.69      0  \n",
       "2 -0.139097 -0.055353 -0.059752  378.66      0  \n",
       "3 -0.221929  0.062723  0.061458  123.50      0  \n",
       "4  0.502292  0.219422  0.215153   69.99      0  \n",
       "\n",
       "[5 rows x 31 columns]"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Question 1 code here\n",
    "df = pd.read_csv('creditcard.csv')\n",
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 2: /5 pts\n",
    "\n",
    "Then split the data into train and test for the outcome/response and the predictor variables. Hold out 50% of observations as the test set.  Pass `random_state=0` to `train_test_split` to ensure you get the same train and tests sets as the solution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Question 2 code here.\n",
    "\n",
    "# Don't use the \"Class\" as a feature\n",
    "X = df.drop('Class', axis='columns').values\n",
    "# Convert the 0/1 coding of \"Class\" to \"Not Fraud\" and \"Fraud\"\n",
    "# y = np.array([\"Not Fraud\",\"Fraud\"])[df.Class.values]\n",
    "y = df.Class.values\n",
    "Xtrain, Xtest, ytrain, ytest = train_test_split(X,\n",
    "                                                y, \n",
    "                                                test_size=0.5, \n",
    "                                                random_state=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 3: /5 pts\n",
    "\n",
    "Read the documentation for [sklearn's `LogisticRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LogisticRegression.html).  In no more than 2 sentences per bullet point, answer the following in your own words.\n",
    "\n",
    "* Does `LogisticRegression` use a penalty by default?  If yes, what penalty?\n",
    "\n",
    "**YOUR ANSWER HERE:** *Yes, it uses L2 penalty by default.*\n",
    "    \n",
    "* If we apply a penalty during learning, what difference do you expect to see in the resulting coefficients, relative to not applying a penalty during learning?\n",
    "\n",
    "**YOUR ANSWER HERE:** *We expect the coefficients to be smaller if a penalty is applied.*\n",
    "    \n",
    "* If using the default settings of `LogisticRegression` in `sklearn`, should you include a column of 1s in your feature/design matrix? Briefly explain why or why not.\n",
    "\n",
    "**YOUR ANSWER HERE:** *A column of 1s should not be included because by default sklearn estimates an intercept.*\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 4: /15 pts\n",
    "\n",
    "Create a instance of sklearn's `LogisticRegression` object for unpenalized logistic regression. Note: If you get a warning about convergence of `coef_`, try increasing the `max_iter` parameter.  I used `max_iter=10000` which seems to supress the warning.\n",
    "\n",
    "Using this object, **run a logisitic regression analysis** of `Class` (y-variable) against `Amount` (x-variable) using your training data. \n",
    "\n",
    "Then **make a scatter-plot of `Amount` and `Outcome` on your test data** and add the class prediction (0 or 1, using `predict`) and the predicted probability of a positive outcome (using `predict_proba`). Note that `predict_proba` will return both p(Outcome=0) and p(Outcome=1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<matplotlib.axes._subplots.AxesSubplot at 0x7fa1acfdde80>"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXQAAAD4CAYAAAD8Zh1EAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZwcVbn/8c/TPVsmmRAmO1lIAgFBCCAJEDEssgVUQDaD7AqIgKL3cgXkqnj1ykX0dwEXEBWJFyTsEDEk4oK4sCRhyQIEspGEJGQZsszW3dV9fn9Uz6QzmZn0zFR3dfd836/XvGa66nTVc6qTZ848darKnHOIiEjxi4QdgIiIBEMJXUSkRCihi4iUCCV0EZESoYQuIlIiysLa8aBBg9yYMWPC2r2ISFGaP3/+Jufc4PbWhZbQx4wZw7x588LavYhIUTKz9zpap5KLiEiJUEIXESkRSugiIiVCCV1EpEQooYuIlIjdJnQzu8/MNpjZog7Wm5ndZWZLzWyBmX0s+DB9npeiMebRGPOob/ZIOUd9c6J1WSzuL9venCCZSrG9OUEs7q9LOUd9zCOZTFGf8TqVcqRSbqdl7bURESl02YzQ7wemdrL+VGB8+utK4O6eh7Urz0tRH/doTHjUNca54rfz2O/mZ7nit/PZHvNoSiRp8lK88M4GrvztfN7/sJnf/GMFW5s9Ujj+7eHXuWL6PDY3xLnv78v9906fx/bmBJsbYlwxfV7rsrZtNjfEldRFpOBZNrfPNbMxwDPOuYPaWfcL4Hnn3EPp10uA45xz6zrb5sSJE11X5qFvb06wuT4OwE1PLOTF5Ztb100eN5BbzzoYgNq+FUz47h+ZPG4gt5z+UW6ZuZhfXHQ467Y2c8odL7QuP+WOFwB4/vrj2t1eZpvJ4wbyy0sm0q8ytGn7IkWpMe5x/79W0hxPhh1KQZk4ppZj9mv32qDdMrP5zrmJ7a0LIkONAFZnvF6TXrZLQjezK/FH8YwePbpLO+lbWUZ1hR/u3JV1O62bu7KOUbXV6X3sWLbvkH7MXVlHv6oy9q3st9PyFqNqq9vdXmabuSvrqK6IdileEYEXl23mh7OXADv+bwpcdew+3U7onQkiobf3MbU77HfO3QvcC/4IvSs7aYh5rSP0SWNqdxpRTxpTy+q6RsAfobcsW7qhnkljaqlv9li3tXmn5S1W1zW2u73MNpPG1NIYT2qELtJFTQl/ZP7c149h/NCakKMpfUHMclkDjMp4PRJYG8B2d9KnLMqA6nL6VUW5/dwJTB43kLKIMXncQO6Ydig1VWUMqC7n1VV1TB43kNvOnsCcReu4c9qhRCJw9/NLmTxuIHdOO5Q5i9a1vnfP6nLuOv/QnbbXts1d5x9GdblG6CJdFUukAKgs0/+ffAiihv4p4FrgNOBI4C7n3BG722ZXa+jgnxiNJ/1/ICkH1ZVRGmMekfTfclGD8vIoDTGP6ooojfEkFREj6aAq/bpPWYQmL9W6viVRNyaSrcvaaxOJ6O9Fka763cur+OaTC3n5mycwtH9V2OGUhB7V0M3sIeA4YJCZrQG+A5QDOOfuAWbhJ/OlQCNwWTBhtxNsWYSysp3/qOhXVb5Lu5r0spqqNm3TJZN+0chOr3da10kbEemamOeXXCrLdMlLPuw2Wznnzt/NegdcE1hEIlIyYp5KLvmkX5sikjMtNfQKjdDzQkdZRHIm5iUpjxpRnYPKCyV0EcmZmJdSuSWPlNBFJGdiXlInRPNIR1pEciaWSCmh55GOtIjkTMxLUamL8vJGCV1EckYll/zSkRaRnPFPiirN5IuOtIjkjF9DV8klX5TQRSRnmr0kleVKM/miIy0iOaNZLvmlIy0iOeOfFFXJJV+U0EUkZ3RSNL90pEUkZ/x56Eoz+aIjLSI5E0uo5JJPSugikjMaoeeXjrSI5IRzTndbzDMldBHJiZbn/+qkaP7oSItITux4/JzSTL7oSItITrQ8fk53W2xjzXyo35CTTSuhi0hOxLwkoBF6q0QT/PFb8OsT4flbc7KLspxsVUR6PZVcMrz3Ijx9DdQtg8MvhRNvyclulNBFJCdaSy69eZZLrB7+/F/wyr0wYBRc/DSMOy5nu1NCF5GcaC259NZ56Mv/BjO/AlvegyO+BCd8Gyr75XSXSugikhO9tuTixeHP34UXfwq1+8Blz8LeH8/LrpXQRSQndiT0XlRyqVsBj30B1r4Kky6Hk78P5X3ytnsldBHJiViil81yWfwkzPwqYHDeb+HAM/IeghK6iOREywi9qtRr6IkmmH0TzP8NjJgI59wHe+4dSihK6CKSE72i5LJpKTxyMWxYDEdfB5/8FkTLQwtHCV1EcqLkLyxa+md49DKIlsEFj8P4E8OOSAldRHKjuVTnoTsHL90Nf7wZhhwI034XWomlrax+dZrZVDNbYmZLzezGdtbvYWa/N7M3zGyxmV0WfKgiUkxKch66F4OZ18Kcm2D/0+ALcwommUMWI3QziwI/A04C1gBzzWymc+7NjGbXAG865z5jZoOBJWb2oHMunpOoRaTgtVwpWhEtkYRevwEevghWvwTHfAOOuwkihdW3bEouRwBLnXPLAcxsBnAGkJnQHVBjZgb0A+oAL+BYRaSIxLwUFdEIkYiFHUrPrV8Iv5sGjZvhnN/AQWeFHVG7svn1MgJYnfF6TXpZpp8CBwBrgYXAdc65VNsNmdmVZjbPzOZt3LixmyGLSDGIecnSOCG64gW471RwKfjC7IJN5pBdQm/v16tr8/oU4HVgL+BQ4Kdm1n+XNzl3r3NuonNu4uDBg7scrIgUj5J4nuibT8MDZ8MeI+DyP8Feh4YdUaeyOdprgFEZr0fij8QzXQY84XxLgRXAR4IJUUSKUSxR5M8TnXcfPHIJ7HWYfz+WPdoWJgpPNgl9LjDezMaaWQUwDZjZps0q4AQAMxsK7A8sDzJQESkuMS9ZnCN05+D52+CZr8P4k+Gip6C6NuyosrLbk6LOOc/MrgXmAFHgPufcYjO7Kr3+HuB7wP1mthC/RHODc25TDuMWkQIX84pwhJ5KwrPfgLm/gkM+D6ffFeqVn12V1YVFzrlZwKw2y+7J+HktcHKwoYlIMfMTehGN0JMJeOJKWPyEfxn/id8FK64ZOrpSVERyIpYoolkuyQQ8/kX/JOiJ34VPfC3siLqlSI62iBQbf5ZLEZRckgn/HuZvPg2n/KBokzlohC4iOVIUJZdkAh67DN76vZ/MJ18TdkQ9ooQuIjlR8BcWeXE/mb/9DJxyK0y+OuyIekwJXURyoqDnoWcm86n/A0d9OeyIAqGELiI5UbBXiraUWd5+BqbeBkddFXZEgSnAoy0ipaAgSy6pFDx9TUkmc1BCF5EcKbgLi5yD2TfCgofhk/9ZcskclNBFJAecc8QLbZbL326DV34BR10DU64PO5qcKKCjLSKlovUB0YVSQ3/5F/D8rf7l/Cd/v+iuAM1WgRxtESklrQm9EEouCx7x78+y/6fg9J8U3FOGglS6PROR0MQS6eeJhl1yWTIbnrwKxkyBc+6DaGlP7FNCF5HA7Rihh5hiVr0Mj14CwyfA+Q9BeVV4seSJErqIBC7mpUfoYd3LZfMyeGga9B8BFzwGlTXhxJFnSugiErjmRIgj9MY6ePBc/+cLHoW+g/IfQ0hKu6AkIqEIreSSaIYZn4eta+CSmTBwn/zuP2RK6CISuNaSSz5nuaRS8PTVsOpFOOc3MPqo/O27QKjkIiKBaxmhV+VzHvpfvw+LHocTb4GDzsrffguIErqIBC6WyPM89PnT4e8/hsMvhaOL9wEVPaWELiKB2zHLJQ8pZtlf4Zmvwz4nwGk/LtmrQLOhhC4igcvbSdHNy+DRS2Hw/nDu/SV/4dDuKKGLSODycul/8zZ46HywiH/hUFX/3O2rSPTuX2cikhOtl/7nquSSSsLjl0PdMrjoKdhzTG72U2SU0EUkcDkvufzle/DuHDjtRzB2Sm72UYRUchGRwLUk9IpoDlLMwsfgH/8Lh18Gky4PfvtFTAldRALX8vg5C3rGyfuv+o+QG/1xOPWHvXpGS3uU0EUkcLFEDp5WtP0DmHEB9B0M5/0WyiqC3X4JUA1dRAIX81LB3mnRi8MjF0PzFvjCHOg3OLhtlxAldBEJXEvJJTB//E9Y/ZL/kIrhE4LbbolRyUVEAhcL8gHRbzy84+HOB50dzDZLVFZH3MymmtkSM1tqZjd20OY4M3vdzBab2d+CDVNEiolfQw+g5LJ+Efz+Otj7aDjpuz3fXonbbcnFzKLAz4CTgDXAXDOb6Zx7M6PNAODnwFTn3CozG5KrgEWk8MW8ZM8vKmraAg9fCH0G+LfDjZYHE1wJy+aIHwEsdc4td87FgRnAGW3afB54wjm3CsA5tyHYMEWkmPR4lksqBU9+yX9QxbnToWZocMGVsGyO+AhgdcbrNellmfYD9jSz581svpld3N6GzOxKM5tnZvM2btzYvYhFpOD5J0V7UHL5+4/gndkw9VYYfWRwgZW4bBJ6ezP3XZvXZcDhwKeAU4Bvmdl+u7zJuXudcxOdcxMHD9a0I5FS1aOTou/+Cf76A5jwOV0J2kXZTFtcA4zKeD0SWNtOm03OuQagwcxeAA4B3gkkShEpKt2eh75lFTxxOQz9KHz6Dl0J2kXZ/AqdC4w3s7FmVgFMA2a2afM0MMXMysysGjgSeCvYUEWkWMQS3ZiH7sX9e5unkv6VoBXVOYmtlO12hO6c88zsWmAOEAXuc84tNrOr0uvvcc69ZWazgQVACviVc25RLgMXkcLVrZLLc9+C9+fDef8HA/fJTWAlLqsrRZ1zs4BZbZbd0+b17cDtwYUmIsUq5qWo6krJZfGT8PI9cNTVcODpuQusxOlKUREJXJcu/d+0FJ7+CoycBCfq4qGeUEIXkUAlU45E0mU3bTHRBI9e4j8L9Jzf6A6KPaSbc4lIoOItTyvK5krRWdfDB4vggsdgwKjdt5dOaYQuIoGKeennie6u5PLag/DaAzDlehh/Uh4iK31K6CISqB3PE+2k5PLBm/CHf4cxU+D4b+YpstKnhC4igYoldvOA6HiDP9+8sgbO/jVEAnwQRi+nGrqIBKq15NJRDf0P18Omd+Dip3TTrYBphC4igeq05PLag/DG7+DYG2DccXmNqzdQQheRQHV4UnTDWzvq5sd+I4TISp8SuogEqt0aemvdvB+c/SvVzXNENXQRCVRrySXz0v9Z34CNS+CiJ6FmWEiRlT6N0EUkULuUXF5/CF5/AI75D9jn+BAjK31K6CISqB0nRSP+qPwP/+bXzY9r9/nyEiAldBEJVEsNvYoYPHIJlFfDWb9U3TwPVEMXkUA1p0sutS98Cza+DRc+Dv2HhxxV76ARuogEKpZIcWbkH1QtfBCm/Dvse0LYIfUaSugiEqg+25bx3+W/JjVqMhx3U9jh9CpK6CISnEQTp7x5I81UYOf82r/PueSNErqIBGf2jQxuXMoNqWuwPUaEHU2vo4QuIsFY+BjMv5+/D7mQuWWHhx1Nr6SELiI9t2kp/P46GHUUzw75IlXZPK1IAqejLiI9k2jy79MSLYdzfk1TMpLd80QlcEroItIzz94AHyyEz94Le4wk5iV3//g5yQkddRHpvjcehlenwyf+DfY7GfDnoWf1gGgJnI66iHTPhrfhma/B3kfD8Te3Lo55KZVcQqKELiJdF2+AR9P3aTl75/nmKrmER0ddRLrGOf/JQxuX+A+raHOfFn+ErtQSBh11Eema1x6ANx7ynwvazv3NYwmVXMKihC4i2Vu/CGZdD2OP7fC5oDEvqZOiIdFRF5HsNG+FRy6GqgGdPhdUJZfw6M45IrJ7zsFTV8OHK+HSZ6DfkA6bapZLeLL6NWpmU81siZktNbMOnyNlZpPMLGlm5wQXooiE7l93wdvPwMnfg70/3mnTWEKzXMKy26NuZlHgZ8CpwIHA+WZ2YAftbgPmBB2kiIRoxd/hT7fAgWfCUVfvtnnM04VFYcnmqB8BLHXOLXfOxYEZwBnttPsK8DiwIcD4RCRM29bCY5fBwH3hjJ+CWafNvWQKL+VUcglJNgl9BLA64/Wa9LJWZjYC+CxwT2cbMrMrzWyemc3buHFjV2MVkXzy4v5Nt+KN8LkHoLJmt2+JJ/0HRKvkEo5sjnp7v5Jdm9d3ADc455Kdbcg5d69zbqJzbuLgwYOzjVFEwvDct2D1y/7IfPD+Wb2lOaGEHqZsZrmsAUZlvB4JrG3TZiIww/w/xwYBp5mZ55x7KpAoRSS/Fj4GL9/j18wPOivrt8U8f0xXWa6SSxiySehzgfFmNhZ4H5gGfD6zgXNubMvPZnY/8IySuUiRWr8IZn4FRh0FJ/1Xl94a0wg9VLtN6M45z8yuxZ+9EgXuc84tNrOr0us7rZuLSBFprIMZn4eqPeC86f5DK7og5rUkdI3Qw5DVhUXOuVnArDbL2k3kzrlLex6WiORd0vPvoLh9HVz2LNQM6/ImWksuGqGHQleKiojvuW/BihfgjJ/DyInd2kTrCF3z0EOhoy4i8PpD8NLP4cir4LALur2Zlhp6lU6KhkIJXaS3WzMffn8djJkCJ3+/R5tSySVcOuoivdn2D+DhC6HfUDi36ydB29JJ0XCphi7SW3kxeOQiaPoQvvhH6Duwx5vUCD1cSugivZFzMPOr/pWg59wHwycEstnWeeg6KRoKHXWR3uiFH8GCGXD8zXDQ2YFtViWXcCmhi/Q2Cx+Dv34fJnwOjvmPQDetkku4dNRFepPVr/hPHho9GU7/yW5vh9tVuvQ/XDrqIr3FhyvhofOh/17wuQehrDLwXcS8FNGIURZVagmDjrpIb9C0BR48D1IeXPBoIDNa2hPz9Pi5MGmWi0ipSyb8B1XULYOLnoJB43O2K/8B0UroYVFCFyllzvlXgS7/K5zxMxg7Jae7iyVSmuESIv0qFSllf/oOvP4gHHcTHHZhzncX85Kagx4iHXmRUvWvn8I/74RJl8OxN+Rllyq5hEtHXqQUvTED/ngzHHgGnPrDwKcndsRP6Cq5hEUJXaTUvPscPH0NjD0GzvolRPKXYJsTmuUSJh15kVKyZh48cjEM/WjO5pp3JualVEMPkY68SKnYuAQePMe/Fe4Fj0FV/7yH4M9DV8klLEroIqVg8zKYfjpEK+CiJ6HfkFDC8KctKq2ERfPQRYpd3XK4/9P+VaCX/gFqx4YWima5hEsJXaSYfbgS7v8MeM1w6TMw5COhhqOSS7iU0EWK1ZbVMP0zEK+HS37vnwgNWcxLUaWToqHRkRcpRlvfh+mfhqatcPFTgT1xqKdiiRSV5Rqhh0UjdJFis22dn8wb6/ybbe11WNgRAeCc090WQ6YjL1JMtqyC+z8F9Rvgwsdh5OFhR9TKSzlSTg+3CJNG6CLFYuM78H9n+jXzC5+AUUeEHdFO9DzR8CmhixSDta/BA2eDReHSWTDsoLAj2kUskX6eqE6KhkZHXqTQrfyHPzWxvC98YXZBJnPIHKErrYRFR16kkC2Z7Y/M++8FX5wDA/cJO6IOqeQSvqwSuplNNbMlZrbUzG5sZ/0FZrYg/fUvMzsk+FBFepkFj8CMz8OQA+CyZ/2kXsBiXrrkohF6aHZ75M0sCvwMOBU4EDjfzA5s02wFcKxzbgLwPeDeoAMV6TWcgxduhyeugL0/DhfPzNlDnYMUS6RH6Kqhhyabk6JHAEudc8sBzGwGcAbwZksD59y/Mtq/BIwMMkiRXsOLwcyvwIKH4eDz4PSfQHlV2FFlRSWX8GWT0EcAqzNerwGO7KT9F4Fn21thZlcCVwKMHj06yxBFeomGTTDjAlj9Ehx/MxzzH3l70lAQVHIJXzYJvb1/Ua7dhmbH4yf0T7S33jl3L+lyzMSJE9vdhkivtHEJ/O482L4ezrkPDjo77Ii6rLXkohF6aLJJ6GuAURmvRwJr2zYyswnAr4BTnXObgwlPpBdY9hd45FL/6UKX/gFGTgw7om5pLbmohh6abI78XGC8mY01swpgGjAzs4GZjQaeAC5yzr0TfJgiJcg5+Oed8MA5sMdIuOLPRZvMQSWXQrDbEbpzzjOza4E5QBS4zzm32MyuSq+/B/g2MBD4ufk1P885V7z/MkVyrelDeOpqWDILDjgdzvw5VNaEHVWPNKvkErqsLv13zs0CZrVZdk/Gz5cDlwcbmkiJev9VePQS/66JU2+DI79UVCc/O6IRevh0LxeRfHEO5v4K5nwT+g7xL+Mv4hJLW6qhh08JXSQfmrfBM1+DRY/D+JPhs7+A6tqwowpUyyyXiqgSeliU0EVybcUL8NQ1sG0NnPBtOPrrECm9pBfzkpRFjDIl9NAooYvkSrwR/nQLvPILqN0HvjCn4O5hHqSYl1L9PGRK6CK5sOpleOrLULcMjvyyPzKvqA47qpyKeUmq9DzRUCmhiwQp0QzP/wD+9RN/bvklz8DYKWFHlRexhEboYVNCFwnK0j/BrG/4o/LDL4OTv1f0c8u7IualqNQIPVRK6CI9tWU1zLkJ3vo9DNwXLnoS9vlk2FHlXcxLaoQeMiV0ke7yYn5p5YUf+RcGnfAdmHyNf0+WXkgnRcOnhC7SVc7Bu8/B7Bv98soBp8MpP4ABo3b/3hLm19BVcgmTErpIV6x6Gf78XXjvn3555cInYN8Two6qIMS8JH0rlVLCpKMvko31i+Av34N3ZvuX7Z/2I/jYJVBWEXZkBSPmpajtq5JLmJTQRTpTtwL++gNY+ChU9ffnkx95FVT0DTuyguPX0FVyCZMSukh71i+Ef97l33slWgGf+BocfR302TPsyAqWZrmETwldpIVz/n1X/nknLPszVPSDo74MH/8K1AwLO7qCF0ukdKfFkCmhiyQT/hzyf94J6173a+QnfBsmfhH6DAg7uqKhkkv4lNCl99qyCuZPh9cegPr1/g20PnMnTJgG5VVhR1d0VHIJnxK69C5JD96dA/N+41+qDzD+JDj8f2G/UyCiEWZ3OOd0YVEBUEKX0uccfLAIFj4GCx6B7Wuh3zA45nr42MUwYHTYERa9RNLhHLqXS8iU0KV01S2HhY/Dosdg49tgUf8eK6f9EPabCtHysCMsGc16nmhBUEKX0uEcbHoX3nkWFj8Fa1/1l4/+OHzqx3DgmdB3ULgxlqiWx88poYdLCV2KWzIBq16EJbP9RF633F8+/BA46Xtw0Fn+fcklp2KtI3SVXMKkhC7FxTn4cAUs/xus+Bss+ws0b/Uv/hl7DBx1tV9O6eU3ysq3mJceoWseeqiU0KXwbVsLK/7uJ/AVL8DW1f7ymuHwkc/A/lNh3PFQ2S/cOHsxlVwKgxK6FBYv7l92v+YVWP0KrJm7I4H32RPGTPEvwR93nH+3Q7Mwo5U0lVwKgxK6hMeL+7NP1i+AdQtg3Rv+lZpes7++/0gYNckvo4w5GoYeDBGNAAuRSi6FQQldcs852LoGNi2BjUvggzdh/Ruw4W1IJfw25X1h2EEw6XIYOQlGHQH99wo3bslaa0LXCD1USugSnOat/iyTuhX+ictN7/oJfNM7EK/f0a56EAyfAJNPgGEH+zNSasfpKs0iFktoHnohUEKX7DVv80faW9fA1lX+9y2r4cOVfgJv3Lxz+5rhMHh/OOxC//ug/f3vmgteclpG6FUquYRKCb23S6WgeQs0bPS/6jdA/QewfR1sX7/zV2zrzu+NlEH/EbDnGDjgM/4ou3Yc7DkWasfqIRC9iEouhUEJvVSkUhDf7pc9mrf6o+nmrdD0ITTV+d8b63b+uWEjNGwCl9x1e5Fyf4RdM8wfVY871r9AZ49R6a+R0G+IyiQCZM5y0Qg9TFkldDObCtwJRIFfOef+p816S68/DWgELnXOvRpwrHheingy1fo65aC6MkpjzCNiRplBPOWoriyjMZYkkjGjraoiSnM8SdI5+masj5i/ncrySOtc2pbtNsQ8KiOWsU2PsoiRcrvuu6Ptt+y7MZYkQooKEjQ1N9I34hFrbCCVaKKPxWlu3E6lixNvbqCSGF5TPalYPRWumURTPZFEA2XJJpJN27H4diKJBlxsO8S2Y/F6/zuuw2PnLIrrsydWXUuyak8ie4zCG3Yo5TVDiVXWUtF/CPQdQnNlLVW1I2iM9qeqLEpT+sG/DTGPPuVRmr0U1eVRIpFgpwumUo7GRJI+5REa4/4+G+PJnOxLuqflM6quiO7y2bT83ymPRqiPee22kc6PYRB2m9DNLAr8DDgJWAPMNbOZzrk3M5qdCoxPfx0J3J3+HhjPS9EQ9/CathDZ9j7xhMcdzy3hrbUfctDwGq775D5UlTnmLFzLU6+u5oBhfblyyt5URBzmkjy/YhOH7FXDAy8u572N29h3UB+mTdyLqojDnEdz0iPiksTicWYvWMMHH25n9IAKTj5gIO++X8fC1ZvYq6aMKfsMgEScBSs3sr2xkcF9jAOH9mHD9npqK+GDzVtZH2uipjzFoD5GJBUn1txEZSpOOf6Mjpp0n/pk9K+6zbLMRw8bFUSq+vKhV0E9fRg2eBCuqoZ4n2G88n4cq67hsEP25um361lcB0MGD+Wi4ycw/dU6VjdW8t3zp9BINf/+6ALmrqzjq5/cl2lHjOa6Ga8zd24dk8bUcue0Q6muiHLFb+czd+XLrctmvLKKu/6ytPX1/PfqmDhmIAP7VgT2DzGVcmxuiPPQy+9x5mEjueFxP85JY2q56/zDAt2XdE/LZ/TVh15r97NpKbnUxzy+8aA+v/bs7hgGwZzreFQHYGaTgVucc6ekX98E4Jy7NaPNL4DnnXMPpV8vAY5zzq3raLsTJ0508+bNyzrQ7c0JNtfHWfvP3/Hx167P+n09kXSGRxlJixJ3URJEIVJOc8p/HaeMBGWkrAwrq6Dei/rrKCNOOUkrp6yiihjlbGo2kpFKKir7EKOcGBVsjhnbvHKaqKCJSohW0bemhsZUBWsajLpEOU1UkiRKn/Iog2sqWVXXSJ/yKMMHVLFuSzNNiSSja6vZuD1GU2JH6SSz/ZiB1Xywbcf6jtoPH1DF8o0N7W4js826Lc3sNaCKSEAX9aScY+2WZgbXVLYbV5D7ku5p+Yw6+mzqGuJsbohz1NhaXlpR19pm8riB/PKSifSrVHW3PuZxxfR5vLh8x+SB7hwfM1D+mB4AAAeHSURBVJvvnJvY3rpstjICWJ3xeg27jr7bazMC2Cmhm9mVwJUAo0d37R7UfSvLqK4oY+u+k/nlum8zf9U2/CKGkSSCI8LHxgzEEeGllVtIuggpIhyxz2BSREhZlL8vrSNBlBQRPCI4F2HKR4aRJErSykhZlNlvbsIj6q8nggFTDxrGs4vWA3DqQcOYvWj9TsUNA6aO72D5fv6zKJ9dtH6n18Cu7VMwda9h7AG82GZdcyLJR/fqz6q6RpoTSfYfWsOKdPL96F79mZ2Or732Bwzvz3ubd6zvqP3+Q2t2SuiZ28hss2JjA/sNrQnsIk3nYPnGhg7jCnJf0j0tn1Gmtp/NPoP7cffzy3ZqM3dlHdUVOs8CUF0RZe7Kup2WBX18skno7f1Xajusz6YNzrl7gXvBH6Fnse9WDTGPzfVx9hg6lpPOuYq/PLFwl990Z5x1MDEvxR13vNC67HNnHgzAgOpyFj3w6i7v+dKnD2ZAdTlbGhMMqC5nfZs2R40byNdO3K81od8w9SNsaUzs0ubWsw5ud/kNUz9CzEvx7KL1O70GOtzOB9ti7a5riaPt/r524n6dtm8bc0ftbz3r4NZ+ttf3zP3efu4hgY266mMeWxoTHcYV5L6ke1o+o84+m/qYx7yVH+7UZtKYWhrjSX1+QGM8yaQxtTk9Ptmckl4DZN66biSwthtteqRPWZQ9q8vpVxWlX1WU28+dwORxAymLGJPHDeTH5x3CgOpy5ixa17rs9nMntLZ/+vX3d3nP7edOYEB1OWZQU1WGGbu0ufP8Q3fa5oDqcn583iG77Pvp19/fZXnL/u9+fulO+5uzaB13P790131NO5QXl21qd93t505o3c6PzzuEsoi1ttld+7Yxz1m0jjunHbrLvlt+zlyW2feW+O46/zCqA3wyTXV5lLvOP4w5i9Zx29k79yPofUn3tHxGnX022bTpzfJxfLKpoZcB7wAnAO8Dc4HPO+cWZ7T5FHAt/iyXI4G7nHNHdLbdrtbQoQRmuRhUlEVoSvizODLbN8Q8+mTMKmm7LpreR+t2ohHiyVRrHM3xJCnnx9m2fZ/yCLFkilRqR7/6tJnBUl0excx2OgNfFY1olou0ymaGRq5ncRS7II5PZzX03Sb09AZOA+7An7Z4n3Puv83sKgDn3D3paYs/BabiT1u8zDnXabbuTkIXEentenpSFOfcLGBWm2X3ZPzsgGt6EqSIiPSMLusSESkRSugiIiVCCV1EpEQooYuIlIisZrnkZMdmG4H3uvn2QcCmAMMpdL2pv+praepNfYXc9ndv59zg9laEltB7wszmdTRtpxT1pv6qr6WpN/UVwuuvSi4iIiVCCV1EpEQUa0K/N+wA8qw39Vd9LU29qa8QUn+LsoYuIiK7KtYRuoiItKGELiJSIoouoZvZVDNbYmZLzezGsOPpLjNbaWYLzex1M5uXXlZrZs+Z2bvp73tmtL8p3eclZnZKxvLD09tZamZ3pe98GSozu8/MNpjZooxlgfXNzCrN7OH08pfNbEw++5epg77eYmbvpz/b19N3K21ZV8x9HWVmfzWzt8xssZldl15ecp9tJ30t7M/WOVc0X/i3710GjMN/jvIbwIFhx9XNvqwEBrVZ9kPgxvTPNwK3pX8+MN3XSmBs+hhE0+teASbjPzXqWeDUAujbMcDHgEW56BtwNXBP+udpwMMF1tdbgOvbaVvsfR0OfCz9cw3+cxIOLMXPtpO+FvRnW2wj9COApc655c65ODADOCPkmIJ0BjA9/fN04MyM5TOcczHn3ApgKXCEmQ0H+jvnXnT+v4rfZrwnNM65F4C6NouD7Fvmth4DTgjrL5MO+tqRYu/rOufcq+mftwNv4T87uOQ+20762pGC6GuxJfSOHkZdjBzwRzObb/7DswGGOufWgf8PChiSXt5Rv0ekf267vBAF2bfW9zjnPGArMDBnkXfPtWa2IF2SaSlBlExf0+WBw4CXKfHPtk1foYA/22JL6Fk9jLpIHO2c+xhwKnCNmR3TSduO+l0Kx6M7fSv0ft8N7AMcCqwDfpxeXhJ9NbN+wOPA15xz2zpr2s6youpvO30t6M+22BJ6zh9GnS/OubXp7xuAJ/HLSR+k/0Qj/X1DunlH/V6T/rnt8kIUZN9a32P+M2/3IPuyR8455z5wziWdcyngl/ifLZRAX82sHD/BPeiceyK9uCQ/2/b6WuifbbEl9LnAeDMba2YV+CcSZoYcU5eZWV8zq2n5GTgZWITfl0vSzS4Bnk7/PBOYlj4rPhYYD7yS/vN2u5kdla69XZzxnkITZN8yt3UO8Jd0fbIgtCS3tM/if7ZQ5H1Nx/Zr4C3n3P/LWFVyn21HfS34zzaMM8g9+QJOwz/jvAy4Oex4utmHcfhnxN8AFrf0A79+9mfg3fT32oz33Jzu8xIyZrIAE9P/qJbhP6jbCqB/D+H/OZrAH4V8Mci+AVXAo/gnnl4BxhVYX/8PWAgswP9PO7xE+voJ/JLAAuD19NdppfjZdtLXgv5sdem/iEiJKLaSi4iIdEAJXUSkRCihi4iUCCV0EZESoYQuIlIilNBFREqEErqISIn4/1u3kQAvq4BVAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# Question 4 code here\n",
    "\n",
    "LOGREG_AMOUNT = LogisticRegression(solver='lbfgs',penalty = 'none',max_iter=10000)\n",
    "Xtrainamount = Xtrain[:,[29]]\n",
    "Xtestamount = Xtest[:,[29]]\n",
    "lr_amount = LOGREG_AMOUNT.fit(Xtrainamount,ytrain)\n",
    "\n",
    "plt.scatter(Xtestamount,ytest, edgecolor = 'white')\n",
    "xp=np.reshape(np.linspace(0,max(Xtestamount),50),(-1,1))\n",
    "\n",
    "yp1=lr_amount.predict(xp)\n",
    "yp2=lr_amount.predict_proba(xp)\n",
    "\n",
    "sns.lineplot(xp[:,0],yp1)\n",
    "sns.lineplot(xp[:,0],yp2[:,1])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 5: 15 pts\n",
    "\n",
    "Compute the label-based criteria we discussed in the Lesson for your amount-only classifier **using the test data**. Use a threshold of 0.5. Answer the questions in this text box below.\n",
    "\n",
    "* How many of the test instances are labeled positive by your classifier?\n",
    "\n",
    "**YOUR ANSWER HERE:** *1 test instance is labeled positive.*\n",
    "\n",
    "* Choose one of the positively-labeled test instances, and explain why the classifier labeled it positive.\n",
    "\n",
    "**YOUR ANSWER HERE:** *The postive-labeled example has a very large value for Amount. Because the coefficient for Amount is positive, the product of the two plus the intercept is greater than 0 (probability greater than 0.5) so the example is labeled positive.*\n",
    "\n",
    "* Is this classifier useful for finding fraudulent transactions? Explain in one or two sentences.\n",
    "\n",
    "**YOUR ANSWER HERE:** *Since the classifier has no ability to find positives it is not useful for identifying fraudulent transactions. ALSO ACCEPT: It might be useful if we used a different threshold.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tp: 0 tn: 142160 fp: 1 fn: 243\n",
      "Accuracy: 0.998 Recall: 0.0 Precision: 0.0 Sensitivity: 0.0 Specificity: 1.0\n",
      "Coefficient for Amount: [[0.00032136]]\n",
      "Value of amount for positive example: [[25691.16]]\n"
     ]
    }
   ],
   "source": [
    "# Calculate performance measures from scratch\n",
    "# TP: true postives \n",
    "# TN: true negatives \n",
    "# FP: False positives \n",
    "# FN: False negatives\n",
    "def compute_performance(yhat, y, classes):\n",
    "    # First, get tp, tn, fp, fn\n",
    "    tp = sum(np.logical_and(yhat == classes[1], y == classes[1]))\n",
    "    tn = sum(np.logical_and(yhat == classes[0], y == classes[0]))\n",
    "    fp = sum(np.logical_and(yhat == classes[1], y == classes[0]))\n",
    "    fn = sum(np.logical_and(yhat == classes[0], y == classes[1]))\n",
    "\n",
    "    print(f\"tp: {tp} tn: {tn} fp: {fp} fn: {fn}\")\n",
    "    \n",
    "    # Accuracy\n",
    "    acc = (tp + tn) / (tp + tn + fp + fn)\n",
    "    \n",
    "    # Precision\n",
    "    # \"Of the ones I labeled +, how many are actually +?\"\n",
    "    precision = tp / (tp + fp)\n",
    "    \n",
    "    # Recall\n",
    "    # \"Of all the + in the data, how many do I correctly label?\"\n",
    "    recall = tp / (tp + fn)    \n",
    "    \n",
    "    # Sensitivity\n",
    "    # \"Of all the + in the data, how many do I correctly label?\"\n",
    "    sensitivity = recall\n",
    "    \n",
    "    # Specificity\n",
    "    # \"Of all the - in the data, how many do I correctly label?\"\n",
    "    specificity = tn / (fp + tn)\n",
    "    \n",
    "    # Print results\n",
    "    \n",
    "    print(\"Accuracy:\",round(acc,3),\"Recall:\",round(recall,3),\"Precision:\",round(precision,3),\n",
    "          \"Sensitivity:\",round(sensitivity,3),\"Specificity:\",round(specificity,3))\n",
    "\n",
    "## Put code here to compute criteria:\n",
    "    \n",
    "ytest_hat = lr_amount.predict(Xtestamount)\n",
    "compute_performance(ytest_hat, ytest, lr_amount.classes_)\n",
    "\n",
    "## Put code here to compute whatever else you might need to answer the question.\n",
    "\n",
    "print(f\"Coefficient for Amount: {lr_amount.coef_}\")\n",
    "\n",
    "print(f\"Value of amount for positive example: {Xtestamount[ytest_hat > 0]}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 6: /10 pts\n",
    "Now fit a logistic regression model to the training data and include all the variables in the data frame in the cell below. You will want to make a new object like you did for the simpler model. Answer the following question.\n",
    "\n",
    "* According to this more complex model, are larger or smaller Amounts more strongly associated with fraud, if all other variables are held equal?\n",
    "\n",
    "**YOUR ANSWER HERE:** *Larger amounts are less associated with fraud according to this model if all other variables are held equal.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-5.97587364e-05,  4.47000284e-01, -3.90926504e-01,\n",
       "        -1.16855369e+00,  1.22331106e-01, -1.98752348e-01,\n",
       "        -6.01736445e-02,  9.12275938e-01, -4.89759298e-01,\n",
       "        -6.09455990e-01, -5.02506686e-01, -6.38380499e-01,\n",
       "         1.30082788e-01, -4.95462755e-01, -8.11838706e-01,\n",
       "        -9.72818723e-01, -4.57659961e-01, -4.98809560e-01,\n",
       "         1.60756429e-01, -2.52049691e-02,  2.50898085e-01,\n",
       "         6.56660801e-01,  6.58097834e-01,  7.25335261e-02,\n",
       "        -5.80101321e-02, -8.53895669e-01,  1.93601661e-01,\n",
       "        -1.23390755e-01,  1.28523891e-01, -6.35277593e-03]])"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Code for Question 6\n",
    "\n",
    "LOGREG = LogisticRegression(solver='lbfgs',penalty = 'none',max_iter=10000)\n",
    "lr_all = LOGREG.fit(Xtrain,ytrain)\n",
    "lr_all.coef_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Question 7: /15 pts\n",
    "\n",
    "In the cell below, Compute the label-based criteria we discussed in the Lesson for new classifier using the test data. (You don't have to copy the function down into this cell; just call it again here.) Use a threshold of 0.5. Answer the questions in this text box below.\n",
    "\n",
    "* How many of the test instances are labeled positive by your classifier?\n",
    "\n",
    "**YOUR ANSWER:** *tp + fp = 154+29 = 183 are labeled positive.*\n",
    "\n",
    "* Is this classifier **better or worse than the amount-only classifier** for finding fraudulent transactions? Explain in one or two sentences.*\n",
    "\n",
    "**YOUR ANSWER:** *Since the classifier has non-zero true positives, it is more useful than the previous classifier for finding fraudulent transactions.*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tp: 154 tn: 142133 fp: 28 fn: 89\n",
      "Accuracy: 0.999 Recall: 0.634 Precision: 0.846 Sensitivity: 0.634 Specificity: 1.0\n"
     ]
    }
   ],
   "source": [
    "# Code for Question 7\n",
    "\n",
    "ytest_hat_all = lr_all.predict(Xtest)\n",
    "compute_performance(ytest_hat_all, ytest, lr_all.classes_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question 8: /15 pts\n",
    "\n",
    "Plot ROC curves for both of your classifiers using the cells below, then answer the following questions, computing whatever quantities you need to answer them.\n",
    "\n",
    "* Which classifier has a higher estimated probability of correctly distinguishing between a positive and a negative instance? How do you know?\n",
    "\n",
    "**YOUR ANSWER:** *The AUROC gives the probability that a + is correctly distinguished from a -.*\n",
    "\n",
    "* How could you explain a result where a logistic regression classifier produces an AUROC that is \"worse than random\", i.e. less than 0.5, even on its training set?\n",
    "\n",
    "**YOUR ANSWER:** *Logistic regression optimizes likelihood, not AUROC. It is possible that a logistic regression classifier could perform worse than random in terms of AUROC.*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.42178603619387894"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAe7ElEQVR4nO3deXxV5b3v8c+PMM9DAjIFIpMKCq0RHCtgPaLXHtrT2qI9rbW1Squ2PZ72anvvae9pb++xfXWwrQOHeqjVDnZwQuvRagGxCAjIPIcokDAkjIEEEnbyu3/sbdyEJOzAXnvt4ft+vfIia60ne/8WgfXd61nPepa5OyIikrvahV2AiIiES0EgIpLjFAQiIjlOQSAikuMUBCIiOa592AW0VX5+vg8fPjzsMkREMsqKFSv2uXtBc9syLgiGDx/O8uXLwy5DRCSjmNn2lrapa0hEJMcpCEREcpyCQEQkxykIRERynIJARCTHBRYEZjbHzCrMbF0L283Mfm5mJWa2xsw+GFQtIiLSsiDPCB4HprWy/XpgVOzrDuDRAGsREZEWBBYE7r4QONBKk+nAEx61BOhtZgODqkdEJJM9+NoW/r51XyCvHeY1gsHAzrjlsti6U5jZHWa23MyWV1ZWpqQ4EZF0UV0b4Wd/28ry7a19tj5zYQaBNbOu2afkuPtsdy929+KCgmbvkBYRyVobd1fhDhcO7hXI64cZBGXA0LjlIcCukGoREUlba8sPAzAuC4NgLvDZ2OihS4HD7r47xHpERNLSuvIqCnp0YkDPzoG8fmCTzpnZ74HJQL6ZlQHfAToAuPss4CXgBqAEqAFuC6oWEZFMtq78MOMG9Qzs9QMLAne/+TTbHbgrqPcXEckGx+rq2VpxhOvGDgjsPXRnsYhIGtu0p4oGh7EBXR8ABYGISFpbF7tQHNSIIVAQiIiktXXlVfTt1pGBvYK5UAwKAhGRtLa2/DDjBvfCrLlbr5JDQSAikqYqjhxn454qiof1CfR9FAQiImnqp69uxR2uG3tOoO+jIBARSUNryg7x+7d20KNTe0YP6B7oeykIRETS0B+XR+fkfHDGhECvD4CCQEQkLZVUHAXg6tHBT7SpIBARSTPryg+zpPQA08aeQ/u84A/TCgIRkTTz8PwSAO6eOjIl76cgEBFJI6WVR3l5/R7umjIisGmnm1IQiIikkV++UUqHvHZ87vKilL2ngkBEJE1UVB3n6RXl3HTxEAp6dErZ+yoIRETSxJxF7xJpaOCLV52b0vdVEIiIpIGq4yf47ZLtXD9uIMPzu6X0vRUEIiJp4HdLd3CkNsLMq0ek/L0VBCIiIauN1DPn7+9wxch+XDgkNSOF4ikIRERC9uzb5VQcqQ3lbAAUBCIiodp/tJb/+5eNjB3UkytH5odSg4JARCREz64s52hthC9NHhH45HItaR/Ku4qI5Ij7/ryGZe8eaHH7gZo6AP7HhQNTVdIpFAQiIgGpOHKcPyzfyfihvSns27XFducP7BHa2QAoCEREArNgcyUA//GxC7lgUM+Qq2mZrhGIiARk/qYKzunZmfMH9gi7lFYpCEREAlAXaeCNrfuYcl5BqN0+iVAQiIgEYPm7BzhaG2HKmP5hl3JaCgIRkQDM21RBx7x2XBHSvQFtoSAQEQnAvM0VTDq3L906pf+YHAWBiEiSbd9fTWllNVPPS/9uIVAQiIgk3fxNFQAZcX0AAg4CM5tmZpvNrMTM7m9mey8ze8HMVpvZejO7Lch6RERSYd7mSs7N75by5wqcqcCCwMzygIeB64ELgJvN7IImze4CNrj7eGAy8GMz6xhUTSIiQaupi7CkdD9TMqRbCII9I5gIlLh7qbvXAU8B05u0caCHRQfZdgcOAJEAaxIRCdSikv3URRoy5voABBsEg4GdcctlsXXxHgLOB3YBa4GvuntD0xcyszvMbLmZLa+srAyqXhGRszZvUwXdO7XnkuF9wy4lYUEGQXO30nmT5euAVcAgYALwkJmdMiGHu89292J3Ly4oKEh+pSIiSeDuLNhcwZUj8+nYPnPG4gRZaRkwNG55CNFP/vFuA57xqBLgHeC8AGsSEQnMpj1H2H34eEZ1C0GwQbAMGGVmRbELwDOAuU3a7ACuATCzAcAYoDTAmkREAjMvNmx08pjM6rkI7JY3d4+Y2d3AK0AeMMfd15vZzNj2WcD3gMfNbC3RrqT73H1fUDWJiARp/qYKxg3uSf+encMupU0CvffZ3V8CXmqyblbc97uAfwiyBhGRVDhYXcfbOw5y95SRYZfSZplzNUNEJI0t3FpJg5NR9w+8R0EgIpIE8zZV0K9bR8YP6R12KW2mIBAROUv1Dc7rWyq5ekwB7dql90NomqMgEBE5S6t2HuRQzYmMGzb6HgWBiMhZmrepgrx2xlWjMmvY6HsUBCIiZ2nepkouHtaHXl06hF3KGVEQiIichd2Hj7Fxd1XGdguBgkBE5KzM3xSdCFNBICKSo+ZtqmBw7y6M6t897FLOmIJAROQM1UbqWVSyj6nn9Sf6WJXMpCAQETlDS0sPcOxEfUZ3C4GCQETkjM3bVEGn9u24bES/sEs5KwoCEZEz4O7M31zB5SP60blDXtjlnBUFgYjIGSjdV832/TUZ3y0ECgIRkTMyP/YQmkycbbQpBYGIyBmYt6mC0QO6M6RP17BLOWsKAhGRNjpy/ATL3j2QFWcDoCAQEWmzRSX7OFHvTB2jIBARyUnzNlXQo3N7Lh7WJ+xSkkJBICLSBg0NzvzNlXxodAHt87LjEJodeyEikiLrd1VReaQ2a7qFQEEgItIm8zZVYAaTx2TmQ2iaoyAQEWmD+ZsrGD+kN/26dwq7lKRREIiIJGj/0VpWlx3KiruJ4ykIREQStGBzJe6Z/RCa5igIREQSNG9zBf17dGLsoJ5hl5JUCgIRkQScqG9g4ZZKJo8pyOiH0DRHQSAikoAV2w9y5Hgk67qFQEEgIpKQ+Zsq6JBnXDkqe4aNvkdBICKSgPmbK5hY1JfundqHXUrSBRoEZjbNzDabWYmZ3d9Cm8lmtsrM1pvZ60HWIyJyJsoO1rBl71GmZNHdxPECizYzywMeBq4FyoBlZjbX3TfEtekNPAJMc/cdZpadf8siktF+u3QHkH3DRt8T5DnORKDE3UsBzOwpYDqwIa7NLcAz7r4DwN0rAqxHRKRVf1mzm7+XVJ6y/rmVuxjWryvnFnQPoargBRkEg4GdcctlwKQmbUYDHcxsAdAD+Jm7P9H0hczsDuAOgMLCwkCKFRH5fy9tpPzQMfr3OHn6iOH53fjFzR8IqargBRkEzQ209Wbe/2LgGqALsNjMlrj7lpN+yH02MBuguLi46WuIiLSo/NAxjtVFTtuu6niE8kPH+Mo1o7j32tEpqCx9BBkEZcDQuOUhwK5m2uxz92qg2swWAuOBLYiInKUnl2zn355b16afGVHQLaBq0leQQbAMGGVmRUA5MIPoNYF4zwMPmVl7oCPRrqOfBliTiOSAPYeP88Tid5mz6B0uH9GPmycm1qXctWMek7N0ZFBrAgsCd4+Y2d3AK0AeMMfd15vZzNj2We6+0cxeBtYADcBj7t62+BYRaeI/F27jV4ve5dyCbvz0UxMY0LNz2CWlNXPPrC734uJiX758edhliEgIaiP1HD52otU2j73xDrMXlvLh8wfw2K3FKaos/ZnZCndv9i8k+26RE5GsVHawhg//5HWOn2hIqP2/XDsq4Iqyh4JARDLC5x9fxvETDUwq6stHxg9qte3Eor6MHtAjRZVlPgWBiKSdl9ft5i9r95y0rrSymitH5vPrz08kr112TQMdNgWBiISuLvJ+d8+B6jpm/uZtAM7Nf38oZ1F+N748ZYRCIAAKAhEJ1f96dm3jXD7xfvjxi/jkJUOb+QlJNgWBiKRE5ZFa/rZxLw1xAxV3HTrWGALfuG5M4/reXTtwU/GQVJeYs9ocBLFZRWe4+28DqEdEstCS0v18+rGl1Dc0P1z9N1+YxJWj8lNclbynxSAws57AXUQnj5sLvArcDXwdWAUoCESkVd//ywbe3nGIFdsPAnDd2AF8d/q4k9p07pBHry4dwihPYlo7I3gSOAgsBm4HvkF0Gojp7r4qBbWJSIZ6aN5Wfva3rZyoj54BXDkyn+kTBnFTsfr801FrQXCuu18IYGaPAfuAQnc/kpLKRCRjzXq9lBP1zsyrR3DLxEIK+3UNuyRpRWtB0Hgft7vXm9k7CgEROZ3dh49xtDbC5y4fzv3Xnxd2OZKA1oJgvJlV8f5zBbrELbu79wy8OhHJOOvKqwD4xwmt3/0r6aPFIHD3vFQWIiKZ7WhthLpIAy+uiT52pKB7p9P8hKSL1kYNdQZmAiOJThM9x91P/5gfEckp7s5vl+7g28+vO+kegSF9uoRXlLRJa11DvyZ6neAN4AZgLPDVVBQlIpnjh69s5tEF2+jSIa/xmsC4wT0x01QQmaK1ILggbtTQfwFvpaYkEckUNXURHl2wDYDn775CM35mqERHDUWU7iLi7sQ/y+q+p9cC8N3pYxUCGay1IJgQGyUE0ZFCGjUkksMi9Q1c9O9/paau/qT1/bp15J8+qHmBMllrQbDa3T+QskpEJHCHa07w6sa9NLQw509rnltVTk1dPef07Nz4MPi8dnBT8VC6d9L8lZmstd9eZj3MWERatbR0P3f97m32Ha07q9eZ9/Wr6dpRB/5s0tpvs7+Z3dvSRnf/SQD1iEhAHppfwr6jdUwbew7/9pELzug1enXpoBDIQq39RvOA7rx/Z7GIZIDaSD0VVbWnrC8/dIzrxg5g1mcuDqEqSWetBcFud/9uyioRkaS488kVLNhc2ey2q0cXpLgayQStBYHOBEQyzJ7Dx3l9SyXTJwziqlEnH/QNuHqMgkBO1VoQXJOyKkRy0OOL3mHRtv1Jfc2KquO4w1evGcW5Bd2T+tqSvVqbdO5AKgsRyRW/+NtWfv/WDnYdPg7A+QOTe0vOzROHKgSkTXT5XyRF3J0HX4s+uQtgxiVDuXvqSIb00UNbJFwKApEUeX7VrsYQeOVrH2LMOZqSQdJDu7ALEMkF+4/W8rU/RB/1/cc7L1MISFrRGYFIEjy6YBsvrN7V4vY9VdHrAV+8qoiJRX1TVZZIQhQEkpO27j3CRx9eRM2J+tM3TsB7M3J++PwBzW4vyu/GzKtHcOGQXkl5P5FkCjQIzGwa8DOidyk/5u4PtNDuEmAJ8Cl3/3OQNYkAlO6rprqunpsnFlLQvWNSXnPauIFcMEiT8krmCSwIzCwPeBi4FigDlpnZXHff0Ey7HwCvBFWLSFPHY2cCX7yqSEMtJecFeUYwEShx91IAM3sKmA5saNLuHuBp4JIAaxEBojdcffOZtby7vxqALh3zQq5IJHxBBsFgYGfcchkwKb6BmQ0GPgZMpZUgMLM7gDsACgsLk16oZI/6Bmdv7MJsvOdWlTN7YSmHaqIP3hvcuws3XjSQgu6dUl2iSNoJMgiam6uo6TMOHgTuc/f61h6F6e6zgdkAxcXFek6CNOtYXT3/9OibbNxd1WKbWy8bRkGPTtw1ZaQeri4SE2QQlAFD45aHAE3H1xUDT8X+Q+YDN5hZxN2fC7AuyULryg/zpd+uYOeBY3Tu0I5//8exp7S5YGAvjdoRaUaQQbAMGGVmRUA5MAO4Jb6Buxe9972ZPQ68qBCQtjh87ATffWEDT79dBsC5+d2Ye8+VenSiSBsE9r/F3SNmdjfR0UB5wBx3X29mM2PbZwX13pI7Fm/bz9NvlzGsX1duvWw4t10xXF0+Im0U6Mcmd38JeKnJumYDwN0/F2QtkvmqayMn9f//58JSXt2wF4AnPz+Jwn6avE3kTOj8WdJaTV2Euat2UVffwI9e2UzV8cgpbX5803iG9u0SQnUi2UFBIGntxTW7uf+ZtY3Lw/t15XsfHde4PGZAD/r37BxGaSJZQ0EgaWlv1XG+8vuVbN9fQ147Y/E3p5JnRu+uHclrp2sAIsmkIJBAHayuY9/R2jb/3CMLtrH0nQOMKOjGTcUj6N9Dn/pFgqIgkEDsrTrO4m37G+fgP1Ov3Xu1RgGJBExBIEn32oa93P7E8sblKWMK+PjFQ9r8OsP7dVMIiKSAgkDOyqsb9vLE4ncbl0/UN7Ck9ADt2xmTx/Tn2zdewNC+XXRAF0ljCgJp1eGaE9RGTn14y+tbKvnBy5sb+/8/UNi7cdttVwznm9efT8f2ehKqSCZQEEiLnnprx0lDN5tzy6RCbrxwIJePzE9RVSKSbAoCadaikn2NIfCv146mbzNP8RrVv4eevyuSBRQE0qwf/3UzAA9+agIf/cDgkKsRkSApCAQAd6fs4DHufHIFx0/Us+NADZ8qHqoQEMkBCgKhocGZ8cslvPXOAQDOO6cHN1w4kM9cNizkykQkFRQEOay+wZm7upwXVu9uDIHvf2wct0ws1HBPkRyiIMgRr2+p5K/r95y0btXOQ6zf9f60zovun8rg3prFUyTXKAhywO7Dx7h1zlsA5DcZ/TN2UE8eu7WYnp070E1P9RLJSfqfn+VW7jjIxx55E4gOA73nmlEhVyQi6Ua3fma5h+eXAHD7lUV8afKIkKsRkXSkM4KsZ3TpkMf/vvGCsAsRkTSlM4IsV1MXYdzgnmGXISJpTEGQ5arr6unaUSd+ItIyBUGWq6mN0K1TXthliEgaUxBkuZq6erp00BmBiLRMQZDlqut0RiAirVMQZLH6BqemVtcIRKR1OkJkGXenpOIor27cyw9fjk4l3aOzfs0i0jIdIbLM86t28bU/rGpc/pcPj+amM3hwvIjkDgVBlli54yDPrSzn14u3A/Cjm8YzdlBPzh+oewhEpHUKgizxxSeWs+9oHb26dGD6hEF8QmcBIpIgBUEWcHf2Ha3jvHN68PLXPhR2OSKSYQIdNWRm08xss5mVmNn9zWz/tJmtiX29aWbjg6wn2xysruPldXu488kVAFw2ol/IFYlIJgrsjMDM8oCHgWuBMmCZmc119w1xzd4Brnb3g2Z2PTAbmBRUTdnkuZXlfP1Pq4k0eOO6f56kR0uKSNsF2TU0EShx91IAM3sKmA40BoG7vxnXfgmgju0EvLltX+PIoPFDevHAxy+if49O9OveKeTKRCQTBRkEg4GdcctltP5p/wvAfze3wczuAO4AKCwsTFZ9GSdS30B1bT23/HIpAC/ecyXjBvcKuSoRyXRBBkFzTz/3ZtZhZlOIBsGVzW1399lEu40oLi5u9jWy0dqywxypPdG4fOeTKzhyPAJA8bA+CgERSYogg6AMGBq3PATY1bSRmV0EPAZc7+77A6wn7a3fdZjF26J/BRt3H+Hpt8tOadOjc3u+cd0YPnLRoFSXJyJZKsggWAaMMrMioByYAdwS38DMCoFngM+4+5YAa0lLm/cc4bE3Snnvem9zB/6ffHI8g3p3AaKnWBcO6aW5g0QkqQI7orh7xMzuBl4B8oA57r7ezGbGts8Cvg30Ax4xM4CIuxcHVVO6+eYza3h7xyEGxw70Q/p04fNXFPGJ4ug184557ejcQTOHikiwAv1o6e4vAS81WTcr7vvbgduDrCEd1Tc4D762pTEEFt0/NeySRCSHqY8hxVZsP8gv5m1lweZKAH78Sd1DJyLhUhCk2K1z3uJobYSCHp343e2TGDWgR9gliUiOUxAELFLfwL/+aTV7q47jDkdrI3x0wiAe+PhF6v8XkbSgIEgyd2dP1fHGkUC3/eottuw9igGXFPXl8hH9mD5hsEJARNKGgiDJfvLqFn4xr+SkdZ3at+Olr1zFiP7dQ6pKRKRlCoIke+btcgB++PGLoisMpp7Xn3zNAyQiaUpBkESvbdhL+aFjXDSkF5+8ZOjpf0BEJA0oCJJg9c5D3PvHVWyrrAbg3mtHh1yRiEjiFARn6UB1Hc+uLGdbZTUfGT+Iq0fnM3lM/7DLEhFJmILgDMzfVMH2/dFP///nhejjFbp2zOPnMyYQmypDRCRjKAjaqC7SwO1PLKc+7slglwzvw/+cdp5CQEQykoKgjRZuqaS+wbntiuF8ZeooAHp37aAQEJGMpSBoo2XbDwBww4UD6dOtY8jViIicvXZhF5BpXt9cSfdO7blkeN+wSxERSQoFQRs8vugdNu05Qjv1AolIFlEQJKik4mjjCKEvTxkZcjUiIsmjawQJenTBNgB+98VJXD4iP+RqRESSR2cECXp2ZfR5wpcW9Qu5EhGR5FIQJOBwzQkaHC4e1od2ukAgIllGXUPNWLxtP79Zuh1i94wdrY0A8JHxA0OsSkQkGAqCOO/uq2bmb1awac8RAEbGPT/goiG9+MDQPmGVJiISGAVBnH/+r6WUHTzGpKK+TJ8wiFsmDQu7JBGRwCkIgP1Ha/n537ZSdvAYBd078Yc7Lwu7JBGRlNHFYmDW69v49eLtAPxsxoSQqxERSa2cPyP40/KdPP12Od065bH0Wx+me6ec/ysRkRyT82cEc1fv4kR9A1+4okghICI5KeeDYN/ROs4f2JN7/2FM2KWIiIQip4OgNlLPxt1V9OysMwERyV05HQS/WvQuAEP6dA23EBGREOXkR+GGBueRBSX86K9bAPj6deoWEpHclXNBUHmkls/OeYuNu6sA+P5Hx+kisYjktEC7hsxsmpltNrMSM7u/me1mZj+PbV9jZh8Msp4T9Q1c8cA8Nu6uomNeO578wkQ+fanuHhaR3BbYR2EzywMeBq4FyoBlZjbX3TfENbseGBX7mgQ8GvszEH9aXkZdfQOjB3Tnz1+6nJ6dOwT1ViIiGSPIM4KJQIm7l7p7HfAUML1Jm+nAEx61BOhtZoFM8fn6lkq+9exaDHjxnqsUAiIiMUF2jg8GdsYtl3Hqp/3m2gwGdsc3MrM7gDsACgsLz6iY7p3ac8OF53DFiHw6ts/pwVIiIicJMgiae4KLn0Eb3H02MBuguLj4lO2JuHhYHy4edvGZ/KiISFYL8qNxGTA0bnkIsOsM2oiISICCDIJlwCgzKzKzjsAMYG6TNnOBz8ZGD10KHHb33U1fSEREghNY15C7R8zsbuAVIA+Y4+7rzWxmbPss4CXgBqAEqAFuC6oeERFpXqB3Urn7S0QP9vHrZsV978BdQdYgIiKt0/AZEZEcpyAQEclxCgIRkRynIBARyXEWvV6bOcysEth+hj+eD+xLYjmZQPucG7TPueFs9nmYuxc0tyHjguBsmNlydy8Ou45U0j7nBu1zbghqn9U1JCKS4xQEIiI5LteCYHbYBYRA+5wbtM+5IZB9zqlrBCIicqpcOyMQEZEmFAQiIjkuK4PAzKaZ2WYzKzGz+5vZbmb289j2NWb2wTDqTKYE9vnTsX1dY2Zvmtn4MOpMptPtc1y7S8ys3sw+kcr6gpDIPpvZZDNbZWbrzez1VNeYbAn82+5lZi+Y2erYPmf0LMZmNsfMKsxsXQvbk3/8cves+iI65fU24FygI7AauKBJmxuA/yb6hLRLgaVh152Cfb4c6BP7/vpc2Oe4dvOIzoL7ibDrTsHvuTewASiMLfcPu+4U7PO3gB/Evi8ADgAdw679LPb5Q8AHgXUtbE/68SsbzwgmAiXuXurudcBTwPQmbaYDT3jUEqC3mQ1MdaFJdNp9dvc33f1gbHEJ0afBZbJEfs8A9wBPAxWpLC4giezzLcAz7r4DwN0zfb8T2WcHepiZAd2JBkEktWUmj7svJLoPLUn68Ssbg2AwsDNuuSy2rq1tMklb9+cLRD9RZLLT7rOZDQY+BswiOyTyex4N9DGzBWa2wsw+m7LqgpHIPj8EnE/0Mbdrga+6e0NqygtF0o9fgT6YJiTWzLqmY2QTaZNJEt4fM5tCNAiuDLSi4CWyzw8C97l7ffTDYsZLZJ/bAxcD1wBdgMVmtsTdtwRdXEAS2efrgFXAVGAE8KqZveHuVUEXF5KkH7+yMQjKgKFxy0OIflJoa5tMktD+mNlFwGPA9e6+P0W1BSWRfS4GnoqFQD5wg5lF3P251JSYdIn+297n7tVAtZktBMYDmRoEiezzbcADHu1ALzGzd4DzgLdSU2LKJf34lY1dQ8uAUWZWZGYdgRnA3CZt5gKfjV19vxQ47O67U11oEp12n82sEHgG+EwGfzqMd9p9dvcidx/u7sOBPwNfzuAQgMT+bT8PXGVm7c2sKzAJ2JjiOpMpkX3eQfQMCDMbAIwBSlNaZWol/fiVdWcE7h4xs7uBV4iOOJjj7uvNbGZs+yyiI0huAEqAGqKfKDJWgvv8baAf8EjsE3LEM3jmxgT3Oaskss/uvtHMXgbWAA3AY+7e7DDETJDg7/l7wONmtpZot8l97p6x01Ob2e+ByUC+mZUB3wE6QHDHL00xISKS47Kxa0hERNpAQSAikuMUBCIiOU5BICKS4xQEIiI5TkEgkqDYDKar4r6Gx2b6PGxmK81so5l9J9Y2fv0mM/tR2PWLtCTr7iMQCdAxd58Qv8LMhgNvuPuNZtYNWGVmL8Y2v7e+C7DSzJ5190WpLVnk9HRGIJIksWkdVhCd7yZ+/TGic+Fk8sSGksUUBCKJ6xLXLfRs041m1o/o/PDrm6zvA4wCFqamTJG2UdeQSOJO6RqKucrMVhKd0uGB2BQIk2Pr1xCd++YBd9+TwlpFEqYgEDl7b7j7jS2tN7PRwN9j1whWpbo4kdNR15BIwGKzvf4HcF/YtYg0R0EgkhqzgA+ZWVHYhYg0pdlHRURynM4IRERynIJARCTHKQhERHKcgkBEJMcpCEREcpyCQEQkxykIRERy3P8H1I1q4yxGKYoAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ROC for Amount-only classifier\n",
    "ytest_prob_amount = lr_amount.predict_proba(Xtestamount)\n",
    "fpr, tpr, _ = roc_curve(ytest, ytest_prob_amount[:,1], pos_label=1)\n",
    "ax=sns.lineplot(fpr,tpr)\n",
    "ax.set(xlabel=\"FPR\",ylabel=\"TPR\")\n",
    "auc(fpr,tpr)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.9275270491872326"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAYm0lEQVR4nO3de3Cc9X3v8fdXd1vyFck32UIy2AYngENkIBkgXJrGdkidNpmWkMI0JeOhDWnO6fQcaNOTTJszc+j9ciD1cYgPbZqpz6TQxM04Ick0iR3AATsxNjbFETK2hcCWr7Il67LS9/yxK7O7fiSv7H12tfp9XjMa9rlo9/sbmeezz+/3PL/H3B0REQlXWbELEBGR4lIQiIgETkEgIhI4BYGISOAUBCIigasodgHjVV9f783NzcUuQ0SkpOzcufOYuzdEbSu5IGhubmbHjh3FLkNEpKSY2cHRtqlrSEQkcAoCEZHAKQhERAKnIBARCZyCQEQkcLEFgZltNLOjZvbKKNvNzP7ezNrMbLeZ3RhXLSIiMro4zwieAlaNsX01sCT1sw74hxhrERGRUcR2H4G7bzWz5jF2WQv8kyfnwd5uZjPNbL67vxVXTSIipcLdeaH9ONvbT0DqcQGtzbO5fWnkPWGXpZg3lDUCh9OWO1LrLggCM1tH8qyBpqamghQnIlIIfYND/OPzb/CzQycz1u8/cpYDx3oAMEuue+gDV026ILCIdZFPyXH3DcAGgNbWVj1JR0RK0uDQMMPuJIacH7x6hG/s6GB3xym6+xIsbqilqvyd3vrqijJuX1rPhvtbqaksj7WuYgZBB7AobXkh0FmkWkREcnKku48j3X3j/r0fvHqU9T96nYGh4fPrGmdO4a5r5nDbkgY+9t6F+SxzXIoZBJuBh81sE3AzcFrjAyIykf3u13fynT1vR3dd5GDe9Bruf9+VAKxYNJPFDbXMnzElfwVeotiCwMz+BbgDqDezDuCLQCWAu68HtgBrgDagF/hUXLWIiOSib3CITS8eouPkuQu2HTvbz5Y9b/PRFQu45/oF5/vtx2P+jBqWL5iRh0rzK86rhj5xke0OfCauzxcRuZij3X088cM22lODsrsOneJMf4KpVeWRg5h3LmvgsY9dR01lyU3cPKbJ1RoRCU7vQIL+weGL75j+O4NDfGVrO1/bfhAc3tU4nYoyY+m8aaxdsYAH3tccT7ETlIJARGKzr7Ob0+cGL+s9HOdsX4KTvQOc6BlM/XeAN0+e4/Wusxw903/J7712xQJ+85YrWdk8+7JqLHUKAhG5LGf6Btnb2U3b0bMMDSeHUY/3DPBc2zF2Hjx5kd8ev5rKMq6orWZWbSW3L22gpb6W2qrxXV5ZVma8/6ormFZTydzpNXmvsdQoCEQC99bpc2ze1ckbx3vH9Xtn+gbZ19l9vn89XZnB1XPquHflIn5lxQIsssc9d9NqKphVW8XsqVVMSR30j3T36SCeJwoCkQmqPzHEoeO9vN51ln9/+S3eOn3hlSz58LNDpwCor6vCxnEpTE1lGdfOm84Hl8/llsVXcM38aVRXJA/SU6vKY78JSiGQPwoCCYq7c6S7n8Tw+AYX0w0NO6+9fYbdHad5ueMU+zq76U9c+vuNpncgQaqnhZlTKrluYTyXHd6+tIE1757HvTdp+pZQKQgkGF974Q3+z9b2yGvEL0V5mXFVQy2/dO1c6mry/79SXXUFLfW1tNTXMqu2kqbZtXn/DBFQEMgkkhga5vv7jnDgeA/u0NOf4OiZfrrO9HP4ZC/tXT2sbJ7Fg7e2UFt96f/0DVjcUMvy+TPo7htUF4WUPAWBTGinegfY8+ZpTvUOcvrcOz+negcu6I7ZdfgUB9MGPCvKjPq6auZMr2ZxfS0fWNrA59dcS0V5/h7DMWWcV6uITEQKApmQevoTbPzJAZ56/g2O9wxkbKupLGPGlEpqKjPv/pw7vYbf+cBVfPQ9jQBUlZdRVnZ5V6uIhEBBELij3X2cusQbftzhRM8AB4718MbxHk70DNDTn+Bs6qenP0FP/xBn+gbpHRhiyHOfqmtk15taZvPIjQtZ0TSTmVMqmZ4KABHJHwVBgH7afpydh07yrZ938tqRM3l5z+qKMmZOrWR6TSW11RVMq6lg7rQaaqsrqKsup7a6gopxfjtfMncaH7lhQV7qE5HRKQgCsq+zm8d/+Au27HkbgNYrZ/FHa66hcebUS37PaTUVLG6oZcGMKXSd7dfAqUgJUhAE4Fx/gv/9wza+sq2dmspyVr1rHr++ciF3XTM3r5+jEBApTQqCEnFuYOiSf/cLm/fyjZ0dfPj6+fzeXVezbN70PFYmIqVOQTBB9Q0O8Z9vn+HQiV7+eftBXjxw4rLe7+o5dTxx3415qk5EJhMFwQT16NO7+eau5COc6+uqeOB9V7Jg5qU90q6hrpoPvWtePssTkUlEQTABvXz4FJtf7uTX3tPIA+9v5rrGGRzTQKyIxERBUCTf2/c2X99+KHLys1fe7KauppLPf/harqirBjQQKyLxURAUwfNtx/idr/2M2XVVXDn7wks3VzbP4rN3LTkfAiIicVIQFNip3gH++9O7mVVbybP/5XZm11YVuyQRCZyCoMDu/+pP6Tx1jsfve49CQEQmBAVBgQwPO784eoY9b3bz4K3NrLlOUyeIyMSgIIjRv+44zEtvnGT/keT9ACOzaH7keoWAiEwcCoI8Onyil/U/fp03T53jZwdP0t2XAGDRrCm8a8F0ls6dxsrmWaxomlXkSkVE3qEgyINjZ/t56rk32PjcAfoHh1k2bxp3LJvD3OnV/PatLcybXjOuh4KLiBSSguAy9A4k+Pr2Q/y/HYdpO3qWFYtm8D8/eh3vboznIeMiInFQEIzDYGKIP392P2939wHQeeocOw+epLzMePjOq/mDDy0rcoUiIuOnIBiHjc+9wVe2tTNveg1TU8+q/diNjTz2seupzONzcEVECklBMIahYWd3xykOnejl0PFeNmxrZ8mcOr73X29Xn7+ITBoKglHsOnyKx77zKtvb35n+ef6MGv7u3hUKARGZVGINAjNbBfwdUA486e6PZW2fAfwz0JSq5S/d/f/GWdPF9A0k+Osf/IINW9sB+KVr5/LIqmUsnDWVKVV6aLqITD6xBYGZlQNPAB8EOoCXzGyzu+9L2+0zwD53/4iZNQCvmdnX3X0grrrGcvrcAA989UVe7jjNncsaePC2Fm69uqEYpYiIFEycZwQ3AW3u3g5gZpuAtUB6EDgwzZJ9LXXACSARY02RHn16N1t/0cXJnkH6Bof44w9fy6dvW1zoMkREiiLOIGgEDqctdwA3Z+3zOLAZ6ASmAb/h7hdM0G9m64B1AE1NTXkt8uCxHr6xo4Mhd37r/c3cc/18Wptn5/UzREQmsjiDIGpE1bOWPwTsAu4CrgK+b2bb3L0745fcNwAbAFpbW7Pf47I8/sM2htzZ8nu3snyBbgQTkfDEefF7B7AobXkhyW/+6T4FPONJbcAB4JoYa8qwdX8X39jZwa1X17Ns3vRCfayIyIQS5xnBS8ASM2sB3gTuBe7L2ucQcDewzczmAsuA9hhrAsDd+dymXXx7dydTq8r5m99YQXmZLgkVkTDFFgTunjCzh4FnSV4+utHd95rZQ6nt64EvAU+Z2R6SXUmPuPuxuGoC6E8M8eBTO/hJ2zFubpnNo6uvoWGaHgkpIuGK9T4Cd98CbMlatz7tdSfwy3HWkO17e4/wk7Zj3H9LE//jnuVUVejeABEJW3AT5Hx/3xGqK8r4/IcVAiIiEFgQ9PYn+PH+Lt7dOIOaSoWAiAgEFgSbdhzm9LlBPnf3kmKXIiIyYQQVBNv2dzF7aiW3LakvdikiIhNGUEFw6EQvLQ11mj1URCRNUEEwNOx6gIyISJagjop9g8NMrQqqySIiFxXUUbFvcIjaaj2LR0QkXXhBUKUgEBFJF0wQDA87fYlhnRGIiGQJJggGhpKPOZiiG8lERDIEFwRVFcE0WUQkJ8EcFQcTySCoVhCIiGQI5qh4vmuoSl1DIiLpggmCwUTyCZdVuqFMRCRDMEfFkTMC3VksIpIpmKPioAaLRUQiBXNUTAwlu4YqyzXhnIhIumCCwEkGgR5SLyKSKZggEBGRaMEFgc4HREQyBRME7sWuQERkYgomCEaUaYxARCRDMEHgOiUQEYkUTBCIiEi0YIJA5wMiItGCCYIRpuuGREQyhBMEOiUQEYkUTBAMF7sAEZEJKpggOE89QyIiGcILAhERyRBrEJjZKjN7zczazOzRUfa5w8x2mdleM/txnPWIiMiFKuJ6YzMrB54APgh0AC+Z2WZ335e2z0zgy8Aqdz9kZnPiqkc3lImIRIvzjOAmoM3d2919ANgErM3a5z7gGXc/BODuR2OsB9AQgYhItjiDoBE4nLbckVqXbikwy8x+ZGY7zeyBGOsREZEIsXUNEf3lO7t/pgJ4L3A3MAV4wcy2u/v+jDcyWwesA2hqaoqhVBGRcMV5RtABLEpbXgh0RuzzXXfvcfdjwFbghuw3cvcN7t7q7q0NDQ2XVIxGCEREosUZBC8BS8ysxcyqgHuBzVn7fAu4zcwqzGwqcDPwaow1YaZRAhGRdLF1Dbl7wsweBp4FyoGN7r7XzB5KbV/v7q+a2XeB3SRv/n3S3V+JqyYREblQnGMEuPsWYEvWuvVZy38B/EWcdYiIyOiCubNYtxGIiEQLJghGkkAjBCIimcIJAhERiaQgEBEJXDBBoCECEZFo4w4CMys3s0/GUUwh6DYCEZFMowaBmU03sz80s8fN7Jct6bNAO/DrhStRRETiNNZ9BF8DTgIvAJ8G/htQBax1910FqC2vNA21iEi0sYJgsbtfB2BmTwLHgCZ3P1OQymJiuoBURCTDWGMEgyMv3H0IOFDqISAiIhca64zgBjPr5p17sKakLbu7T4+9OhERid2oQeDu5YUsJG7DxS5ARGSCGjUIzKwGeAi4muTsoBvdPVGowuKiy0dFRDKNNUbwj0ArsAdYA/xVQSoSEZGCGmuMYHnaVUNfBV4sTEkiIlJIuV41VPJdQrqNQEQk2lhnBCtSVwlB8kqhSXHVkIYIREQyjRUEL7v7ewpWiYiIFMVYXUPqTBERCcBYZwRzzOz3R9vo7n8dQz2xceWaiEiksYKgHKhjsnWrT67WiIhctrGC4C13/9OCVSIiIkUx1hiBvjuLiARgrCC4u2BVFIKGCEREIo0aBO5+opCFFIqeRyAikimYh9eLiEi0YIJAPUMiItGCCYIRmoZaRCRTcEEgIiKZFAQiIoELJgg0DbWISLRYg8DMVpnZa2bWZmaPjrHfSjMbMrOPx1kP6C45EZFssQWBmZUDTwCrgeXAJ8xs+Sj7/RnwbFy1iIjI6OI8I7gJaHP3dncfADYBayP2+yzwNHA0xlpERGQUcQZBI3A4bbkjte48M2sEfhVYH2MdgKahFhEZTZxBENUdn300/lvgEXcfGvONzNaZ2Q4z29HV1XV5RelGAhGRDGNNQ325OoBFacsLgc6sfVqBTamDcz2wxswS7v7N9J3cfQOwAaC1tVVf7UVE8ijOIHgJWGJmLcCbwL3Afek7uHvLyGszewr4dnYIiIhIvGILAndPmNnDJK8GKgc2uvteM3sotT32cYHMegr5aSIipSPOMwLcfQuwJWtdZAC4+2/FWcsIDRGIiGQK5s5iERGJpiAQEQlcMEGgIQIRkWjBBMEIDRGIiGQKLghERCRTMEHgun5URCRSMEEwQpePiohkCi4IREQkk4JARCRwCgIRkcAFGAQaJBARSRdgEIiISDoFgYhI4IIJAt1GICISLZggGKH7CEREMgUXBCIikklBICISuGCCQGMEIiLRggmCERoiEBHJFFwQiIhIJgWBiEjgggkC18MqRUQiBRMEI0w3EoiIZAgmCHTVkIhItHCCoNgFiIhMUMEEwQh1DImIZAouCEREJJOCQEQkcOEEgQYJREQihRMEKbp6VEQkU3BBICIimRQEIiKBizUIzGyVmb1mZm1m9mjE9k+a2e7Uz/NmdkNctWiKCRGRaLEFgZmVA08Aq4HlwCfMbHnWbgeAD7j79cCXgA1x1XO+Lt1JICKSIc4zgpuANndvd/cBYBOwNn0Hd3/e3U+mFrcDC+MqRucDIiLR4gyCRuBw2nJHat1oHgS+E7XBzNaZ2Q4z29HV1ZXHEkVEJM4giOqDifxibmZ3kgyCR6K2u/sGd29199aGhoZLKkaTzomIRKuI8b07gEVpywuBzuydzOx64Elgtbsfj7Ge1OfF/QkiIqUlzjOCl4AlZtZiZlXAvcDm9B3MrAl4Brjf3ffHWIuIiIwitjMCd0+Y2cPAs0A5sNHd95rZQ6nt64EvAFcAX049MCbh7q1x1SQiIheKs2sId98CbMlatz7t9aeBT8dZw/nP0nVDIiKRwrmzWDkgIhIpnCAQEZFIwQSBLh8VEYkWTBCM0OWjIiKZggsCERHJpCAQEQlcMEGgIQIRkWjBBMEITUMtIpIpuCAQEZFMCgIRkcAFEwSuGwlERCIFEwQjdB+BiEim4IJAREQyKQhERAIXTBBohEBEJFowQTBCYwQiIpmCCwIREcmkIBARCVwwQaDbCEREogUTBCM015CISKbggkBERDIpCEREAhdcEOjyURGRTMEFgYiIZFIQiIgELpgguKKuihubZjK1qrzYpYiITCjBBMHK5tn8ya+8m4Wzpha7FBGRCSWYIBARkWgKAhGRwCkIREQCpyAQEQlcrEFgZqvM7DUzazOzRyO2m5n9fWr7bjO7Ma5aairLaWmojevtRURKVmxBYGblwBPAamA58AkzW56122pgSepnHfAPcdUDUFddEefbi4iUpDjPCG4C2ty93d0HgE3A2qx91gL/5EnbgZlmNj/GmkREJEucQdAIHE5b7kitG+8+mNk6M9thZju6urryXqiISMjiDIKo6d2yHw+Tyz64+wZ3b3X31oaGhrwUJyIiSXEGQQewKG15IdB5CfuIiEiM4gyCl4AlZtZiZlXAvcDmrH02Aw+krh66BTjt7m/FWJOIiGSJ7TIad0+Y2cPAs0A5sNHd95rZQ6nt64EtwBqgDegFPhVXPSIiEi3W6yndfQvJg336uvVprx34TJw1iIjI2HRnsYhI4Cz5pbx0mFkXcPASf70eOJbHckqB2hwGtTkMl9PmK9098rLLkguCy2FmO9y9tdh1FJLaHAa1OQxxtVldQyIigVMQiIgELrQg2FDsAopAbQ6D2hyGWNoc1BiBiIhcKLQzAhERyaIgEBEJ3KQMgon0ZLRCyaHNn0y1dbeZPW9mNxSjzny6WJvT9ltpZkNm9vFC1heHXNpsZneY2S4z22tmPy50jfmWw7/tGWb272b2cqrNJT1VjZltNLOjZvbKKNvzf/xy90n1Q3Jeo9eBxUAV8DKwPGufNcB3SE6DfQvw02LXXYA2vx+YlXq9OoQ2p+33HySnOvl4sesuwN95JrAPaEotzyl23QVo8x8Bf5Z63QCcAKqKXftltPl24EbglVG25/34NRnPCEJ8MtpF2+zuz7v7ydTidpJTfpeyXP7OAJ8FngaOFrK4mOTS5vuAZ9z9EIC7l3q7c2mzA9PMzIA6kkGQKGyZ+ePuW0m2YTR5P35NxiDI25PRSsh42/MgyW8UpeyibTazRuBXgfVMDrn8nZcCs8zsR2a208weKFh18cilzY8D15J8lske4HPuPlyY8ooi78evyfg097w9Ga2E5NweM7uTZBDcGmtF8culzX8LPOLuQ8kviyUvlzZXAO8F7gamAC+Y2XZ33x93cTHJpc0fAnYBdwFXAd83s23u3h13cUWS9+PXZAyCEJ+MllN7zOx64ElgtbsfL1Btccmlza3AplQI1ANrzCzh7t8sTIl5l+u/7WPu3gP0mNlW4AagVIMglzZ/CnjMkx3obWZ2ALgGeLEwJRZc3o9fk7FrKMQno120zWbWBDwD3F/C3w7TXbTN7t7i7s3u3gz8K/C7JRwCkNu/7W8Bt5lZhZlNBW4GXi1wnfmUS5sPkTwDwszmAsuA9oJWWVh5P35NujMCD/DJaDm2+QvAFcCXU9+QE17CMzfm2OZJJZc2u/urZvZdYDcwDDzp7pGXIZaCHP/OXwKeMrM9JLtNHnH3kp2e2sz+BbgDqDezDuCLQCXEd/zSFBMiIoGbjF1DIiIyDgoCEZHAKQhERAKnIBARCZyCQEQkcAoCkRylZjDdlfbTnJrp87SZ/dzMXjWzL6b2TV//n2b2l8WuX2Q0k+4+ApEYnXP3FekrzKwZ2Obu95hZLbDLzL6d2jyyfgrwczP7N3d/rrAli1yczghE8iQ1rcNOkvPdpK8/R3IunFKe2FAmMQWBSO6mpHUL/Vv2RjO7guT88Huz1s8ClgBbC1OmyPioa0gkdxd0DaXcZmY/Jzmlw2OpKRDuSK3fTXLum8fc/e0C1iqSMwWByOXb5u73jLbezJYCP0mNEewqdHEiF6OuIZGYpWZ7/V/AI8WuRSSKgkCkMNYDt5tZS7ELEcmm2UdFRAKnMwIRkcApCEREAqcgEBEJnIJARCRwCgIRkcApCEREAqcgEBEJ3P8HZctvgNOgb08AAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ROC for all-variable classifier\n",
    "ytest_prob = lr_all.predict_proba(Xtest)\n",
    "fpr, tpr, _ = roc_curve(ytest, ytest_prob[:,1], pos_label=1)\n",
    "ax=sns.lineplot(fpr,tpr)\n",
    "ax.set(xlabel=\"FPR\",ylabel=\"TPR\")\n",
    "auc(fpr,tpr)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Question 9: /15 pts\n",
    "\n",
    "Plot precision-recall curves for both of your classifiers using the cell below. Be sure to label your axes.\n",
    "\n",
    "* Which classifier is preferable if we want to recover at least 60% of fraudulent transactions?\n",
    "\n",
    "**YOUR ANSWER:** *The model with all variables has a precision of about 0.8 at 0.6 recall, and the amount-only classifier has precision almost 0 at 0.6 recall, so the model with all variables is preferable.*\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Text(0, 0.5, 'Precision'), Text(0.5, 0, 'Recall')]"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAY4AAAEGCAYAAABy53LJAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deZhdVZ3u8e+vhlPzmBoyD4RKICJoKBIgiDIpATU4daPY0EiLKOD03NvSdrdeH7Wb1uvElQZRQHHCCTFolGZQxCGQREIghCSVuTJWUqkhNQ+/+8felZxTqapzTqVODcn74amnztl7rV1rJeG8tfbae21zd0RERBKVNtYNEBGRiUXBISIiSVFwiIhIUhQcIiKSFAWHiIgkJWOsGzAaysrKfPbs2WPdDBGRCWXNmjUH3b28//ZTIjhmz57N6tWrx7oZIiITipntGGi7TlWJiEhSFBwiIpIUBYeIiCRFwSEiIklRcIiISFIUHCIikhQFh4iIJCWlwWFmV5rZRjOrMbM7BthvZnZXuH+dmS2M2veAmR0ws5f71Sk1syfMbHP4vSRV7Xd3unp6U3V4EZEJKWXBYWbpwN3AUmAB8F4zW9Cv2FKgKvy6Gbgnat93gSsHOPQdwFPuXgU8Fb5Piab2bvY1tqfq8CIiE1IqRxyLgBp33+runcDDwLJ+ZZYBD3lgJVBsZlMA3P2PQP0Ax10GfC98/T3gmpS0HsChVw+6EhGJkcrgmAbsinpfG25Ltkx/le6+FyD8XjFQITO72cxWm9nqurq6pBoeTU9IFBGJlcrgsAG29f8UTqTMsLj7fe5e7e7V5eXHrdGVxHFGojUiIiePVAZHLTAj6v10YM8wyvS3v+90Vvj9wAm2c0jKDRGRWKkMjlVAlZnNMbMIcC2wvF+Z5cD14dVV5wONfaehhrAcuCF8fQPwq5FstIiIDC1lweHu3cBtwOPABuCn7r7ezG4xs1vCYiuArUAN8G3gI331zezHwF+B+WZWa2Y3hbvuBK4ws83AFeH7lOnVkENEJEZKn8fh7isIwiF6271Rrx24dZC67x1k+yHgshFspoiIJEF3jsehq6pERGIpOEREJCkKjjg04hARiaXgiEOxISISS8ERh4JDRCSWgiMOnakSEYml4BARkaQoOOLQ5LiISCwFh4iIJEXBEYeWHBERiaXgEBGRpCg44tAch4hILAWHiIgkRcERh8YbIiKxFBxx6EyViEgsBYeIiCRFwRGH62SViEgMBYeIiCRFwRFHb+9Yt0BEZHxRcIiISFIUHHH0ao5DRCSGgkNERJKi4IhHAw4RkRgKjji0VpWISCwFRxyKDRGRWAqOOBQcIiKxFBwiIpIUBUccmuIQEYml4BARkaQoOOJwLTkiIhIjpcFhZlea2UYzqzGzOwbYb2Z2V7h/nZktjFfXzF5nZivNbK2ZrTazRansg4iIxEpZcJhZOnA3sBRYALzXzBb0K7YUqAq/bgbuSaDul4DPufvrgM+E70VEZJSkcsSxCKhx963u3gk8DCzrV2YZ8JAHVgLFZjYlTl0HCsPXRcCeFPYBx3UToIhIlIwUHnsasCvqfS2wOIEy0+LU/TjwuJn9X4Lgu3CgH25mNxOMYpg5c+bweiAiIsdJ5YjDBtjW/1f3wcoMVffDwCfcfQbwCeD+gX64u9/n7tXuXl1eXp5gkwf+oRpwiIgck8rgqAVmRL2fzvGnlQYrM1TdG4BHwtc/IzitJSIioySVwbEKqDKzOWYWAa4Flvcrsxy4Pry66nyg0d33xqm7B3hj+PpSYHMK+xCMOFL5A0REJpiUzXG4e7eZ3QY8DqQDD7j7ejO7Jdx/L7ACuAqoAVqBG4eqGx76g8A3zCwDaCecxxARkdGRyslx3H0FQThEb7s36rUDtyZaN9z+J+DckW3pELzvqqqBpl1ERE49unNcRESSouAQEZGkKDji0OS4iEgsBYeIiCRFwRGHu24AFBGJpuAQEZGkKDji8PA/EREJKDhERCQpCo54NMchIhJDwSEiIklRcIiISFIUHHHoLJWISCwFh4iIJEXBEY8mx0VEYig4REQkKQqOOHQDoIhILAWHiIgkRcERh6M5DhGRaAoOERFJioIjDj3ISUQkloJDRESSouAQEZGkKDjiCJ4AqJNVIiJ9FBwiIpIUBUc8rslxEZFoCg4REUmKgiMBmuIQETlGwSEiIklRcMThugNQRCSGgkNERJKS0uAwsyvNbKOZ1ZjZHQPsNzO7K9y/zswWJlLXzG4P9603sy+lsg8iIhIrI1UHNrN04G7gCqAWWGVmy939lahiS4Gq8GsxcA+weKi6ZnYJsAw42907zKwiVX0APY9DRKS/VI44FgE17r7V3TuBhwk+8KMtAx7ywEqg2MymxKn7YeBOd+8AcPcDKeyDiIj0k1BwmNkSM3vCzDaZ2VYz22ZmW+NUmwbsinpfG25LpMxQdecBbzCz58zsGTM7b5A232xmq81sdV1dXZymDs71zHERkRiJnqq6H/gEsAboSbCODbCt/0fwYGWGqpsBlADnA+cBPzWz07zfglLufh9wH0B1dbU++kVERkiiwdHo7r9N8ti1wIyo99OBPQmWiQxRtxZ4JAyK582sFygDhj+sGIKuxhURiZXoHMfvzezLZnaBmS3s+4pTZxVQZWZzzCwCXAss71dmOXB9eHXV+QQBtTdO3UeBSwHMbB5ByBxMsB8iInKCEh1xLA6/V0dtc8IP8IG4e7eZ3QY8DqQDD7j7ejO7Jdx/L7ACuAqoAVqBG4eqGx76AeABM3sZ6ARu6H+aamS5llUXEYmSUHC4+yXDObi7ryAIh+ht90a9duDWROuG2zuB9w+nPSIicuISvaqqyMy+2neVkpl9xcyKUt04EREZfxKd43gAaAb+LvxqAh5MVaPGE02Oi4jESnSOY667vyvq/efMbG0qGiQiIuNboiOONjO7qO+NmS0B2lLTpPFFNwCKiMRKdMTxYeB74byGAfXAP6aqUSIiMn4lelXVWuAcMysM3zeltFXjihY5FBGJNmRwmNn73f0HZvbJftsBcPevprBtIiIyDsUbceSF3wtS3ZDxSk8AFBGJNWRwuPu3wu+fG53miIjIeJfoDYBfMrNCM8s0s6fM7KCZ6e5tEZFTUKKX4745nBB/K8HqtPOA/52yVo0jOlMlIhIr0eDIDL9fBfzY3etT1J5xSYsciogck+h9HI+Z2asEN/19xMzKgfbUNWt8UW6IiByT0IjD3e8ALgCq3b0LaOH454eftBQcIiLHxLuP41J3f9rM3hm1LbrII6lq2HiiGwBFRI6Jd6rqjcDTwNsG2OecMsEhIiJ94t3H8dnw+42j05zxSaeqRESOSfQ+jv8ws+Ko9yVm9oXUNUtERMarRC/HXeruDX1v3P0wwaW5pwRdjisickyiwZFuZll9b8wsB8gaoryIiJykEr2P4wfAU2b2IMFc8QeA76WsVeOMxhsiIsck+jyOL5nZOuByggc5fd7dH09py8YRnakSETkm0REHwAag292fNLNcMytw9+ZUNWw8UXCIiByT6FVVHwR+Dnwr3DQNeDRVjRpvdAOgiMgxiU6O3wosAZoA3H0zUJGqRo03Cg4RkWMSDY4Od+/se2NmGZxCc8Y6VSUickyiwfGMmX0ayDGzK4CfAY+lrlkiIjJeJRocnwLqgJeADwErgH9LVaPGG404RESOiXtVlZmlAevc/Szg26lvkoiIjGdxRxzu3gu8aGYzR6E9IiIyziV6qmoKsN7MnjKz5X1f8SqZ2ZVmttHMaszsjgH2m5ndFe5fZ2YLk6j7v8zMzawswT4Mm85UiYgck+gNgJ9L9sBmlg7cDVwB1AKrzGy5u78SVWwpUBV+LQbuARbHq2tmM8J9O5NtV9L9QHMcIiLR4j0BMBu4BTidYGL8fnfvTvDYi4Aad98aHuthgsfNRgfHMuAhD5afXWlmxWY2BZgdp+7XgH8GfpVgW06IVscVETkm3qmq7wHVBKGxFPhKEseeBuyKel8bbkukzKB1zeztwG53fzGJtgybmekGQBGRKPFOVS1w99cCmNn9wPNJHNsG2Nb/E3iwMgNuN7Nc4F+BN8f94WY3AzcDzJx5YvP6ig0RkWPijTi6+l4kcYqqTy0wI+r9dGBPgmUG2z4XmENwldf2cPvfzGxy/x/u7ve5e7W7V5eXlyfZ9GMGSrDBuDtf+Z+NPPpCLb29vcP+mSIi41m8Ecc5ZtYUvjaCO8eb6Jszdi8cou4qoMrM5gC7gWuB9/Ursxy4LZzDWAw0uvteM6sbqK67rydqjawwPKrd/WACfR02TzADdta38v+ergHgm7/fwr9ffSYXzyvHLJn4EREZ34YMDndPH+6B3b3bzG4DHgfSgQfcfb2Z3RLuv5fgDvSrgBqgFbhxqLrDbcsJSeIzf+O+YJX56xbN5PFX9nHDg6s4Z3oR379pEYU5kRQ1UERkdCXzPI6kufsKgnCI3nZv1GsnWHk3oboDlJl94q2ML9E5jk37g+D4l6vP5DNvX8A9f9jC15/czHee3cYn3zw/dQ0UERlFKQ2Ok0XiwXGE6SU55GcFf6wfv3wev3/1AD9YuYOVWw9x5pRCPrfsrNQ1VERkFCR65/gpK5zMSajspv3NzKssiNl26yWnU9/axfPbD/Oj53fS0No5SG0RkYlBwZEAdzh0pIOe3sEDpKunly11R44LjisWVPKxy6r44jVn0dXj/Oi5lN/sLiKSUgqOOMwMd+erT2xi84HBH7G+41ALXT3O/Mn5x9X/xBXzuO78WVRV5PPwql2DHEFEZGJQcCSg7kgHP3xuJz95fvDRwsZ9RwCoqigYtMwb55Wzs76Vzm7d4yEiE5eCIwENrcF9kC/WNg5aZuP+ZtIMTq/IH7TM1OIcAA4e6RjZBoqIjCIFxxB+9PxOvvXMlqPBUXPgyKAT5Zv3NzN7Uh7ZmYPf+jK1OBuAumYFh4hMXAqOIexrbGNdbSMNbUFwNLV3U3u4bcCyGwe4oqq/KUXBiGNPw8DHEBGZCBQcQyjLz6K719nX2H502992Hj6uXHtXD9sPtjCvcvDTVAAVhVkA7GlMPjge/PM23v+dlXT3aH5ERMaWgmMI5QXBB/2u+lbSDNIM1u5qOK7c1roWeh3mTR56xFGWn4VBTBAl4vGX9/H5X7/Cn2oO8cSG/UnVFREZaQqOIZTnB8Gxs76V/KwMppXk8PLu4yfI+5YamR/nVFVmehqFOZnsb0o8ODbsbeLjP1nLrNI8SnIzefBP25LogYjIyFNwDKEsHHHsb2qnMCeTuWX5bD/Uely5dbUNZKQZs8vy4h6zODeTAwlOjrd2dnPjd1eRlZHGD/5pEde8bhqrth9m9+Hj2yAiMloUHEPoG3E4UJiTyfzJBdQ1d9DaGftokpXb6jmtPI/M9Ph/nJPyIhw6ktiyI1/63avsa2zn7usWMq0kl/efPxMHfr1ub7JdEREZMQqOIeRlZZCVEfwRleRksmBK8PiRTfuO3UHe3tXD5v3NnDurJKFjlhdkcTiB9ape2dPED1bu5C2vmcyS08sAmFtRwOTCbJ7SPIeIjCEFRxzFuZkATMqPHA2O9Xubju5fu7OBrh7nDVWJPWWwojCbhtauAe8H6e7pZcehFtbuPMwHH1pNflYGX7gmdjXdC0+fxNraRtq7eobbJRGRE6Jl1eMoyslkf1MHpXlZzC7PIz3NeDUqOP64uQ6AC06blNDxphRm093rHG7tojTv2MOdthw4wnu+9VfqW4LRSJrBQx9YdPTKrj5XvmYyj/xtN//yi3UU50Z4zbRC3rVwOp09vXziJ2spzM7kznedfaLdFhEZlIIjjqKcYMRRnp9FZnoa04qzjz7pD+C5rfXMLM2lJC+xJ/xNKQruHt/f1H40OBpaO3n//c/R1dPLZ9+2gLL8LOaW57FgatFx9d9QVU5+Vga/XLsnWPIdWLW9nvW7m3h5TxBof189g9cneOpMRCRZCo44ivuCozD4kJ9Tls8r4YijrbOHl3Y38rZzpiR8vL67yzfsbeLMKYW4Ox97eC11zR18/6bFXDB36JFLTiSdlZ++jHQzIhlp3PrDNfxkVS15Wen829VnctdTm/k/j63nV7ddNJzuiojEpeCIo2+OoywvOGW0aHYpz2yqY0vdETbvb6azp5erz048OKoqC8hIM9bsOMw7F07np6t38cymOm6/9PS4odGn7wmDAN9830Ke3XyQRXNKycvKoK2zh688sYnV2+upnl2aRE9FRBKjyfE4SsPAqCwMTjG9NRxdLF+7hxUv7SM7M+3oVU+JiGSkMWtSLi/vbuRIRzd3/vZV5lXm8/HL5w2rfRnpaVxyRgV5YZjceNEcciPp3PXU5mEdT0QkHgVHHK+fWcw/XTSHBVODK6pmTcpjTlkej724hz/VHKR6VglZGYOviDuQM6cUsqWuhTt/+yqHW7u4851nk55mI9Le/KwM3n3udP5Uc5AtQzx4SkRkuBQccWSmp3H5mRWYHftgv2JBBVsPtnC4tZPrFs9K+pjnzCjmSEc3P3puB0vPmszCEZ7Ivu3S08nOTOfjP1lL7xCPuxURGQ4FRxzukGaxo4GblpzGO18/jUc/soSlr018fqPPeWFQRDLS+NzbXzMi7YxWUZDNp66cz0u7m/jP37466DNERESGQ5PjcThOWr/TSJVF2Xz171837GOeMaWQsvwI/3TRaVSEcycj7foLZvOXmkN8+9mt/HzNLnod3rVwGlkZ6fxu/T5K8yJ8/6ZF5Eb0T0BEkmOnwm+j1dXVvnr16qTrNbZ28dy2Q0zKy+Lc2SN7OsndY05/pUJvr/P1Jzfx0u5GjrR3s2pH8CyR2ZNy2VnfytnTi3nkwxceF4wiIgBmtsbdq/tv16+bcTiQloITeqkODYC0NOOTb55/9H19SyeRjDTyszL41jNb+M/fvsqftxxMeLmUkdbd00tGAgtDisj4ouBIQP85jokqeomTGy6czV1Pb+a7f9k+asGxv6mdu39fw5Mb9tPQ2kVrZw9LTp/Et6+v1ikzkQlE/7fG0evHz3GcDLIz03nLgsk8tm4PDa2dFOcmtmTKcGzY08Q3ntrMkxv20+vOubNKuHBuGQb84m+1XPutlTx665KT8s9Z5GSk4IjHT95Lzz5w0RweeWE3D6/axS1vnDuix3Z3/rCxjm8+vZk1OxvIykjjmtdP4/ZLT2fWpGMPvDprWhGfXb6e+/+8jQ++4bQRbYOIpIaCI45UzXGMB2dNK+L0inx+/PxOPnTxaSMy79LV08sv/lbLt57ZyraDLRTnZHLrm+bywYtPG3BUc/0Fs/jlC7V8+Xcb+fWLeyjJi3DpGRX8/Xkz2NPQzub9zWytO8LlCyo5vWLoR/OKyOhQcMTh7qSfrMkBXHveDL7wmw2s3dXA62cO/8qxpvYuHvzTNh5auYNDRzqZUZrD5685i7+rnj7knfVmxj3Xnct/rNjAq/ua2dPQxB821vGZX62PKXfvM1tZ8bGLmFqcO+w2isjISOnluGZ2JfANIB34jrvf2W+/hfuvAlqBf3T3vw1V18y+DLwN6AS2ADe6e8NQ7TiRy3GfraljfmUBVZUn52+7ze1dnPeFJ5k/uYBHPrKE9DSjpaObbQdbmF2Wd3RBxca2Lh7663YOtwTzIaV5ESblRSjLj/DIC7t59IU9tHX1cM70Im699HQuP6NyWHMW7s6v1u5h7a4GqirzOaOyAAfef/9zzCjJ5bHbLyI7M7klXkRkeAa7HDdlwWFm6cAm4AqgFlgFvNfdX4kqcxVwO0FwLAa+4e6Lh6prZm8Gnnb3bjP7LwB3/9RQbTnR4DhjcsFJfZrkob9s5zPL17NoTinuzpodh+n14Fkk76meTld3Lz9bU0trZ8/RZ4BESzO4ZH4FH7usirNnFKekjY+9uIfbf/wCV712Mne/b+GoXM4scqobi/s4FgE17r41bMDDwDLglagyy4CHPEivlWZWbGZTgNmD1XX3/4mqvxJ4dwr7AEC6nbynqgD+4YJZbNzfzIqX9hLJSOO9i2ZyzvRivr9yO995dhsGXFRVxj+/5QwWTC3kSHs3Te1dHG7tZG9jO2dNK2JacU5K2/i2c6by4q4GvvOnbVz/wPP893ULKcjOTOnPFJGBpTI4pgG7ot7XEowq4pWZlmBdgA8APxnoh5vZzcDNADNnzkym3cc5iac4gGCe4YvveC1ffMdrY7b/3XkzONLRTVtnT8wjbItyMynKzWRGaS5nTx+9dv7r1WdSlJPJ157cxFu+9kduXDKHC+aWcta01IxyRGRgqfxIHOhcQv+zHIOViVvXzP4V6AZ+ONAPd/f73L3a3avLy0/sBreT5QbA4cjPyjjuuedjxcy4/bIqvn/TYtq6evjiig28856/sr+pfaybJnJKSWVw1AIzot5PB/YkWGbIumZ2A/BW4DofhcW2dF/a+LLk9DKe+/Tl/Pr2JfT2Ol/8zYaxbpLIKSWVwbEKqDKzOWYWAa4Flvcrsxy43gLnA43uvneouuHVVp8C3u7urSls/1Gn8ohjvIpkpHHWtGLe8fpp/HrdHjbtaxrrJomcMlIWHO7eDdwGPA5sAH7q7uvN7BYzuyUstgLYCtQA3wY+MlTdsM43gQLgCTNba2b3pqoPEJwzG6mn88nIu2PpGWRnpvPZ5a/g7nr2iAjQ1tnDjkMtrN5eT2Nr14gfX8uqD6GxtYs/b6mjenYpFQWpeW6GnLgvP/4qd/9+C9kZabR395JmkBvJ4OJ5ZXz00irmVRZQe7iNHYda6HXHLHgmiv5OZSLp7XUa2ro40NxOXXMHB5o6ONDcEbwOtwWvOzjS0X203oM3nscl8yuG9TO1rPqwGTbgXL2MF7ddUsWOQ61kZ6ZTnJNJd6+zu6GNp189wG9f2kckI42O7t6YOulpxqVnVPDvV5/JzEl59PQ6exraeHl3IzvqW/nAkjlEMk7yy+lkTLk7Te3d1Ld0Ut/SSV1zO/ub2qlv6aKl89j2YF8HB4900NVz/C/6uZF0KgqyKC/I4swphVw8L3jdt+2c6SN/1aGCIw6z4EvGr5xIOt9838Ljth9u6eSbv99MU1s3C6YWMr+ygMx0o7vXefSFPfxy7W6eeGU/k/IiNHd00xkVLk1tXfzzlWeMZjdkguvu6aW+tTPmAz/661BLJ4f7ve7uHfiMT1ZGGpPyIpTmRyjNy6KqooCKwizK87OoKMyioiD7aDjkZY3+x7iCIw7DNDk+QZXkRfj3tw78TPcL5pbx8SuqeOgvO9hR38LU4hzmluczf3IBX39yE99+divvPnc6p5Xnj3KrZSz19DqHWjrY19jOvsZ29jd3sLehjf1N7Rxo7uBgcweHW7vodSc3kk5eVgatnT3Ut3TS2Db4XEJRTialecFSPTNKczlnejGl+RGKcjIoyMqksjCbisIsSnIjTMqPjPvn02iOYwiNrV2s3HqIRXNKKclL3fMqZHzZ39jGJV95hnkV+fzy1iVa3uQk0NLRTV1zB/sa29jX1MHexnb2Nbaxv7mDuqYODrV00NDWRWNbFwN9JOZG0inOzaQ0/GDPTE/jSEc37lBekHU0FCblRSg5+jqLkrxMSnKD8hOR5jiGSaeqTj2VRTl88op5fOE3G/j2s1u5+eLgWSUd3T00tHbR0NrFpPwIZfnj48bIU1VPr1Pf0smBpnb2NraztykMg6YODjS1c/BIcFqooa2T9q7e4+qnGRTnRijOzaS8IIvXTC2ioiCLKUXZVBZlM6Uom8mF2ZQXZJMT0cKa0RQccZgmx09JH1gyh9+8tJf/WPEqP35+F01tXRxq6Ty6PzszjZ/fciFnTSsaw1aenFo7g9HB/qZ29jQEp4z2NrVzoCm4cuhQSycNrcGpoYGmCHIjwUUSJXkR5k/Op6LgWBBMKcymsjCbyUXZlORG9NTJYVJwxGEGJ/kahzKAtDTjpx+6gK89sYlnNx9kXmU+M0tzmVKUQ3FOJl9YsYEbH1zFRy+v4u3nTKUoRwsuDqW316lv7eRAUwf7mtqOBsK+MAwOHukIRgetXbR19RxXPxgdZFKcEyzlf+aUAioKwlFBUTZTinLC0UGWRgejQHMcQ2hs7WLVjnoWzynVSqwS44Udh7nlh2vY39RBRppx7qwS3nbOVOZVFnCguZ3tB1uYW57PlWdNHtYcibvT3NFNa0cPuVnp5EcyxuVvx22dPeHooI29Te3sbQhOG/XdV9B39dBgo4OczHRKcoPRQVl+cJVQZWE2U4qPnSqqKAxGB7oRd/RpjmOYgjkO/YOVWK+fVcLKf7mMF3Y28NPVu3jilf3826MvH1fu7OlFfPGa1zK7LJe8SAa97kev1Kk93EZXT3DuvaO7l12HW9l5qJXth1rZWd9CS8ex37wNjl7Fk5eVwaS8CFkZaUQy0ijOjZAbSedwayczSnO5ZF4FC6YV0tzeTWNrF2UFEcrzsxL+d9w3Oug7XbS7oY29je3sb2yn7khwZVF9azA6aO0ceHRQlJN59Aqh+ZUFwemhwiymFOcEcwjh6GC8Xz0kA9OIYwiNrV2s2VnP4jmTxuRaaZk4enudbYda2FnfyuTCbKYW5/CTVTu566mao3fxpqcZ7j7gb94AkfQ0ppfmMKs0l1mT8sLLMtNp7eihuaObxrYuGlo6OdzWxcEjHfT0OJ09vUeXvs/Lyhj0ktCC7AxmleYyc1IuTW3dZGYYZXlZ5Galc7A5CIn6lg7qW7toaO0csI3ZmWmU5kYozguCqO8+gqlhGEwpyjl6hZFGBycHjTiGKQ3TVVUSV1qaMbc8n7lR933cfPFc3n3uDH62ehetnT0cbu0kI82YEQbDlKLso4/BzcpIo7Iw+4Q/cA+3dPKXLYfYuK+JkrwIJbkRdje08cqeJmoONPPnmkMUZGfQ3eM0tB2is7s3GB3kRSjNjXB6ZQGVhVlMLgyCYGpxMDooyx+bG81kfNK/hHhMq+PK8JXmRfjQG+eO2s8ryYtw9dlTuPrsKQmVd3edipWk6XqhOPS/lJzMFBoyHAqOeHQDoIhIDAVHHLoBUEQkloIjjjTTo2NFRKIpOOIx03lgEZEoCo44DE2Qi4hEU3DEYWhyXEQkmoIjDtOpKhGRGAqOONL0JyQiEkMfi3HoD0hEJI+3kXkAAAceSURBVJY+F+MwXYsrIhJDwRFHuuY3RERiKDji0AKHIiKxFBxx6EyViEgsBUccuhRXRCSWgiMOjThERGIpOOJIU3KIiMRQcMShq6pERGKlNDjM7Eoz22hmNWZ2xwD7zczuCvevM7OF8eqaWamZPWFmm8PvJSnuQyoPLyIy4aQsOMwsHbgbWAosAN5rZgv6FVsKVIVfNwP3JFD3DuApd68Cngrfp4zOVImIxErliGMRUOPuW929E3gYWNavzDLgIQ+sBIrNbEqcusuA74Wvvwdck8I+kK7kEBGJkcrgmAbsinpfG25LpMxQdSvdfS9A+L1ioB9uZjeb2WozW11XVzesDuRnZ1BZmD2suiIiJ6tUBsdAv6p7gmUSqTskd7/P3avdvbq8vDyZqkelpxnZmenDqisicrJKZXDUAjOi3k8H9iRYZqi6+8PTWYTfD4xgm0VEJI5UBscqoMrM5phZBLgWWN6vzHLg+vDqqvOBxvD001B1lwM3hK9vAH6Vwj6IiEg/Gak6sLt3m9ltwONAOvCAu683s1vC/fcCK4CrgBqgFbhxqLrhoe8EfmpmNwE7gfekqg8iInI8c09q6mBCqq6u9tWrV491M0REJhQzW+Pu1f23685xERFJioJDRESSouAQEZGkKDhERCQpp8TkuJnVATuGWb0MODiCzZkI1OdTg/p8ajiRPs9y9+PuoD4lguNEmNnqga4qOJmpz6cG9fnUkIo+61SViIgkRcEhIiJJUXDEd99YN2AMqM+nBvX51DDifdYch4iIJEUjDhERSYqCQ0REkqLgCJnZlWa20cxqzOy455iHS7/fFe5fZ2YLx6KdIymBPl8X9nWdmf3FzM4Zi3aOpHh9jip3npn1mNm7R7N9Iy2R/prZm8xsrZmtN7NnRruNIy2Bf9dFZvaYmb0Y9vnGsWjnSDKzB8zsgJm9PMj+kf38cvdT/otg6fYtwGlABHgRWNCvzFXAbwmeTng+8NxYt3sU+nwhUBK+Xnoq9Dmq3NMEy/6/e6zbneK/42LgFWBm+L5irNs9Cn3+NPBf4etyoB6IjHXbT7DfFwMLgZcH2T+in18acQQWATXuvtXdO4GHgWX9yiwDHvLASqC470mEE1TcPrv7X9z9cPh2JcGTGCeyRP6eAW4HfsHEf7pkIv19H/CIu+8EcPdToc8OFJiZAfkEwdE9us0cWe7+R4J+DGZEP78UHIFpwK6o97XhtmTLTCTJ9ucmgt9YJrK4fTazacA7gHtHsV2pksjf8TygxMz+YGZrzOz6UWtdaiTS528CZxI8jvol4GPu3js6zRszI/r5lbInAE4wNsC2/tcpJ1JmIkm4P2Z2CUFwXJTSFqVeIn3+OvApd+8JfiGd0BLpbwZwLnAZkAP81cxWuvumVDcuRRLp81uAtcClwFzgCTN71t2bUt24MTSin18KjkAtMCPq/XSC30aSLTORJNQfMzsb+A6w1N0PjVLbUiWRPlcDD4ehUQZcZWbd7v7o6DRxRCX67/qgu7cALWb2R+AcYKIGRyJ9vhG404OT/zVmtg04A3h+dJo4Jkb080unqgKrgCozm2NmEeBaYHm/MsuB68OrE84HGt1972g3dATF7bOZzQQeAf5hAv8GGi1un919jrvPdvfZwM+Bj0zQ0IDE/l3/CniDmWWYWS6wGNgwyu0cSYn0eSfBCAszqwTmA1tHtZWjb0Q/vzTiANy928xuAx4nuCrjAXdfb2a3hPvvJbjC5iqgBmgl+K1lwkqwz58BJgH/Hf4G3u0TeGXRBPt80kikv+6+wcx+B6wDeoHvuPuAl3ROBAn+HX8e+K6ZvURwCudT7j6hl1o3sx8DbwLKzKwW+CyQCan5/NKSIyIikhSdqhIRkaQoOEREJCkKDhERSYqCQ0REkqLgEBGRpCg4REZAuJLuWjN7OVx5tXiEj7/dzMrC10dG8tgiyVJwiIyMNnd/nbufRbDY3K1j3SCRVFFwiIy8vxIuIGdmc83sd+ECgs+a2Rnh9koz+2X4TIgXzezCcPujYdn1ZnbzGPZBZFC6c1xkBJlZOsFyFveHm+4DbnH3zWa2GPhvgsX17gKecfd3hHXyw/IfcPd6M8sBVpnZL06CNcLkJKPgEBkZOWa2FpgNrCFYcTWf4GFYP4taaTcr/H4pcD2Au/cAjeH2j5rZO8LXM4AqQMEh44qCQ2RktLn768ysCPg1wRzHd4EGd39dIgcwszcBlwMXuHurmf0ByE5Nc0WGT3McIiPI3RuBjwL/C2gDtpnZe+Doc5/7ntv+FPDhcHu6mRUCRcDhMDTOIHjEp8i4o+AQGWHu/gLBs66vBa4DbjKzF4H1HHuM6ceAS8IVWtcArwF+B2SY2TqCFVxXjnbbRRKh1XFFRCQpGnGIiEhSFBwiIpIUBYeIiCRFwSEiIklRcIiISFIUHCIikhQFh4iIJOX/A7EzBWqEtuYeAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# ROC for Amount-only classifier\n",
    "ytest_prob_amount = lr_amount.predict_proba(Xtestamount)\n",
    "precision, recall, _ = precision_recall_curve(ytest, ytest_prob_amount[:,1], pos_label=1)\n",
    "ax=sns.lineplot(recall,precision)\n",
    "ax.set(xlabel=\"Recall\",ylabel=\"Precision\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Text(0, 0.5, 'Precision'), Text(0.5, 0, 'Recall')]"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEGCAYAAABo25JHAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjIsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+WH4yJAAAgAElEQVR4nO3deXxddZ3/8dfn3ux7mr1plqZ7KaWlAVpAFpFSEAcRUBTFBQdxQH2Mo6OP3++n/ubh/Oah4jiog2IHQRwcGVGEOgJlp2zFtnShC23TdEu6ZGnSNEmb9fv7497WUJLmps3JuTf3/Xw88si955x7+vmScN4553y/32POOUREJH4F/C5ARET8pSAQEYlzCgIRkTinIBARiXMKAhGROJfgdwEjlZ+f7yorK/0uQ0QkpqxZs6bJOVcw2LqYC4LKykpWr17tdxkiIjHFzHYPtU6XhkRE4pyCQEQkzikIRETinIJARCTOKQhEROKcZ0FgZg+YWYOZbRxivZnZT8ysxsw2mNm5XtUiIiJD8/KM4FfAklOsvxqYFv66Hfi5h7WIiMgQPBtH4JxbYWaVp9jkOuDXLjQP9kozyzGzEufcfi/q2XrgCH/esM+LXQNw6YxCFlTkerZ/ERGv+DmgrBTYO+B9XXjZe4LAzG4ndNZAeXn5af1jNQ3t/PTFmtP67HCcgxe2NvDLT59HUVaKJ/+GiIhX/AwCG2TZoE/Jcc4tBZYCVFdXn9aTdD44t4QPzv3g6Xx0WPc8t40fP7+d2oYOBYGIxBw/ew3VAWUD3k8CvLt246FFVXk4Bxv3Hfa7FBGREfMzCJYBt4Z7Dy0EDnt1f8Br88pzSE4I8Ha9gkBEYo9nl4bM7LfAZUC+mdUB3wESAZxz9wFPAtcANUAn8FmvavFackKQ6spcNtS1+l2KiMiIedlr6OPDrHfAnV79+2Ptwin53L18K41HuijITPa7HBGRiGlk8Si5ZFpomu9Xaxp9rkREZGQUBKPkrIlZZKcm8vLWvwbBoY5unlhXzzf/sIEl96zg6Y0HfKxQRGRwMfdgmmgVCBjzy3J4ZXsTv3x1J89sOsCqXYfod5CZkkAwYPzk+e3MK8umODvV73JFRE7QGcEoWlCRS3NHN9/9n80cPtrDXe+fxrK7LmLdtxfztcUz2Ly/jVe2NfldpojIu+iMYBR9ZEEpFfnpnDMpm4q89Hetu35+Kd976h3+/PZ+bjqvbIg9iIiMPQXBKCrNSaM0J23QdenJCdxwbim/eXMPTe1d5GeoZ5GIRAddGhpDn1xYQW+/46HXd7Fm9yFCPWhFRPylIBhD04oyObs0m5++UMMNP3+DJ9bt42DbMb/LEpE4pyAYY/+weDq3XTyZ6UUZ/Pj57exvHToIjvX0saGuVWcOIuIp3SMYY5fNKOSyGYU8s+kAt//nGn7z5m7mTsomEAhNxtrV28eKbU38ecM+ntvSQHtXL9+6djbXzi3RzKYi4gkFgU+unF3EdfMm8uiaOupbj3LX5VN5etMBHl9bT9uxXnLSErl2bgl7DnXy/affoTgrhQ/OLXnXPmob28lITqBQASEiZ0BB4BMz456PzaNiQhr/8cpOPnH/myQlBLh6TjEfnl/KxVPzSQwGaDzSxVX3rODfnt3G4rOK2NHYzpNvH+Cpt/ezvaGdc8pyeOLOi/xujojEMIu168/V1dVu9erVfpcxat6uO0xmSgLr61q5bHoh2WmJ79nm6Y0HuOPhNeSlJ9Hc0Y0ZnF85geTEIK9sb2TdtxYP+jkRkePMbI1zrnqwdToj8FlhVjJFWSlU5qcPuc2SOcXccG4pDUe6WDKnmMWziynITObN2mZWbGtk+eYDfLRag9RE5PQoCHwW6Q3gf1wy8z3bHn8gzqvbmxQEInLa1H00RgwWGMkJQRZU5LJRT0YTkTOgIIhxi6ry2NnUQWtnt9+liEiMUhDEuIVT8nDAmzsP+V2KiMQo3SOIcXMnZZOUEOCNHc1cdVax3+XENeccTe3d7GruYGdTB7ubO8hOTeSy6YVML870uzyRISkIYlxyQpDZJVmsrG32u5S40dXbx86mDmoa2qlpaGdHYwc7m9rZ1dRJe1fvie2CAaOv33H38q18ffEM/vaSKszMx8pFBqcgGAfmlGbz8Mrd3HTf61w+s5CPzC/VU9BGQduxnr8e7Bva2dEYer3nUCf94eE3ZjApN5Wq/AyqKyZQkZdGZX46k/PSKc1NZWdjB1/577X8y1Pv8OuVu1k8u5irzipi7qQc6luPsqupg13NHWSmJHD5jEKNEhdfaEDZOPDytgY+8+Aq8jOSaTzSxSXTClh66wJSEoN+lxYTevr6qW3sYMv+Nrbsb2Pz/ja2HjhCw5GuE9skBQNMzk9namEGUwozmFqYwdSCDKoK0of979zR1ctvVu5m5c5DvLq9ie6+/iG3nVqYQWdXL5dML+DvLptCed7Q40tERuJUA8oUBONEc3sXE9KT+PnLO7j76a3MnZTNN5bMpLpyAkkJkfUJcM7hHCcmwItFzjkOtnWxoa6VjfWH2VB/mI31bRRkJvOR+aV8YHYR+1qPhg/6R9iyv42ahvYTB+ekYICphRnMLMlkWmFm6IBfmEFZbioJwTPvW9He1cuydfU0tXdTNiGVyrx0KvLSaThyjCfW7WPbgSMEAsazmw8yOT+d39+xiLwRPsSop6+f+paj7D7UyZ7mDva2HCUvI4lLpxUwsyTrjNsgsUlBEGeWrqjlnue2kZwQ4Oe3nMvCKflDbrujsZ3H3qrj6Y0HmFaYyacvrGTRlLwxrPbMdHT1sr6ulbV7Wnlrdwsb6g/TGP5LPmAwvSiTsyZmU9PYzvq9re/6bEFmMrNKsphVksms4ixmlWRRVZBO4igc8M/U71bv5Zt/2MDMkiw+tbCCQx3d7GrqIDkxwKKqfOaX59Da2cOeQx3sbu4MH/Q72X2og32tx+jr/+v/10nBAN19/aQmBvnhTXP54NyJPrZM/KIgiDMH247R1N7F9fe+zoLKHB767AXvOivo6Orl8XX1/G51Hev3thIwOK9yAm/taSE3LYnH77yIiTnRd4/BOcfeQ0dZs+cQb+1u5a09Lbxz4MiJg97UwgzOmZTD2aVZnD0ph9klWaQm/fWyzZu1zdQ0tlMxIZ2ZJZlR/bjQ3r5+fv7SDpauqOVI+AZ0YWYynd1977ohfVxuWiLleelUTEijIi+N8glpVOSlU5GXRmFmMlv2t/GZB1fRdqyHJ+68mBnqxRR3FARx6su/Xcuf1u/jv7+wkPMn57H1wBEeXrmbP66tp72rlxlFmdy4YBLXzZtIYVYKT729ny/+5i0+PG8i99w83+/ycc6x7WA7b+xoYmXtIVbvPkRTe2jgXHpSkHnlOSwoz2V+RS7nluWOy4n3+vr62dtylPzMZDKSE+jt6+flbY3Utx4lLz05dNDPSyMrZfi2r9vTwk2/eIMFFbn81+cXxvQlQBk5TToXp+66fArPv3OQ2x5azYyiTFbvbiEpGODauSXcsrCCc8tz3tWd8YpZRVw+o4DH1+2jq7efL1xSxfTiTNKSxubXxDnHjsYO3qhtZuWOZlbWNtPcETrwl+akcsm0As6tyOXc8lxmFGcSjIMDWTAYeNeEhAnBAFfMKjqtfc0rz+Xj55fz6zd28/i6ej5y7qTRKlNinM4IxrH+fsfyTQd44LWdNHd087HqMm6qLmNCetKQn6lr6eC+l2t57K16unr7+eKlU/j7K6d7dtA93NnDiu2NvLS1kVe2N57oqVOSncKiqjwWTsljUVUeZRPSPPn3403b0R4W/9sK+p3j2b+/hOy0oX8XZHzRpaE41tXbR2IgMKLLAH39jkMd3Xzyl29y8PAxlt110ah1Y+zvd2za18ZLWxt4aVsja/e00O8gJy2Ri6fmc9HUfBZV5VGRl6bBVx5ZtraeL//3Oj4wq5Cf3bIg4l5lEtt0aSiOJSeMfCxBMGAUZCbz+Ysn8/Xfb+CZTQf4/CVTTruGYz19vLq9ieWbDvDi1kaa2kN/9c+dlM1dl0/l0hmFzCvLiYtLPdHg2nMm8vL2Rv7wVj1X/3gFF07JZ0phOh+tLhuzy4ASXfRTlyFdPaeYHz27jV+v3M0nFlaM6CDR3tXLS1sbeHrjAV58p4GO7j4yUxK4dHoBl88o5JLpBRRkRm+vnfEsEDC+d8NcJuen8/yWhhOdB57ZdJAHPn0eKUkaiBhvPL00ZGZLgB8DQeB+59z3TlqfDTwMlBMKpR865x481T51aWhs3b+ilv/35BaumVvCv1x/NtmpQ/dOOdbTx3NbDvL42n2s2N5Id28/eelJLD6riCVzSlhUlafLEFGov9/x/aff4Rcrarnt4sl869rZfpckHvDl0pCZBYF7gSuBOmCVmS1zzm0esNmdwGbn3IfMrADYama/cc5pcv0ocdWcYhrau1i6opbntxzkillFfO6iySd6HPX1O96sbeaPa+t5auMB2rt6KcpK5pYLyllyVjHVlRN0ySfKBQLGP141g3V7W/nPlbu55YJyqgoy/C5LxpCXl4bOB2qcc7UAZvYIcB0wMAgckGmhu4IZwCHgvaNlxDcTc1L5+lUz+NDcifzXX/bw+Np6/rxhP1MK0lk0JY/nNjdwoO0YGckJXD2nmOvnl3JBVZ4O/jEmGAzwf/9mNh/66Wt8Z9km7v909WndX5LY5NmlITO7EVjinPt8+P2ngAucc3cN2CYTWAbMBDKBjznn/jzIvm4HbgcoLy9fsHv3bk9qluHVNrbz5s5D/PLVnexq6uCyGQV8eH4pH5hVpEnuxoHvPLGRh97YzfzyHB78zHnkqHvpuOFXr6HB/iQ8OXWuAtYB7wemAM+a2SvOubZ3fci5pcBSCN0j8KBWiVBVQQZVBRm8f2Yh+RnJ+st/nPnWtbNZUJHL1x7dwEd/8QYP33aBpsaOA17euasDyga8nwTsO2mbzwKPuZAaYCehswOJckVZKQqBcSghGOBv5pXy75+Yz/aGdv7h0fXE2lgjGTkvg2AVMM3MJptZEnAzoctAA+0BrgAwsyJgBlDrYU0iEoHFZxVz68IKXtnexI+e2aYwGOc8uzTknOs1s7uA5YS6jz7gnNtkZneE198HfBf4lZm9TehS0jecc01e1SQikfvWB2ezraGdn75YQ1KC8aUrpvtdknhEU0yIyJCOdvdx432vs7u5kw3fWawZS2PYqW4Wa3SPiAwpNSnIR6vLaO/q5e36w36XIx5REIjIKV08LfSEu1e2N/pciXhFQSAip1SVn86E9CRe29HsdyniEQWBiJySmTG/LIdN9Yfp7+/3uxzxgIJARIa1sCqPtmO9bN7fNvzGEnMUBCIyrOP3CV6r0eWh8UhBICLDmlGUQU5aIm/UKgjGIwWBiAwrEAgwtzSb9XtbNcp4HFIQiEhELpyaT0tnD9sOHvG7FBllCgIRichlMwoA+KHmHhp3FAQiEpGZxVl87LxJPLv5IP+5Us8EGU8UBCISse9edzaT89N58LVdfpcio0hBICIRS0oIcP38UnY2dfDOAY0pGC8UBCIyIh+eNxGAP6yp87kSGS0KAhEZkfK8dKYXZbB800G/S5FRoiAQkRFbclYxew51snm/pqYeDxQEIjJi188vxYAn1p78GHKJRQoCERmxyQUZzCzJ5M9v76ezu9fvcuQMKQhE5LR8rLqMupajXPS9F6htbPe7HDkDCgIROS2fvrCSH9wwl5bOHp7bohvHsUxBICKnxcy4ccEkEgLG3kNH/S5HzoCCQEROWyBgTEhPYl+rgiCWKQhE5IzkZySzv+2Y32XIGVAQiMgZKcpKoelIl99lyBlQEIjIGSnNTeFQRzf9/ZqaOlYpCETkjEzKSaO339HUrrOCWKUgEJEzMik3FYC9LbphHKsUBCJyRo4HQV1Lp8+VyOlSEIjIGSk9EQQ6I4hVCgIROSP5GckkBo16BUHM8jQIzGyJmW01sxoz++YQ21xmZuvMbJOZvexlPSIy+sw0qCzWJXi1YzMLAvcCVwJ1wCozW+ac2zxgmxzgZ8AS59weMyv0qh4R8U5+RjIHNKgsZnl5RnA+UOOcq3XOdQOPANedtM0ngMecc3sAnHMNHtYjIh4pzkpW99EY5mUQlAJ7B7yvCy8baDqQa2YvmdkaM7t1sB2Z2e1mttrMVjc2NnpUroicrtLcNA51dNOnQWUxKaIgMLOLzOxZM9tmZrVmttPMaof72CDLTv4tSQAWAB8ErgK+ZWbT3/Mh55Y656qdc9UFBQWRlCwiY2hSbir9Dho11URMivQewS+BvwfWAH0RfqYOKBvwfhJw8nPt6oAm51wH0GFmK4BzgG0R/hsiEgUm5aYBsPdQB8XZKT5XIyMV6aWhw865p5xzDc655uNfw3xmFTDNzCabWRJwM7DspG2eAN5nZglmlgZcAGwZUQtExHdl4bEE9a26YRyLIj0jeNHM7gYeA06c+znn3hrqA865XjO7C1gOBIEHnHObzOyO8Pr7nHNbzOxpYAPQD9zvnNt4mm0REZ+UTQifEWh0cUyKNAguCH+vHrDMAe8/1Yecc08CT5607L6T3t8N3B1hHSIShbJTE0lOCGh0cYyKKAicc5d7XYiIxC4zIy89if0aVBaTIu01lG1mPzrehdPM/tXMsr0uTkRiR0FmMvsP6x5BLIr0ZvEDwBHgo+GvNuBBr4oSkdgzrzyHmoZ2ntq43+9SZIQivUcwxTl3w4D3/2Rm67woSERi09eunMHzWxr4+qMb6Ojq44ZzSzEbbDiRRJtIzwiOmtnFx9+Y2UWALgaKyAmZqYks/VQ1OWmJfO3R9Tyx7uRhQxKtIg2CLwL3mtkuM9sN/Dtwh3dliUgsmj0xi+e+eim5aYk8unrv8B+QqBBpr6F1wDlmlhV+3+ZpVSISs1ISg1w2o5An395Px7Fe0lM8m+RYRskpf0Jm9knn3MNm9tWTlgPgnPuRh7WJSIy6dm4Jf1xbzzNbDnL9/JPnmpRoM9ylofTw98whvkRE3uOS6QWkJwdZvvGA36VIBE55RuCc+0X4+z+NTTkiMh4kBgPMmZjN2r0tfpciEYh0QNkPzCzLzBLN7HkzazKzT3pdnIjErkVT8jjY1sUBjTaOepH2GlocvkF8LaGpo6cDX/esKhGJeRdNzQPg9drhJioWv0UaBInh79cAv3XOHfKoHhEZJ+aV5ZKcEOD1HU1+lyLDiLRf15/M7B1Cg8j+zswKAE0qIiJDSgwGmFGcyepduk8Q7SI6I3DOfRNYBFQ753qADt77IHoRkXd537R8djV3clCT0UW1UwaBmb0//P0jwOXAdeHXS4ALvS9PRGLZ4tnFADyzWd1Io9lwl4YuBV4APjTIOkfoiWUiIoOaOymbnNREnttykE8tqvS7HBnCcOMIvhP+/tmxKUdExhMz44KqCbxa00RvXz8JwUj7p8hYinQcwb+YWc6A97lm9s/elSUi48UVMwvp6Opj7d5Wv0uRIUQaz1c75078FJ1zLYS6koqInNKl0wsBeHV7o8+VyFAiDYKgmSUff2NmqUDyKbYXEQGgKDuFiTkpvL5DA8uiVaTjCB4GnjezBwndJP4c8JBnVYnIuLKgIpfnNjfQ1+8IBvTUsmgT6TiCHwD/DMwCzgK+G14mIjKs900r4GhPH+t1nyAqjeSJEVuAXufcc2aWZmaZzrkjXhUmIuPHJVPzAXhhawPnVuT6XI2cLNJeQ38L/B74RXhRKfC4V0WJyPhSnJNKRV4aL77T4HcpMohIbxbfCVwEtAE457YDhV4VJSLjzyXTC3hn/xHajnb7XYqcJNIg6HLOnfjpmVkCoZvGIiIRWTy7iD7neOEddSONNpEGwctm9r+AVDO7EngU+JN3ZYnIeLOwKo+0pCC/X1OHc/o7MppEGgTfABqBt4EvAE8C/8erokRk/EkMBvjMhZW8WtPEvS/W8PTG/X6XJGHD9hoyswCwwTk3B/gP70sSkfHqq1dO5/UdzfzwmW0AvPAPl1JVkOFzVTLsGYFzrh9Yb2blY1CPiIxjCcEAD332fH5ww9kAbNnf5nNFApFfGioBNoUfXL/s+NdwHzKzJWa21cxqzOybp9juPDPrM7MbIy1cRGJTdloiS84uAWDLAQ1FigaRDij7p5Hu2MyCwL3AlYQeeL/KzJY55zYPst33geUj/TdEJDZlpSSSk5pIzcF2v0sRhgkCM0sB7gCmErpR/EvnXG+E+z4fqHHO1Yb39Qihx1tuPmm7LwF/AM4bQd0iEuNKc1PZ1dzhdxnC8JeGHgKqCYXA1cC/jmDfpcDeAe/rwstOMLNS4HrgvlPtyMxuN7PVZra6sVF9kEXGg6r8dOpbj/pdhjB8EMx2zn3SOfcL4EbgfSPY92BTDJ7cefge4BvOub5T7cg5t9Q5V+2cqy4oKBhBCSISraYVZXDkWC8tHRpp7Lfh7hH0HH/hnOs1G9H0sXVA2YD3k4B9J21TDTwS3m8+cI2Z9TrnNI+RyDg3ozgTgK0Hj7CwKs/nauLbcEFwjpkd799lhEYWt4VfO+dc1ik+uwqYZmaTgXrgZuATAzdwzk0+/trMfgX8j0JAJD5MKwwFwY7GdgWBz4Z7eH3wdHccPoO4i1BvoCDwgHNuk5ndEV5/yvsCIjK+TcxJBaC+RfcJ/DaS5xGMmHPuSULTUQxcNmgAOOc+42UtIhJdUhKDpCUFOXD4mN+lxL1IB5SJiIy6CelJNBxREPhNQSAivpmQnkRju3oN+U1BICK+KcxMVvfRKKAgEBHfFGWl0NLZrecT+ExBICK+Kc5OoafP0XY00plrxAsKAhHxzcTsUBfSg23qQuonBYGI+GZidgoA+9SF1FcKAhHxTXF4UNk+TT7nKwWBiPimMDMZ0BmB3xQEIuKb9OQEUhIDHFQQ+EpBICK+yk1L4mCbgsBPCgIR8VV+RhJNGl3sKwWBiPgqPyOZlk4FgZ8UBCLiq8KsFFo6NLrYTwoCEfFVSVYKx3r7ae/S6GK/KAhExFeluaGxBA1HunyuJH4pCETEVyXh0cX7NajMNwoCEfFV8fEg0FgC3ygIRMRXBZnh+YZ0RuAbBYGI+CorJYGkhIDOCHykIBARX5kZuWmJenaxjxQEIuK7CelJNB7RoDK/KAhExHf5Gckc0rOLfaMgEBHfTcxJpbG9S6OLfaIgEBHfTclPp7u3n0YNKvOFgkBEfDelMAOAnU0dPlcSnxQEIuK7qoJQENQ0tvtcSXxSEIiI70pzUgkY7GhQEPhBQSAivktKCJCXkczu5k6/S4lLCgIRiQolWSnUtWiaCT94GgRmtsTMtppZjZl9c5D1t5jZhvDX62Z2jpf1iEj0Ks9L40CbgsAPngWBmQWBe4GrgdnAx81s9kmb7QQudc7NBb4LLPWqHhGJbpV56Rw+2qsH1PjAyzOC84Ea51ytc64beAS4buAGzrnXnXMt4bcrgUke1iMiUWxOaRYA6/e2DLOljDYvg6AU2DvgfV142VBuA57ysB4RiWKLpuQTMHh+S4PfpcSdBA/3bYMsG3T8uJldTigILh5i/e3A7QDl5eWjVZ+IRJHs1ESmFmbw2o5mv0uJO16eEdQBZQPeTwL2nbyRmc0F7geuc84N+hvgnFvqnKt2zlUXFBR4UqyI+O+iKflsP3iEtmM9fpcSV7wMglXANDObbGZJwM3AsoEbmFk58BjwKefcNg9rEZEYcMWsQvodrNja6HcpccWzIHDO9QJ3AcuBLcDvnHObzOwOM7sjvNm3gTzgZ2a2zsxWe1WPiES/8yfnkZWSwG//ssfvUuKKl/cIcM49CTx50rL7Brz+PPB5L2sQkdiRlBDgQ+dM5JFVe2loO0ZhVorfJcUFjSwWkajyuYsq6et3PPTGLr9LiRsKAhGJKlMKM6nKT+e1mia/S4kbCgIRiTrTizI1Ad0YUhCISNSZVZJJS2ePnmM8RhQEIhJ1zinLAWDL/jafK4kPCgIRiTqzS47PO9TqcyXxQUEgIlGnIDOZzOQENu3TGcFYUBCISNQxMyrz0/QM4zGiIBCRqDStKJM9zZ04N+hclTKKFAQiEpXmTMzmaE8f9a16apnXFAQiEpXODj+oZmP9YZ8rGf8UBCISlWaEew5tqFMQeE1BICJRKSslkYKMZN45oJ5DXlMQiEjUqsxPY0djh99ljHsKAhGJWjOLM6lvOUpPX7/fpYxrCgIRiVrzynLp7Xe8tafF71LGNQWBiEStJXOKSUkM8OCru/wuZVxTEIhI1EpPTuCq2cW8sLWBw0f1QHuvKAhEJKrddvFkunv7uffFGr9LGbcUBCIS1eaW5XDFzEIeeHWnBpd5REEgIlHv7pvOISM5gVvuf5PlGw/Q36/5h0aTgkBEot6E9CR+94VFZKUk8IWH17Doe89T36JHWY4WBYGIxITpxZk8+9VLufvGubR09PC//7jR75LGDQWBiMSMlMQgN1WX8ZmLKnlpWyMvvtPgd0njgoJARGLOV6+czsTsFL7++/XqVjoKFAQiEnNSEoP85OPzOdTRzSfvf5N9rbpfcCYUBCISk6orJ/Dd6+aw7eARrvy3FTy6eq/fJcUsBYGIxKxbFlbw1Ffex6TcNL7++w187dH1mqDuNCT4XYCIyJmoKsjgz1+6mG8/sYn/+sseXq1p4uo5xcyZmM2155SQnBD0u8SoZ7H2YOjq6mq3evVqv8sQkSi0fNMB7n56Kzsa23FAdmoi188v5arZRcyamEVOWpLfJfrGzNY456oHXacgEJHxprevn9dqmrhvRS0rdzRz/Cg3IT2Jqvx0PjCriAun5pGfkUxC0EgMBELfgwGSEwKYma/1e8G3IDCzJcCPgSBwv3Pueyett/D6a4BO4DPOubdOtU8FgYiMROORLjbUtbKx/jCb97exsb6N+tajQ26fnhSkPC+dqvx00pKCpCQGKZ+QyqIp+cwsziQhGJu3Vk8VBJ7dIzCzIHAvcCVQB6wys2XOuc0DNrsamBb+ugD4efi7iMioKMhM5opZRVwxq+jEst3NHbxdf5im9m56+/rp7u2np6+f7r5+6luOsr2hndd2NNHVE1reG57bKCFgJCcGSAgECJqRmhSkJDuFyvx0JuenU5iZTEpikNTEUIAkJwZICgZISgidaSQlHH8dDL0PBggE/O/2CRcAAAbpSURBVD/78PJm8flAjXOuFsDMHgGuAwYGwXXAr13otGSlmeWYWYlzbr+HdYlInKvIS6ciLz2ibZ1z1LUc5Y3aZjbUtXKspz8UHn39HDnWS33rUTa/3UZnd99p1ZIQCF2SSggaSeHvgfClKQPMjGAAEoMBPn5+OZ9/X9Vp/TunrGHU9/hXpcDAjr11vPev/cG2KQXeFQRmdjtwO0B5efmoFyoiMhQzo2xCGmUT0vhoddmg2zjnaOnsoaWzm+7e0BnGsZ4+jvX00dHdR2d3L53dofdd4fVdvX109zq6e/vo6uunp9fR0xc6A+l3DucI3dtwkBA0khOCFGQme9JGL4NgsPOdk29IRLINzrmlwFII3SM489JEREaPmTEhPYkJ6bHZK8nLux51wMD4nATsO41tRETEQ14GwSpgmplNNrMk4GZg2UnbLANutZCFwGHdHxARGVueXRpyzvWa2V3AckLdRx9wzm0yszvC6+8DniTUdbSGUPfRz3pVj4iIDM7TKSacc08SOtgPXHbfgNcOuNPLGkRE5NRic2SEiIiMGgWBiEicUxCIiMQ5BYGISJyLudlHzawR2H2aH88HmkaxnFigNscHtTk+nEmbK5xzBYOtiLkgOBNmtnqo2ffGK7U5PqjN8cGrNuvSkIhInFMQiIjEuXgLgqV+F+ADtTk+qM3xwZM2x9U9AhERea94OyMQEZGTKAhEROLcuAwCM1tiZlvNrMbMvjnIejOzn4TXbzCzc/2oczRF0OZbwm3dYGavm9k5ftQ5moZr84DtzjOzPjO7cSzr80IkbTazy8xsnZltMrOXx7rG0RbB73a2mf3JzNaH2xzTsxib2QNm1mBmG4dYP/rHL+fcuPoiNOX1DqAKSALWA7NP2uYa4ClCT0hbCLzpd91j0OYLgdzw66vjoc0DtnuB0Cy4N/pd9xj8nHMIPRe8PPy+0O+6x6DN/wv4fvh1AXAISPK79jNo8yXAucDGIdaP+vFrPJ4RnA/UOOdqnXPdwCPAdSdtcx3waxeyEsgxs5KxLnQUDdtm59zrzrmW8NuVhJ4GF8si+TkDfAn4A9AwlsV5JJI2fwJ4zDm3B8A5F+vtjqTNDsg0MwMyCAVB79iWOXqccysItWEoo378Go9BUArsHfC+LrxspNvEkpG25zZCf1HEsmHbbGalwPXAfYwPkfycpwO5ZvaSma0xs1vHrDpvRNLmfwdmEXrM7dvAV5xz/WNTni9G/fjl6YNpfGKDLDu5j2wk28SSiNtjZpcTCoKLPa3Ie5G0+R7gG865vtAfizEvkjYnAAuAK4BU4A0zW+mc2+Z1cR6JpM1XAeuA9wNTgGfN7BXnXJvXxflk1I9f4zEI6oCyAe8nEfpLYaTbxJKI2mNmc4H7gaudc81jVJtXImlzNfBIOATygWvMrNc59/jYlDjqIv3dbnLOdQAdZrYCOAeI1SCIpM2fBb7nQhfQa8xsJzAT+MvYlDjmRv34NR4vDa0CppnZZDNLAm4Glp20zTLg1vDd94XAYefc/rEudBQN22YzKwceAz4Vw38dDjRsm51zk51zlc65SuD3wN/FcAhAZL/bTwDvM7MEM0sDLgC2jHGdoymSNu8hdAaEmRUBM4DaMa1ybI368WvcnRE453rN7C5gOaEeBw845zaZ2R3h9fcR6kFyDVADdBL6iyJmRdjmbwN5wM/CfyH3uhieuTHCNo8rkbTZObfFzJ4GNgD9wP3OuUG7IcaCCH/O3wV+ZWZvE7ps8g3nXMxOT21mvwUuA/LNrA74DpAI3h2/NMWEiEicG4+XhkREZAQUBCIicU5BICIS5xQEIiJxTkEgIhLnFAQigwjPVrrOzDaGZ7bMGeX97zKz/PDr9tHct8hIKQhEBnfUOTfPOTeH0ARgd/pdkIhXFAQiw3uD8KReZjbFzJ4OT+j2ipnNDC8vMrM/hufEX29mF4aXPx7edpOZ3e5jG0SGNO5GFouMJjMLEpq+4JfhRUuBO5xz283sAuBnhCY7+wnwsnPu+vBnMsLbf845d8jMUoFVZvaHcTDPk4wzCgKRwaWa2TqgElhDaEbLDEIP+Hl0wGymyeHv7wduBXDO9QGHw8u/bGbXh1+XAdMABYFEFQWByOCOOufmmVk28D+E7hH8Cmh1zs2LZAdmdhnwAWCRc67TzF4CUrwpV+T06R6ByCk45w4DXwa+BhwFdprZTXDi2bHHn/38PPDF8PKgmWUB2UBLOARmEnqsoEjUURCIDMM5t5bQs3JvBm4BbjOz9cAm/vrYxK8Al4dnwFwDnAU8DSSY2QZCM2SuHOvaRSKh2UdFROKczghEROKcgkBEJM4pCERE4pyCQEQkzikIRETinIJARCTOKQhEROLc/weRi+FgttjdQAAAAABJRU5ErkJggg==\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "# PRC for all-variable classifier\n",
    "ytest_prob = lr_all.predict_proba(Xtest)\n",
    "precision, recall, _ = precision_recall_curve(ytest, ytest_prob[:,1], pos_label=1)\n",
    "ax=sns.lineplot(recall,precision)\n",
    "ax.set(xlabel=\"Recall\",ylabel=\"Precision\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}