Credit Risk Evaluator.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predictions "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I predict that the Logistic regression model will perform better using scaled data by reducing any sparsity that can introduce noise. In comparison, I dont think the random forest classifier model will improve much after scaling and will produce similar scores for both sclaed and unscaled data, as these models perform well with large data sets. I expect the random forest classifier model to perform better with this large dataset, and also predict that scaling the data will lead to slightly better model scores in the random forest classifier model and substantially better scores in the logistic regression. I expect scaling the data to also reduce overfitting effects in both models."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import dependancies\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.preprocessing import StandardScaler"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Read in the datasets\n",
    "train_df = pd.read_csv(Path('Resources/2019loans.csv'))\n",
    "test_df = pd.read_csv(Path('Resources/2020Q1loans.csv'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>loan_amnt</th>\n",
       "      <th>int_rate</th>\n",
       "      <th>installment</th>\n",
       "      <th>home_ownership</th>\n",
       "      <th>annual_inc</th>\n",
       "      <th>verification_status</th>\n",
       "      <th>pymnt_plan</th>\n",
       "      <th>dti</th>\n",
       "      <th>delinq_2yrs</th>\n",
       "      <th>inq_last_6mths</th>\n",
       "      <th>...</th>\n",
       "      <th>percent_bc_gt_75</th>\n",
       "      <th>pub_rec_bankruptcies</th>\n",
       "      <th>tax_liens</th>\n",
       "      <th>tot_hi_cred_lim</th>\n",
       "      <th>total_bal_ex_mort</th>\n",
       "      <th>total_bc_limit</th>\n",
       "      <th>total_il_high_credit_limit</th>\n",
       "      <th>hardship_flag</th>\n",
       "      <th>debt_settlement_flag</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>7000.0</td>\n",
       "      <td>0.1894</td>\n",
       "      <td>256.38</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>75000.0</td>\n",
       "      <td>Not Verified</td>\n",
       "      <td>n</td>\n",
       "      <td>28.62</td>\n",
       "      <td>0.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>...</td>\n",
       "      <td>87.5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>352260.0</td>\n",
       "      <td>62666.0</td>\n",
       "      <td>35000.0</td>\n",
       "      <td>10000.0</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>low_risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>40000.0</td>\n",
       "      <td>0.1614</td>\n",
       "      <td>975.71</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>102000.0</td>\n",
       "      <td>Source Verified</td>\n",
       "      <td>n</td>\n",
       "      <td>11.72</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>294664.0</td>\n",
       "      <td>109911.0</td>\n",
       "      <td>9000.0</td>\n",
       "      <td>71044.0</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>low_risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>11000.0</td>\n",
       "      <td>0.2055</td>\n",
       "      <td>294.81</td>\n",
       "      <td>RENT</td>\n",
       "      <td>45000.0</td>\n",
       "      <td>Verified</td>\n",
       "      <td>n</td>\n",
       "      <td>37.25</td>\n",
       "      <td>1.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>...</td>\n",
       "      <td>7.7</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>92228.0</td>\n",
       "      <td>36007.0</td>\n",
       "      <td>33000.0</td>\n",
       "      <td>46328.0</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>low_risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4000.0</td>\n",
       "      <td>0.1612</td>\n",
       "      <td>140.87</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>38000.0</td>\n",
       "      <td>Not Verified</td>\n",
       "      <td>n</td>\n",
       "      <td>42.89</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>100.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>284273.0</td>\n",
       "      <td>52236.0</td>\n",
       "      <td>13500.0</td>\n",
       "      <td>52017.0</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>low_risk</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>14000.0</td>\n",
       "      <td>0.1797</td>\n",
       "      <td>505.93</td>\n",
       "      <td>MORTGAGE</td>\n",
       "      <td>43000.0</td>\n",
       "      <td>Source Verified</td>\n",
       "      <td>n</td>\n",
       "      <td>22.16</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>...</td>\n",
       "      <td>25.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>120280.0</td>\n",
       "      <td>88147.0</td>\n",
       "      <td>33300.0</td>\n",
       "      <td>78680.0</td>\n",
       "      <td>N</td>\n",
       "      <td>N</td>\n",
       "      <td>low_risk</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 84 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   loan_amnt  int_rate  installment home_ownership  annual_inc  \\\n",
       "0     7000.0    0.1894       256.38       MORTGAGE     75000.0   \n",
       "1    40000.0    0.1614       975.71       MORTGAGE    102000.0   \n",
       "2    11000.0    0.2055       294.81           RENT     45000.0   \n",
       "3     4000.0    0.1612       140.87       MORTGAGE     38000.0   \n",
       "4    14000.0    0.1797       505.93       MORTGAGE     43000.0   \n",
       "\n",
       "  verification_status pymnt_plan    dti  delinq_2yrs  inq_last_6mths  ...  \\\n",
       "0        Not Verified          n  28.62          0.0             2.0  ...   \n",
       "1     Source Verified          n  11.72          2.0             0.0  ...   \n",
       "2            Verified          n  37.25          1.0             3.0  ...   \n",
       "3        Not Verified          n  42.89          1.0             0.0  ...   \n",
       "4     Source Verified          n  22.16          1.0             0.0  ...   \n",
       "\n",
       "   percent_bc_gt_75  pub_rec_bankruptcies  tax_liens  tot_hi_cred_lim  \\\n",
       "0              87.5                   0.0        0.0         352260.0   \n",
       "1               0.0                   0.0        0.0         294664.0   \n",
       "2               7.7                   0.0        0.0          92228.0   \n",
       "3             100.0                   0.0        0.0         284273.0   \n",
       "4              25.0                   0.0        0.0         120280.0   \n",
       "\n",
       "  total_bal_ex_mort  total_bc_limit  total_il_high_credit_limit  \\\n",
       "0           62666.0         35000.0                     10000.0   \n",
       "1          109911.0          9000.0                     71044.0   \n",
       "2           36007.0         33000.0                     46328.0   \n",
       "3           52236.0         13500.0                     52017.0   \n",
       "4           88147.0         33300.0                     78680.0   \n",
       "\n",
       "   hardship_flag  debt_settlement_flag    target  \n",
       "0              N                     N  low_risk  \n",
       "1              N                     N  low_risk  \n",
       "2              N                     N  low_risk  \n",
       "3              N                     N  low_risk  \n",
       "4              N                     N  low_risk  \n",
       "\n",
       "[5 rows x 84 columns]"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['low_risk', 'low_risk', 'low_risk', ..., 'high_risk', 'high_risk',\n",
       "       'high_risk'], dtype=object)"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['target'].values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert categorical data to numeric and separate target feature for training data\n",
    "train_dum = train_df.drop('target', axis=1)\n",
    "X = pd.get_dummies(train_dum)\n",
    "y = train_df['target']\n",
    "new_values = {'low_risk' : 0, 'high_risk' : 1}   \n",
    "y = y.replace(new_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Convert categorical data to numeric and separate target feature for testing data\n",
    "test_dum = test_df.drop('target', axis=1)\n",
    "X_test = pd.get_dummies(test_dum)\n",
    "y_test = test_df['target']\n",
    "new_values = {'low_risk' : 0, 'high_risk' : 1}   \n",
    "y_test = y_test.replace(new_values)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['loan_amnt', 'int_rate', 'installment', 'annual_inc', 'dti',\n",
       "       'delinq_2yrs', 'inq_last_6mths', 'open_acc', 'pub_rec', 'revol_bal',\n",
       "       'total_acc', 'out_prncp', 'out_prncp_inv', 'total_pymnt',\n",
       "       'total_pymnt_inv', 'total_rec_prncp', 'total_rec_int',\n",
       "       'total_rec_late_fee', 'recoveries', 'collection_recovery_fee',\n",
       "       'last_pymnt_amnt', 'collections_12_mths_ex_med', 'policy_code',\n",
       "       'acc_now_delinq', 'tot_coll_amt', 'tot_cur_bal', 'open_acc_6m',\n",
       "       'open_act_il', 'open_il_12m', 'open_il_24m', 'mths_since_rcnt_il',\n",
       "       'total_bal_il', 'il_util', 'open_rv_12m', 'open_rv_24m', 'max_bal_bc',\n",
       "       'all_util', 'total_rev_hi_lim', 'inq_fi', 'total_cu_tl', 'inq_last_12m',\n",
       "       'acc_open_past_24mths', 'avg_cur_bal', 'bc_open_to_buy', 'bc_util',\n",
       "       'chargeoff_within_12_mths', 'delinq_amnt', 'mo_sin_old_il_acct',\n",
       "       'mo_sin_old_rev_tl_op', 'mo_sin_rcnt_rev_tl_op', 'mo_sin_rcnt_tl',\n",
       "       'mort_acc', 'mths_since_recent_bc', 'mths_since_recent_inq',\n",
       "       'num_accts_ever_120_pd', 'num_actv_bc_tl', 'num_actv_rev_tl',\n",
       "       'num_bc_sats', 'num_bc_tl', 'num_il_tl', 'num_op_rev_tl',\n",
       "       'num_rev_accts', 'num_rev_tl_bal_gt_0', 'num_sats', 'num_tl_120dpd_2m',\n",
       "       'num_tl_30dpd', 'num_tl_90g_dpd_24m', 'num_tl_op_past_12m',\n",
       "       'pct_tl_nvr_dlq', 'percent_bc_gt_75', 'pub_rec_bankruptcies',\n",
       "       'tax_liens', 'tot_hi_cred_lim', 'total_bal_ex_mort', 'total_bc_limit',\n",
       "       'total_il_high_credit_limit', 'home_ownership_ANY',\n",
       "       'home_ownership_MORTGAGE', 'home_ownership_OWN', 'home_ownership_RENT',\n",
       "       'verification_status_Not Verified',\n",
       "       'verification_status_Source Verified', 'verification_status_Verified',\n",
       "       'pymnt_plan_n', 'initial_list_status_f', 'initial_list_status_w',\n",
       "       'application_type_Individual', 'application_type_Joint App',\n",
       "       'hardship_flag_N', 'hardship_flag_Y', 'debt_settlement_flag_N'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Check column names\n",
    "X_test.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['loan_amnt', 'int_rate', 'installment', 'annual_inc', 'dti',\n",
       "       'delinq_2yrs', 'inq_last_6mths', 'open_acc', 'pub_rec', 'revol_bal',\n",
       "       'total_acc', 'out_prncp', 'out_prncp_inv', 'total_pymnt',\n",
       "       'total_pymnt_inv', 'total_rec_prncp', 'total_rec_int',\n",
       "       'total_rec_late_fee', 'recoveries', 'collection_recovery_fee',\n",
       "       'last_pymnt_amnt', 'collections_12_mths_ex_med', 'policy_code',\n",
       "       'acc_now_delinq', 'tot_coll_amt', 'tot_cur_bal', 'open_acc_6m',\n",
       "       'open_act_il', 'open_il_12m', 'open_il_24m', 'mths_since_rcnt_il',\n",
       "       'total_bal_il', 'il_util', 'open_rv_12m', 'open_rv_24m', 'max_bal_bc',\n",
       "       'all_util', 'total_rev_hi_lim', 'inq_fi', 'total_cu_tl', 'inq_last_12m',\n",
       "       'acc_open_past_24mths', 'avg_cur_bal', 'bc_open_to_buy', 'bc_util',\n",
       "       'chargeoff_within_12_mths', 'delinq_amnt', 'mo_sin_old_il_acct',\n",
       "       'mo_sin_old_rev_tl_op', 'mo_sin_rcnt_rev_tl_op', 'mo_sin_rcnt_tl',\n",
       "       'mort_acc', 'mths_since_recent_bc', 'mths_since_recent_inq',\n",
       "       'num_accts_ever_120_pd', 'num_actv_bc_tl', 'num_actv_rev_tl',\n",
       "       'num_bc_sats', 'num_bc_tl', 'num_il_tl', 'num_op_rev_tl',\n",
       "       'num_rev_accts', 'num_rev_tl_bal_gt_0', 'num_sats', 'num_tl_120dpd_2m',\n",
       "       'num_tl_30dpd', 'num_tl_90g_dpd_24m', 'num_tl_op_past_12m',\n",
       "       'pct_tl_nvr_dlq', 'percent_bc_gt_75', 'pub_rec_bankruptcies',\n",
       "       'tax_liens', 'tot_hi_cred_lim', 'total_bal_ex_mort', 'total_bc_limit',\n",
       "       'total_il_high_credit_limit', 'home_ownership_ANY',\n",
       "       'home_ownership_MORTGAGE', 'home_ownership_OWN', 'home_ownership_RENT',\n",
       "       'verification_status_Not Verified',\n",
       "       'verification_status_Source Verified', 'verification_status_Verified',\n",
       "       'pymnt_plan_n', 'initial_list_status_f', 'initial_list_status_w',\n",
       "       'application_type_Individual', 'application_type_Joint App',\n",
       "       'hardship_flag_N', 'hardship_flag_Y', 'debt_settlement_flag_N',\n",
       "       'debt_settlement_flag_Y'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [],
   "source": [
    "# add missing dummy variables to testing set\n",
    "X_test['debt_settlement_flag_Y'] = 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 0, ..., 0, 0, 0])"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# check added values\n",
    "X_test['debt_settlement_flag_Y'].values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logistic Regression model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Training Data Score: 0.6528735632183909\n",
      "Testing Data Score: 0.6528735632183909\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jessica/opt/anaconda3/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    }
   ],
   "source": [
    "# Train the Logistic Regression model on the unscaled data and print the model score\n",
    "classifier = LogisticRegression()\n",
    "\n",
    "# Fit our model using the training data\n",
    "classifier.fit(X, y)\n",
    "print(f\"Training Data Score: {classifier.score(X, y)}\")\n",
    "print(f\"Testing Data Score: {classifier.score(X, y)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# RandomForestClassifier "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score with training data: 1.0\n",
      "Score with test data: 0.6384517226712038\n"
     ]
    }
   ],
   "source": [
    "# Train a Random Forest Classifier model and print the model score\n",
    "model = RandomForestClassifier(random_state=1)\n",
    "model.fit(X, y)\n",
    "print(f\"Score with training data: {model.score(X, y)}\")\n",
    "print(f\"Score with test data: {model.score(X_test, y_test)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Unscaled data analysis "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For the unscaled data, the random forest model seems to not be performing as well on the test data, compared to the train data suggesting overfitting. \n",
    "The logistic regression model has performed the same on both test and train data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Scaling the data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scale the data\n",
    "scaler = StandardScaler().fit(X)\n",
    "X_train_scaled = scaler.transform(X)\n",
    "X_test_scaled = scaler.transform(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score with training data: 0.710919540229885\n",
      "Score with test data: 0.7598894087622289\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/jessica/opt/anaconda3/lib/python3.8/site-packages/sklearn/linear_model/_logistic.py:814: ConvergenceWarning: lbfgs failed to converge (status=1):\n",
      "STOP: TOTAL NO. of ITERATIONS REACHED LIMIT.\n",
      "\n",
      "Increase the number of iterations (max_iter) or scale the data as shown in:\n",
      "    https://scikit-learn.org/stable/modules/preprocessing.html\n",
      "Please also refer to the documentation for alternative solver options:\n",
      "    https://scikit-learn.org/stable/modules/linear_model.html#logistic-regression\n",
      "  n_iter_i = _check_optimize_result(\n"
     ]
    }
   ],
   "source": [
    "# Train the Logistic Regression model on the scaled data and print the model score\n",
    "model = LogisticRegression()\n",
    "model.fit(X_train_scaled, y)\n",
    "print(f\"Score with training data: {model.score(X_train_scaled, y)}\")\n",
    "print(f\"Score with test data: {model.score(X_test_scaled, y_test)}\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Score with training data: 0.9999178981937603\n",
      "Score with test data: 0.6495108464483199\n"
     ]
    }
   ],
   "source": [
    "# Train a Random Forest Classifier model on the scaled data and print the model score\n",
    "model = RandomForestClassifier()\n",
    "model.fit(X_train_scaled, y)\n",
    "print(f\"Score with training data: {model.score(X_train_scaled, y)}\")\n",
    "print(f\"Score with test data: {model.score(X_test_scaled, y_test)}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The linear regression model performed better once the data was scaled, whereas the random forest classifier model has a similar performance for both scaled and unscaled data, suggesting scaling data has little impact to this models outcome. \n",
    "The random forest model shows overfitting again, where the training data produces a better score than the test data.\n",
    "Overall, both models used don't fit this type of data well, and seem to be unreliable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}