Ensembles_credit.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Predicting Credit Card Default Rates using Ensembles\n",
    "\n",
    "Dataset has 1000 rows and 17 columns\n",
    "\n",
    "(1 = Good, 2 = Bad) \n",
    "\n",
    "The rows represent the actual classification and the columns the predicted classification. \n",
    "\n",
    "It is worse to class a customer as good when they are bad (5), than it is to class a customer as bad when they are good (1). \n",
    "\n",
    "Attribute Information:\n",
    "Attribute 1: (qualitative) \n",
    "Status of existing checking account \n",
    "A11 : ... < 0 DM \n",
    "A12 : 0 <= ... < 200 DM \n",
    "A13 : ... >= 200 DM / salary assignments for at least 1 year \n",
    "A14 : no checking account \n",
    "\n",
    "Attribute 2: (numerical) \n",
    "Duration in month \n",
    "\n",
    "Attribute 3: (qualitative) \n",
    "Credit history \n",
    "A30 : no credits taken/ all credits paid back duly \n",
    "A31 : all credits at this bank paid back duly \n",
    "A32 : existing credits paid back duly till now \n",
    "A33 : delay in paying off in the past \n",
    "A34 : critical account/ other credits existing (not at this bank) \n",
    "\n",
    "Attribute 4: (qualitative) \n",
    "Purpose \n",
    "A40 : car (new) \n",
    "A41 : car (used) \n",
    "A42 : furniture/equipment \n",
    "A43 : radio/television \n",
    "A44 : domestic appliances \n",
    "A45 : repairs \n",
    "A46 : education \n",
    "A47 : (vacation - does not exist?) \n",
    "A48 : retraining \n",
    "A49 : business \n",
    "A410 : others \n",
    "\n",
    "Attribute 5: (numerical) \n",
    "Credit amount \n",
    "\n",
    "Attibute 6: (qualitative) \n",
    "Savings account/bonds \n",
    "A61 : ... < 100 DM \n",
    "A62 : 100 <= ... < 500 DM \n",
    "A63 : 500 <= ... < 1000 DM \n",
    "A64 : .. >= 1000 DM \n",
    "A65 : unknown/ no savings account \n",
    "\n",
    "Attribute 7: (qualitative) \n",
    "Present employment since \n",
    "A71 : unemployed \n",
    "A72 : ... < 1 year \n",
    "A73 : 1 <= ... < 4 years \n",
    "A74 : 4 <= ... < 7 years \n",
    "A75 : .. >= 7 years \n",
    "\n",
    "Attribute 8: (numerical) \n",
    "Installment rate in percentage of disposable income \n",
    "\n",
    "Attribute 9: (qualitative) \n",
    "Personal status and sex \n",
    "A91 : male : divorced/separated \n",
    "A92 : female : divorced/separated/married \n",
    "A93 : male : single \n",
    "A94 : male : married/widowed \n",
    "A95 : female : single \n",
    "\n",
    "Attribute 10: (qualitative) \n",
    "Other debtors / guarantors \n",
    "A101 : none \n",
    "A102 : co-applicant \n",
    "A103 : guarantor \n",
    "\n",
    "Attribute 11: (numerical) \n",
    "Present residence since \n",
    "\n",
    "Attribute 12: (qualitative) \n",
    "Property \n",
    "A121 : real estate \n",
    "A122 : if not A121 : building society savings agreement/ life insurance \n",
    "A123 : if not A121/A122 : car or other, not in attribute 6 \n",
    "A124 : unknown / no property \n",
    "\n",
    "Attribute 13: (numerical) \n",
    "Age in years \n",
    "\n",
    "Attribute 14: (qualitative) \n",
    "Other installment plans \n",
    "A141 : bank \n",
    "A142 : stores \n",
    "A143 : none \n",
    "\n",
    "Attribute 15: (qualitative) \n",
    "Housing \n",
    "A151 : rent \n",
    "A152 : own \n",
    "A153 : for free \n",
    "\n",
    "Attribute 16: (numerical) \n",
    "Number of existing credits at this bank \n",
    "\n",
    "Attribute 17: (qualitative) \n",
    "Job \n",
    "A171 : unemployed/ unskilled - non-resident \n",
    "A172 : unskilled - resident \n",
    "A173 : skilled employee / official \n",
    "A174 : management/ self-employed/ \n",
    "highly qualified employee/ officer \n",
    "\n",
    "Attribute 18: (numerical) \n",
    "Number of people being liable to provide maintenance for \n",
    "\n",
    "Attribute 19: (qualitative) \n",
    "Telephone \n",
    "A191 : none \n",
    "A192 : yes, registered under the customers name \n",
    "\n",
    "Attribute 20: (qualitative) \n",
    "foreign worker \n",
    "A201 : yes \n",
    "A202 : no "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "#Always import this if before the libraries\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "# calculate accuracy measures and confusion matrix\n",
    "from sklearn import metrics\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "#DT does not take strings as input for the model fit step\n",
    "from sklearn.feature_extraction.text import CountVectorizer  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "credit_df = pd.read_csv(\"credit.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>count</th>\n",
       "      <th>mean</th>\n",
       "      <th>std</th>\n",
       "      <th>min</th>\n",
       "      <th>25%</th>\n",
       "      <th>50%</th>\n",
       "      <th>75%</th>\n",
       "      <th>max</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>months_loan_duration</th>\n",
       "      <td>1000.0</td>\n",
       "      <td>20.903</td>\n",
       "      <td>12.058814</td>\n",
       "      <td>4.0</td>\n",
       "      <td>12.0</td>\n",
       "      <td>18.0</td>\n",
       "      <td>24.00</td>\n",
       "      <td>72.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>amount</th>\n",
       "      <td>1000.0</td>\n",
       "      <td>3271.258</td>\n",
       "      <td>2822.736876</td>\n",
       "      <td>250.0</td>\n",
       "      <td>1365.5</td>\n",
       "      <td>2319.5</td>\n",
       "      <td>3972.25</td>\n",
       "      <td>18424.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>percent_of_income</th>\n",
       "      <td>1000.0</td>\n",
       "      <td>2.973</td>\n",
       "      <td>1.118715</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.00</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>years_at_residence</th>\n",
       "      <td>1000.0</td>\n",
       "      <td>2.845</td>\n",
       "      <td>1.103718</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.0</td>\n",
       "      <td>3.0</td>\n",
       "      <td>4.00</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>1000.0</td>\n",
       "      <td>35.546</td>\n",
       "      <td>11.375469</td>\n",
       "      <td>19.0</td>\n",
       "      <td>27.0</td>\n",
       "      <td>33.0</td>\n",
       "      <td>42.00</td>\n",
       "      <td>75.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>existing_loans_count</th>\n",
       "      <td>1000.0</td>\n",
       "      <td>1.407</td>\n",
       "      <td>0.577654</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>2.00</td>\n",
       "      <td>4.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>dependents</th>\n",
       "      <td>1000.0</td>\n",
       "      <td>1.155</td>\n",
       "      <td>0.362086</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.00</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                       count      mean          std    min     25%     50%  \\\n",
       "months_loan_duration  1000.0    20.903    12.058814    4.0    12.0    18.0   \n",
       "amount                1000.0  3271.258  2822.736876  250.0  1365.5  2319.5   \n",
       "percent_of_income     1000.0     2.973     1.118715    1.0     2.0     3.0   \n",
       "years_at_residence    1000.0     2.845     1.103718    1.0     2.0     3.0   \n",
       "age                   1000.0    35.546    11.375469   19.0    27.0    33.0   \n",
       "existing_loans_count  1000.0     1.407     0.577654    1.0     1.0     1.0   \n",
       "dependents            1000.0     1.155     0.362086    1.0     1.0     1.0   \n",
       "\n",
       "                          75%      max  \n",
       "months_loan_duration    24.00     72.0  \n",
       "amount                3972.25  18424.0  \n",
       "percent_of_income        4.00      4.0  \n",
       "years_at_residence       4.00      4.0  \n",
       "age                     42.00     75.0  \n",
       "existing_loans_count     2.00      4.0  \n",
       "dependents               1.00      2.0  "
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_df.describe().transpose()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>checking_balance</th>\n",
       "      <th>months_loan_duration</th>\n",
       "      <th>credit_history</th>\n",
       "      <th>purpose</th>\n",
       "      <th>amount</th>\n",
       "      <th>savings_balance</th>\n",
       "      <th>employment_duration</th>\n",
       "      <th>percent_of_income</th>\n",
       "      <th>years_at_residence</th>\n",
       "      <th>age</th>\n",
       "      <th>other_credit</th>\n",
       "      <th>housing</th>\n",
       "      <th>existing_loans_count</th>\n",
       "      <th>job</th>\n",
       "      <th>dependents</th>\n",
       "      <th>phone</th>\n",
       "      <th>default</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>6</td>\n",
       "      <td>critical</td>\n",
       "      <td>furniture/appliances</td>\n",
       "      <td>1169</td>\n",
       "      <td>unknown</td>\n",
       "      <td>&gt; 7 years</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>67</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1 - 200 DM</td>\n",
       "      <td>48</td>\n",
       "      <td>good</td>\n",
       "      <td>furniture/appliances</td>\n",
       "      <td>5951</td>\n",
       "      <td>&lt; 100 DM</td>\n",
       "      <td>1 - 4 years</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>22</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>unknown</td>\n",
       "      <td>12</td>\n",
       "      <td>critical</td>\n",
       "      <td>education</td>\n",
       "      <td>2096</td>\n",
       "      <td>&lt; 100 DM</td>\n",
       "      <td>4 - 7 years</td>\n",
       "      <td>2</td>\n",
       "      <td>3</td>\n",
       "      <td>49</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>unskilled</td>\n",
       "      <td>2</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>42</td>\n",
       "      <td>good</td>\n",
       "      <td>furniture/appliances</td>\n",
       "      <td>7882</td>\n",
       "      <td>&lt; 100 DM</td>\n",
       "      <td>4 - 7 years</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>45</td>\n",
       "      <td>none</td>\n",
       "      <td>other</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled</td>\n",
       "      <td>2</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>&lt; 0 DM</td>\n",
       "      <td>24</td>\n",
       "      <td>poor</td>\n",
       "      <td>car</td>\n",
       "      <td>4870</td>\n",
       "      <td>&lt; 100 DM</td>\n",
       "      <td>1 - 4 years</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>53</td>\n",
       "      <td>none</td>\n",
       "      <td>other</td>\n",
       "      <td>2</td>\n",
       "      <td>skilled</td>\n",
       "      <td>2</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>unknown</td>\n",
       "      <td>36</td>\n",
       "      <td>good</td>\n",
       "      <td>education</td>\n",
       "      <td>9055</td>\n",
       "      <td>unknown</td>\n",
       "      <td>1 - 4 years</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>35</td>\n",
       "      <td>none</td>\n",
       "      <td>other</td>\n",
       "      <td>1</td>\n",
       "      <td>unskilled</td>\n",
       "      <td>2</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>unknown</td>\n",
       "      <td>24</td>\n",
       "      <td>good</td>\n",
       "      <td>furniture/appliances</td>\n",
       "      <td>2835</td>\n",
       "      <td>500 - 1000 DM</td>\n",
       "      <td>&gt; 7 years</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>53</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>skilled</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>1 - 200 DM</td>\n",
       "      <td>36</td>\n",
       "      <td>good</td>\n",
       "      <td>car</td>\n",
       "      <td>6948</td>\n",
       "      <td>&lt; 100 DM</td>\n",
       "      <td>1 - 4 years</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>35</td>\n",
       "      <td>none</td>\n",
       "      <td>rent</td>\n",
       "      <td>1</td>\n",
       "      <td>management</td>\n",
       "      <td>1</td>\n",
       "      <td>yes</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>unknown</td>\n",
       "      <td>12</td>\n",
       "      <td>good</td>\n",
       "      <td>furniture/appliances</td>\n",
       "      <td>3059</td>\n",
       "      <td>&gt; 1000 DM</td>\n",
       "      <td>4 - 7 years</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>61</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>1</td>\n",
       "      <td>unskilled</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>no</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>1 - 200 DM</td>\n",
       "      <td>30</td>\n",
       "      <td>critical</td>\n",
       "      <td>car</td>\n",
       "      <td>5234</td>\n",
       "      <td>&lt; 100 DM</td>\n",
       "      <td>unemployed</td>\n",
       "      <td>4</td>\n",
       "      <td>2</td>\n",
       "      <td>28</td>\n",
       "      <td>none</td>\n",
       "      <td>own</td>\n",
       "      <td>2</td>\n",
       "      <td>management</td>\n",
       "      <td>1</td>\n",
       "      <td>no</td>\n",
       "      <td>yes</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  checking_balance  months_loan_duration credit_history               purpose  \\\n",
       "0           < 0 DM                     6       critical  furniture/appliances   \n",
       "1       1 - 200 DM                    48           good  furniture/appliances   \n",
       "2          unknown                    12       critical             education   \n",
       "3           < 0 DM                    42           good  furniture/appliances   \n",
       "4           < 0 DM                    24           poor                   car   \n",
       "5          unknown                    36           good             education   \n",
       "6          unknown                    24           good  furniture/appliances   \n",
       "7       1 - 200 DM                    36           good                   car   \n",
       "8          unknown                    12           good  furniture/appliances   \n",
       "9       1 - 200 DM                    30       critical                   car   \n",
       "\n",
       "   amount savings_balance employment_duration  percent_of_income  \\\n",
       "0    1169         unknown           > 7 years                  4   \n",
       "1    5951        < 100 DM         1 - 4 years                  2   \n",
       "2    2096        < 100 DM         4 - 7 years                  2   \n",
       "3    7882        < 100 DM         4 - 7 years                  2   \n",
       "4    4870        < 100 DM         1 - 4 years                  3   \n",
       "5    9055         unknown         1 - 4 years                  2   \n",
       "6    2835   500 - 1000 DM           > 7 years                  3   \n",
       "7    6948        < 100 DM         1 - 4 years                  2   \n",
       "8    3059       > 1000 DM         4 - 7 years                  2   \n",
       "9    5234        < 100 DM          unemployed                  4   \n",
       "\n",
       "   years_at_residence  age other_credit housing  existing_loans_count  \\\n",
       "0                   4   67         none     own                     2   \n",
       "1                   2   22         none     own                     1   \n",
       "2                   3   49         none     own                     1   \n",
       "3                   4   45         none   other                     1   \n",
       "4                   4   53         none   other                     2   \n",
       "5                   4   35         none   other                     1   \n",
       "6                   4   53         none     own                     1   \n",
       "7                   2   35         none    rent                     1   \n",
       "8                   4   61         none     own                     1   \n",
       "9                   2   28         none     own                     2   \n",
       "\n",
       "          job  dependents phone default  \n",
       "0     skilled           1   yes      no  \n",
       "1     skilled           1    no     yes  \n",
       "2   unskilled           2    no      no  \n",
       "3     skilled           2    no      no  \n",
       "4     skilled           2    no     yes  \n",
       "5   unskilled           2   yes      no  \n",
       "6     skilled           1    no      no  \n",
       "7  management           1   yes      no  \n",
       "8   unskilled           1    no      no  \n",
       "9  management           1    no     yes  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_df.head(10)  #let us look at the raw data. As is evident, there are lot of missing values "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1000, 17)"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>months_loan_duration</th>\n",
       "      <th>amount</th>\n",
       "      <th>percent_of_income</th>\n",
       "      <th>years_at_residence</th>\n",
       "      <th>age</th>\n",
       "      <th>existing_loans_count</th>\n",
       "      <th>dependents</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "      <td>1000.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>20.903000</td>\n",
       "      <td>3271.258000</td>\n",
       "      <td>2.973000</td>\n",
       "      <td>2.845000</td>\n",
       "      <td>35.546000</td>\n",
       "      <td>1.407000</td>\n",
       "      <td>1.155000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>12.058814</td>\n",
       "      <td>2822.736876</td>\n",
       "      <td>1.118715</td>\n",
       "      <td>1.103718</td>\n",
       "      <td>11.375469</td>\n",
       "      <td>0.577654</td>\n",
       "      <td>0.362086</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>4.000000</td>\n",
       "      <td>250.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>19.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>12.000000</td>\n",
       "      <td>1365.500000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>27.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>18.000000</td>\n",
       "      <td>2319.500000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>33.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>24.000000</td>\n",
       "      <td>3972.250000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>42.000000</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>72.000000</td>\n",
       "      <td>18424.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>75.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       months_loan_duration        amount  percent_of_income  \\\n",
       "count           1000.000000   1000.000000        1000.000000   \n",
       "mean              20.903000   3271.258000           2.973000   \n",
       "std               12.058814   2822.736876           1.118715   \n",
       "min                4.000000    250.000000           1.000000   \n",
       "25%               12.000000   1365.500000           2.000000   \n",
       "50%               18.000000   2319.500000           3.000000   \n",
       "75%               24.000000   3972.250000           4.000000   \n",
       "max               72.000000  18424.000000           4.000000   \n",
       "\n",
       "       years_at_residence          age  existing_loans_count   dependents  \n",
       "count         1000.000000  1000.000000           1000.000000  1000.000000  \n",
       "mean             2.845000    35.546000              1.407000     1.155000  \n",
       "std              1.103718    11.375469              0.577654     0.362086  \n",
       "min              1.000000    19.000000              1.000000     1.000000  \n",
       "25%              2.000000    27.000000              1.000000     1.000000  \n",
       "50%              3.000000    33.000000              1.000000     1.000000  \n",
       "75%              4.000000    42.000000              2.000000     1.000000  \n",
       "max              4.000000    75.000000              4.000000     2.000000  "
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "credit_df.describe(include=[np.number])\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1000 entries, 0 to 999\n",
      "Data columns (total 17 columns):\n",
      "checking_balance        1000 non-null object\n",
      "months_loan_duration    1000 non-null int64\n",
      "credit_history          1000 non-null object\n",
      "purpose                 1000 non-null object\n",
      "amount                  1000 non-null int64\n",
      "savings_balance         1000 non-null object\n",
      "employment_duration     1000 non-null object\n",
      "percent_of_income       1000 non-null int64\n",
      "years_at_residence      1000 non-null int64\n",
      "age                     1000 non-null int64\n",
      "other_credit            1000 non-null object\n",
      "housing                 1000 non-null object\n",
      "existing_loans_count    1000 non-null int64\n",
      "job                     1000 non-null object\n",
      "dependents              1000 non-null int64\n",
      "phone                   1000 non-null object\n",
      "default                 1000 non-null object\n",
      "dtypes: int64(7), object(10)\n",
      "memory usage: 132.9+ KB\n"
     ]
    }
   ],
   "source": [
    "credit_df.info()  # many columns are of type object i.e. strings. These need to be converted to ordinal type"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Decision tree in Python can take only numerical / categorical colums. It cannot take string / obeject types. \n",
    "# The following code loops through each column and checks if the column type is object then converts those columns\n",
    "# into categorical with each distinct value becoming a category or code.\n",
    "\n",
    "for feature in credit_df.columns: # Loop through all columns in the dataframe\n",
    "    if credit_df[feature].dtype == 'object': # Only apply for columns with categorical strings\n",
    "        credit_df[feature] = pd.Categorical(credit_df[feature]).codes # Replace strings with an integer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 1000 entries, 0 to 999\n",
      "Data columns (total 17 columns):\n",
      "checking_balance        1000 non-null int8\n",
      "months_loan_duration    1000 non-null int64\n",
      "credit_history          1000 non-null int8\n",
      "purpose                 1000 non-null int8\n",
      "amount                  1000 non-null int64\n",
      "savings_balance         1000 non-null int8\n",
      "employment_duration     1000 non-null int8\n",
      "percent_of_income       1000 non-null int64\n",
      "years_at_residence      1000 non-null int64\n",
      "age                     1000 non-null int64\n",
      "other_credit            1000 non-null int8\n",
      "housing                 1000 non-null int8\n",
      "existing_loans_count    1000 non-null int64\n",
      "job                     1000 non-null int8\n",
      "dependents              1000 non-null int64\n",
      "phone                   1000 non-null int8\n",
      "default                 1000 non-null int8\n",
      "dtypes: int64(7), int8(10)\n",
      "memory usage: 64.5 KB\n"
     ]
    }
   ],
   "source": [
    "credit_df.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Pairplot using sns\n",
    "# import seaborn as sns\n",
    "# sns.pairplot(credit_df, hue = \"default\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_labels = [ 'checking_balance', 'months_loan_duration' ,'credit_history' , 'purpose, amount', 'savings_balance' , 'employment_duration'\n",
    ", 'percent_of_income', 'years_at_residence', 'age', 'other_credit', 'housing', 'existing_loans_count', 'job', 'dependents', 'phone', 'default']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_char_label = ['No', 'Yes']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [],
   "source": [
    "# capture the target column (\"default\") into separate vectors for training set and test set\n",
    "\n",
    "X = credit_df.drop(\"default\" , axis=1)\n",
    "\n",
    "y = credit_df.pop(\"default\")\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "# splitting data into training and test set for independent attributes\n",
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "X_train, X_test, train_labels, test_labels = train_test_split(X, y, test_size=.30, random_state=1)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "# invoking the decision tree classifier function. Using 'entropy' method of finding the split columns. Other option \n",
    "# could be gini index.  Restricting the depth of the tree to 5 (no particular reason for selecting this)\n",
    "\n",
    "#dt_model = DecisionTreeClassifier(criterion = 'entropy' , max_depth = 5, random_state = 100)\n",
    "                                  \n",
    "dt_model = DecisionTreeClassifier(criterion = 'entropy' )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=None,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dt_model.fit(X_train, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           Imp\n",
      "checking_balance      0.122194\n",
      "months_loan_duration  0.068224\n",
      "credit_history        0.066657\n",
      "purpose               0.066136\n",
      "amount                0.200756\n",
      "savings_balance       0.084224\n",
      "employment_duration   0.038207\n",
      "percent_of_income     0.038241\n",
      "years_at_residence    0.049381\n",
      "age                   0.111005\n",
      "other_credit          0.045786\n",
      "housing               0.007996\n",
      "existing_loans_count  0.028725\n",
      "job                   0.037778\n",
      "dependents            0.024258\n",
      "phone                 0.010430\n"
     ]
    }
   ],
   "source": [
    "from IPython.display import Image  \n",
    "#import pydotplus as pydot\n",
    "from sklearn import tree\n",
    "from os import system\n",
    "\n",
    "Credit_Tree_File = open('d:\\credit_tree.dot','w')\n",
    "dot_data = tree.export_graphviz(dt_model, out_file=Credit_Tree_File, feature_names = list(X_train), class_names = list(train_char_label))\n",
    "\n",
    "Credit_Tree_File.close()\n",
    "\n",
    "# importance of features in the tree building ( The importance of a feature is computed as the \n",
    "#(normalized) total reduction of the criterion brought by that feature. It is also known as the Gini importance )\n",
    "\n",
    "print (pd.DataFrame(dt_model.feature_importances_, columns = [\"Imp\"], index = X_train.columns))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# You can also copy the script in the .dot file and paste it at http://webgraphviz.com/ to get tree view \n",
    "#or create a .png as below\n",
    "\n",
    "# system(\"dot -Tpng D:\\credit_tree.dot -o D:/credit_tree.png\")\n",
    "# Image(\"d:\\credit_tree.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_predict = dt_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n",
      "0.67\n"
     ]
    }
   ],
   "source": [
    "print(dt_model.score(X_train , train_labels))\n",
    "print(dt_model.score(X_test , test_labels))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[166  48]\n",
      " [ 51  35]]\n"
     ]
    }
   ],
   "source": [
    "print(metrics.confusion_matrix(test_labels, y_predict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Look at the class level metric for the defaulter class. It is poor. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "#                                      Regularising the Decision Tree"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "DecisionTreeClassifier(class_weight=None, criterion='entropy', max_depth=7,\n",
       "            max_features=None, max_leaf_nodes=None,\n",
       "            min_impurity_decrease=0.0, min_impurity_split=None,\n",
       "            min_samples_leaf=1, min_samples_split=2,\n",
       "            min_weight_fraction_leaf=0.0, presort=False, random_state=None,\n",
       "            splitter='best')"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "\n",
    "reg_dt_model = DecisionTreeClassifier(criterion = 'entropy', max_depth = 7)\n",
    "reg_dt_model.fit(X_train, train_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "                           Imp\n",
      "checking_balance      0.122194\n",
      "months_loan_duration  0.068224\n",
      "credit_history        0.066657\n",
      "purpose               0.066136\n",
      "amount                0.200756\n",
      "savings_balance       0.084224\n",
      "employment_duration   0.038207\n",
      "percent_of_income     0.038241\n",
      "years_at_residence    0.049381\n",
      "age                   0.111005\n",
      "other_credit          0.045786\n",
      "housing               0.007996\n",
      "existing_loans_count  0.028725\n",
      "job                   0.037778\n",
      "dependents            0.024258\n",
      "phone                 0.010430\n"
     ]
    }
   ],
   "source": [
    "credit_tree_regularized = open('d:\\credit_tree_regularized.dot','w')\n",
    "dot_data = tree.export_graphviz(reg_dt_model, out_file= credit_tree_regularized , feature_names = list(X_train), class_names = list(train_char_label))\n",
    "\n",
    "credit_tree_regularized.close()\n",
    "\n",
    "print (pd.DataFrame(dt_model.feature_importances_, columns = [\"Imp\"], index = X_train.columns))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You can also copy the script in the .dot file and paste it at http://webgraphviz.com/ to get tree view\n",
    "# or create a .png as below\n",
    "# system(\"dot -Tpng D:\\credit_tree_regularized.dot -o D:/credit_tree_regularized.png\")\n",
    "# Image(\"d:\\credit_tree_regularized.png\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_predict = reg_dt_model.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.6933333333333334"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "reg_dt_model.score(X_test , test_labels)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[[172  42]\n",
      " [ 50  36]]\n"
     ]
    }
   ],
   "source": [
    "print(metrics.confusion_matrix(test_labels, y_predict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#                             Ensemble Learning - Bagging"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import BaggingClassifier\n",
    "\n",
    "bgcl = BaggingClassifier(base_estimator=dt_model, n_estimators=50)\n",
    "\n",
    "#bgcl = BaggingClassifier(n_estimators=50)\n",
    "bgcl = bgcl.fit(X_train, train_labels)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.7466666666666667\n",
      "[[185  29]\n",
      " [ 47  39]]\n"
     ]
    }
   ],
   "source": [
    "y_predict = bgcl.predict(X_test)\n",
    "\n",
    "print(bgcl.score(X_test , test_labels))\n",
    "\n",
    "print(metrics.confusion_matrix(test_labels, y_predict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "# Ensemble Learning - AdaBoosting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import AdaBoostClassifier\n",
    "abcl = AdaBoostClassifier(base_estimator=dt_model, n_estimators=10)\n",
    "#abcl = AdaBoostClassifier( n_estimators=50)\n",
    "abcl = abcl.fit(X_train, train_labels)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.6433333333333333\n",
      "[[159  55]\n",
      " [ 52  34]]\n"
     ]
    }
   ],
   "source": [
    "y_predict = abcl.predict(X_test)\n",
    "print(abcl.score(X_test , test_labels))\n",
    "\n",
    "print(metrics.confusion_matrix(test_labels, y_predict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#                     Ensemble Learning - GradientBoost"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import GradientBoostingClassifier\n",
    "gbcl = GradientBoostingClassifier(n_estimators = 50)\n",
    "gbcl = gbcl.fit(X_train, train_labels)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.73\n",
      "[[190  24]\n",
      " [ 57  29]]\n"
     ]
    }
   ],
   "source": [
    "y_predict = gbcl.predict(X_test)\n",
    "print(gbcl.score(X_test , test_labels))\n",
    "print(metrics.confusion_matrix(test_labels, y_predict))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ensemble RandomForest Classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "rfcl = RandomForestClassifier(n_estimators = 50)\n",
    "rfcl = rfcl.fit(X_train, train_labels)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.74\n",
      "[[191  23]\n",
      " [ 55  31]]\n"
     ]
    }
   ],
   "source": [
    "y_predict = rfcl.predict(X_test)\n",
    "print(rfcl.score(X_test , test_labels))\n",
    "print(metrics.confusion_matrix(test_labels, y_predict))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}