lectures/11_ensembles.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 11: Ensembles\n",
    "\n",
    "UBC 2020-21\n",
    "\n",
    "Instructor: Varada Kolhatkar\n",
    "\n",
    "<blockquote>\n",
    "The interests of truth require a diversity of opinions.    \n",
    "    \n",
    "by John Stuart Mill\n",
    "</blockquote>\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports, announcements, LOs "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "%matplotlib inline\n",
    "import string\n",
    "import sys\n",
    "from collections import deque\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "sys.path.append(\"code/.\")\n",
    "\n",
    "from plotting_functions import *\n",
    "from sklearn import datasets\n",
    "from sklearn.compose import ColumnTransformer, make_column_transformer\n",
    "from sklearn.dummy import DummyClassifier, DummyRegressor\n",
    "from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.model_selection import (\n",
    "    GridSearchCV,\n",
    "    RandomizedSearchCV,\n",
    "    cross_val_score,\n",
    "    cross_validate,\n",
    "    train_test_split,\n",
    ")\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler\n",
    "from sklearn.svm import SVC, SVR\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from utils import *"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Announcements\n",
    "\n",
    "- HW4 deadline is extended. Due on Friday at 11:59pm.\n",
    "- HW5 will be released tomorrow. It'll be due Monday, October 31st. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Lecture learning objectives\n",
    "\n",
    "From this lecture, you will be able to \n",
    "\n",
    "- Use `scikit-learn`'s `RandomForestClassifier` and explain its main hyperparameters. \n",
    "- Explain randomness in random forest algorithm. \n",
    "- Use other tree-based models such as as `XGBoost` and `LGBM`.  \n",
    "- Employ ensemble classifier approaches, in particular model averaging and stacking.\n",
    "- Explain voting and stacking and the differences between them.\n",
    "- Use `scikit-learn` implementations of these ensemble methods."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- **Ensembles** are models that combine multiple machine learning models to create more powerful models. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The Netflix prize\n",
    "\n",
    "![](img/netflix.png)\n",
    "\n",
    "[Source](https://netflixtechblog.com/netflix-recommendations-beyond-the-5-stars-part-1-55838468f429)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Most of the winning solutions for Kaggle competitions involve some kind of ensembling. For example: \n",
    "\n",
    "![](img/fraud_detection_kaggle.png)\n",
    "\n",
    "<!-- <img src=\"img/fraud_detection_kaggle.png\" width=\"600\" height=\"600\"> -->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Key idea: Groups can often make better decisions than individuals, especially when group members are diverse enough. \n",
    "\n",
    "[The Wisdom of Crowds](http://wisdomofcrowds.blogspot.com/2009/12/introduction-part-i.html)\n",
    "\n",
    "![](img/wisdom_of_crowds.jpg)\n",
    "\n",
    "<!-- <img src=\"img/wisdom_of_crowds.jpg\" width=\"300\" height=\"300\"> -->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tree-based ensemble models \n",
    "- A number of ensemble models in ML literature.\n",
    "- Most successful ones on a variety of datasets are tree-based models. \n",
    "- We'll briefly talk about two such models: \n",
    "    - Random forests\n",
    "    - Gradient boosted trees\n",
    "- We'll also talk about averaging and stacking. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Tree-based models \n",
    "\n",
    "- Decision trees models are \n",
    "    - Interpretable \n",
    "    - They can capture non-linear relationships\n",
    "    - They don't require scaling of the data and theoretically can work with categorical features. \n",
    "- But with a single decision trees are likely to overfit. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Key idea: Combine multiple trees to build stronger models.\n",
    "- These kinds of models are extremely popular in industry and machine learning competitions  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Data\n",
    "\n",
    "- Let's work with [the adult census data set](https://www.kaggle.com/uciml/adult-census-income). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>age</th>\n",
       "      <th>workclass</th>\n",
       "      <th>fnlwgt</th>\n",
       "      <th>education</th>\n",
       "      <th>education.num</th>\n",
       "      <th>marital.status</th>\n",
       "      <th>occupation</th>\n",
       "      <th>relationship</th>\n",
       "      <th>race</th>\n",
       "      <th>sex</th>\n",
       "      <th>capital.gain</th>\n",
       "      <th>capital.loss</th>\n",
       "      <th>hours.per.week</th>\n",
       "      <th>native.country</th>\n",
       "      <th>income</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2729</th>\n",
       "      <td>35</td>\n",
       "      <td>Private</td>\n",
       "      <td>359131</td>\n",
       "      <td>Bachelors</td>\n",
       "      <td>13</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Prof-specialty</td>\n",
       "      <td>Wife</td>\n",
       "      <td>White</td>\n",
       "      <td>Female</td>\n",
       "      <td>7298</td>\n",
       "      <td>0</td>\n",
       "      <td>8</td>\n",
       "      <td>NaN</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6850</th>\n",
       "      <td>40</td>\n",
       "      <td>NaN</td>\n",
       "      <td>78255</td>\n",
       "      <td>HS-grad</td>\n",
       "      <td>9</td>\n",
       "      <td>Divorced</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Not-in-family</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>25</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>414</th>\n",
       "      <td>46</td>\n",
       "      <td>Private</td>\n",
       "      <td>102771</td>\n",
       "      <td>Masters</td>\n",
       "      <td>14</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Tech-support</td>\n",
       "      <td>Husband</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>1977</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7887</th>\n",
       "      <td>33</td>\n",
       "      <td>Private</td>\n",
       "      <td>145434</td>\n",
       "      <td>11th</td>\n",
       "      <td>7</td>\n",
       "      <td>Never-married</td>\n",
       "      <td>Craft-repair</td>\n",
       "      <td>Own-child</td>\n",
       "      <td>White</td>\n",
       "      <td>Male</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>40</td>\n",
       "      <td>United-States</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>23438</th>\n",
       "      <td>28</td>\n",
       "      <td>Private</td>\n",
       "      <td>210945</td>\n",
       "      <td>Assoc-acdm</td>\n",
       "      <td>12</td>\n",
       "      <td>Married-civ-spouse</td>\n",
       "      <td>Adm-clerical</td>\n",
       "      <td>Wife</td>\n",
       "      <td>Black</td>\n",
       "      <td>Female</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>35</td>\n",
       "      <td>Haiti</td>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       age workclass  fnlwgt   education  education.num      marital.status  \\\n",
       "2729    35   Private  359131   Bachelors             13  Married-civ-spouse   \n",
       "6850    40       NaN   78255     HS-grad              9            Divorced   \n",
       "414     46   Private  102771     Masters             14  Married-civ-spouse   \n",
       "7887    33   Private  145434        11th              7       Never-married   \n",
       "23438   28   Private  210945  Assoc-acdm             12  Married-civ-spouse   \n",
       "\n",
       "           occupation   relationship   race     sex  capital.gain  \\\n",
       "2729   Prof-specialty           Wife  White  Female          7298   \n",
       "6850              NaN  Not-in-family  White    Male             0   \n",
       "414      Tech-support        Husband  White    Male             0   \n",
       "7887     Craft-repair      Own-child  White    Male             0   \n",
       "23438    Adm-clerical           Wife  Black  Female             0   \n",
       "\n",
       "       capital.loss  hours.per.week native.country income  \n",
       "2729              0               8            NaN   >50K  \n",
       "6850              0              25  United-States  <=50K  \n",
       "414            1977              40  United-States   >50K  \n",
       "7887              0              40  United-States  <=50K  \n",
       "23438             0              35          Haiti  <=50K  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "adult_df_large = pd.read_csv(\"data/adult.csv\")\n",
    "train_df, test_df = train_test_split(adult_df_large, test_size=0.6, random_state=42)\n",
    "train_df_nan = train_df.replace(\"?\", np.NaN)\n",
    "test_df_nan = test_df.replace(\"?\", np.NaN)\n",
    "train_df_nan.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "numeric_features = [\"age\", \"fnlwgt\", \"capital.gain\", \"capital.loss\", \"hours.per.week\"]\n",
    "categorical_features = [\n",
    "    \"workclass\",\n",
    "    \"marital.status\",\n",
    "    \"occupation\",\n",
    "    \"relationship\",\n",
    "    \"native.country\",\n",
    "]\n",
    "ordinal_features = [\"education\"]\n",
    "binary_features = [\"sex\"]\n",
    "drop_features = [\"race\", \"education.num\"]\n",
    "target_column = \"income\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "education_levels = [\n",
    "    \"Preschool\",\n",
    "    \"1st-4th\",\n",
    "    \"5th-6th\",\n",
    "    \"7th-8th\",\n",
    "    \"9th\",\n",
    "    \"10th\",\n",
    "    \"11th\",\n",
    "    \"12th\",\n",
    "    \"HS-grad\",\n",
    "    \"Prof-school\",\n",
    "    \"Assoc-voc\",\n",
    "    \"Assoc-acdm\",\n",
    "    \"Some-college\",\n",
    "    \"Bachelors\",\n",
    "    \"Masters\",\n",
    "    \"Doctorate\",\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "assert set(education_levels) == set(train_df[\"education\"].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "numeric_transformer = make_pipeline(StandardScaler())\n",
    "\n",
    "ordinal_transformer = make_pipeline(\n",
    "    OrdinalEncoder(categories=[education_levels], dtype=int)\n",
    ")\n",
    "\n",
    "categorical_transformer = make_pipeline(\n",
    "    SimpleImputer(strategy=\"constant\", fill_value=\"missing\"),\n",
    "    OneHotEncoder(handle_unknown=\"ignore\", sparse=False),\n",
    ")\n",
    "\n",
    "binary_transformer = make_pipeline(\n",
    "    SimpleImputer(strategy=\"constant\", fill_value=\"missing\"),\n",
    "    OneHotEncoder(drop=\"if_binary\", dtype=int),\n",
    ")\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (numeric_transformer, numeric_features),\n",
    "    (ordinal_transformer, ordinal_features),\n",
    "    (binary_transformer, binary_features),\n",
    "    (categorical_transformer, categorical_features),\n",
    "    (\"drop\", drop_features),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "X_train = train_df_nan.drop(columns=[target_column])\n",
    "y_train = train_df_nan[target_column]\n",
    "\n",
    "X_test = test_df_nan.drop(columns=[target_column])\n",
    "y_test = test_df_nan[target_column]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Do we have class imbalance? \n",
    "\n",
    "- There is class imbalance. But without any context, both classes seem equally important. \n",
    "- Let's use accuracy as our metric. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<=50K    0.75906\n",
       ">50K     0.24094\n",
       "Name: income, dtype: float64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df_nan[\"income\"].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "scoring_metric = \"accuracy\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's store all the results in a dictionary called `results`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = {}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Baselines"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "####  `DummyClassifier` baseline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "dummy = DummyClassifier(strategy=\"stratified\")\n",
    "results[\"Dummy\"] = mean_std_cross_val_scores(\n",
    "    dummy, X_train, y_train, return_train_score=True, scoring=scoring_metric\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### `DecisionTreeClassifier` baseline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Let's try decision tree classifier on our data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Dummy</th>\n",
       "      <td>0.003 (+/- 0.000)</td>\n",
       "      <td>0.002 (+/- 0.000)</td>\n",
       "      <td>0.638 (+/- 0.004)</td>\n",
       "      <td>0.634 (+/- 0.003)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Decision tree</th>\n",
       "      <td>0.066 (+/- 0.002)</td>\n",
       "      <td>0.007 (+/- 0.000)</td>\n",
       "      <td>0.811 (+/- 0.002)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        fit_time         score_time         test_score  \\\n",
       "Dummy          0.003 (+/- 0.000)  0.002 (+/- 0.000)  0.638 (+/- 0.004)   \n",
       "Decision tree  0.066 (+/- 0.002)  0.007 (+/- 0.000)  0.811 (+/- 0.002)   \n",
       "\n",
       "                     train_score  \n",
       "Dummy          0.634 (+/- 0.003)  \n",
       "Decision tree  1.000 (+/- 0.000)  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_dt = make_pipeline(preprocessor, DecisionTreeClassifier(random_state=123))\n",
    "results[\"Decision tree\"] = mean_std_cross_val_scores(\n",
    "    pipe_dt, X_train, y_train, return_train_score=True, scoring=scoring_metric\n",
    ")\n",
    "pd.DataFrame(results).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Decision tree is clearly overfitting. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Random forests "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### General idea\n",
    "\n",
    "- A single decision tree is likely to overfit\n",
    "- Use a collection of diverse decision trees\n",
    "- Each tree overfits on some part of the data but we can reduce overfitting by averaging the results \n",
    "    - can be shown mathematically"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `RandomForestClassifier` \n",
    "\n",
    "- Before understanding the details let's first try it out.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Dummy</th>\n",
       "      <td>0.003 (+/- 0.000)</td>\n",
       "      <td>0.002 (+/- 0.000)</td>\n",
       "      <td>0.638 (+/- 0.004)</td>\n",
       "      <td>0.634 (+/- 0.003)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Decision tree</th>\n",
       "      <td>0.066 (+/- 0.002)</td>\n",
       "      <td>0.007 (+/- 0.000)</td>\n",
       "      <td>0.811 (+/- 0.002)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Random forests</th>\n",
       "      <td>0.469 (+/- 0.407)</td>\n",
       "      <td>0.028 (+/- 0.002)</td>\n",
       "      <td>0.852 (+/- 0.004)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         fit_time         score_time         test_score  \\\n",
       "Dummy           0.003 (+/- 0.000)  0.002 (+/- 0.000)  0.638 (+/- 0.004)   \n",
       "Decision tree   0.066 (+/- 0.002)  0.007 (+/- 0.000)  0.811 (+/- 0.002)   \n",
       "Random forests  0.469 (+/- 0.407)  0.028 (+/- 0.002)  0.852 (+/- 0.004)   \n",
       "\n",
       "                      train_score  \n",
       "Dummy           0.634 (+/- 0.003)  \n",
       "Decision tree   1.000 (+/- 0.000)  \n",
       "Random forests  1.000 (+/- 0.000)  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "\n",
    "pipe_rf = make_pipeline(\n",
    "    preprocessor, RandomForestClassifier(random_state=123, n_jobs=-1)\n",
    ")\n",
    "results[\"Random forests\"] = mean_std_cross_val_scores(\n",
    "    pipe_rf, X_train, y_train, return_train_score=True, scoring=scoring_metric\n",
    ")\n",
    "pd.DataFrame(results).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "The validation scores are better although it seems likes we are still overfitting. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### How do they work? \n",
    "\n",
    "- Decide how many decision trees we want to build\n",
    "    - can control with `n_estimators` hyperparameter \n",
    "- `fit` a diverse set of that many decision trees by **injecting randomness** in the classifier construction\n",
    "- `predict` by voting (classification) or averaging (regression) of predictions given by individual models"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Inject randomness in the classifier construction\n",
    "\n",
    "To ensure that the trees in the random forest are different we inject randomness in two ways:  \n",
    "\n",
    "1. Data: **Build each tree on a bootstrap sample** (i.e., a sample drawn **with replacement** from the training set)\n",
    "2. Features: **At each node, select a random subset of features** (controlled by `max_features` in `scikit-learn`) and look for the best possible test involving one of these features   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "An example of a bootstrap samples\n",
    "Suppose this is your original dataset: [1,2,3,4]\n",
    "- a sample drawn with replacement: [1,1,3,4]\n",
    "- a sample drawn with replacement: [3,2,2,2]\n",
    "- a sample drawn with replacement: [1,2,4,4]\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{seealso}\n",
    "There is also something called [`ExtraTreesClassifier`](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.ExtraTreesClassifier.html), where we add more randomness by consider a random subset of features at each split and **random threshold**. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### The random forests classifier \n",
    "\n",
    "- Create a collection (ensemble) of trees. Grow each tree on an independent bootstrap sample from the data.\n",
    "- At each node:\n",
    "    - Randomly select a subset of features out of all features (independently for each node).\n",
    "    - Find the best split on the selected features. \n",
    "    - Grow the trees to maximum depth.\n",
    "    \n",
    "- Prediction time    \n",
    "    - Vote the trees to get predictions for new example. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example \n",
    "\n",
    "- Let's create a random forest with 3 estimators. \n",
    "- I'm using `max_depth=2` for easy visualization. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "pipe_rf_demo = make_pipeline(\n",
    "    preprocessor, RandomForestClassifier(max_depth=2, n_estimators=3, random_state=123)\n",
    ")\n",
    "pipe_rf_demo.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's get the feature names of transformed features."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['age',\n",
       " 'fnlwgt',\n",
       " 'capital.gain',\n",
       " 'capital.loss',\n",
       " 'hours.per.week',\n",
       " 'education',\n",
       " 'sex',\n",
       " 'x0_Federal-gov',\n",
       " 'x0_Local-gov',\n",
       " 'x0_Never-worked']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "feature_names = (\n",
    "    numeric_features\n",
    "    + ordinal_features\n",
    "    + binary_features\n",
    "    + list(\n",
    "        pipe_rf_demo.named_steps[\"columntransformer\"]\n",
    "        .named_transformers_[\"pipeline-4\"]\n",
    "        .named_steps[\"onehotencoder\"]\n",
    "        .get_feature_names_out()\n",
    "    )\n",
    ")\n",
    "feature_names[:10]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's sample a test example. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Classes:  ['<=50K' '>50K']\n",
      "Prediction by random forest:  ['<=50K']\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>0</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>age</th>\n",
       "      <td>0.471457</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>fnlwgt</th>\n",
       "      <td>-0.541792</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>capital.gain</th>\n",
       "      <td>-0.146334</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>capital.loss</th>\n",
       "      <td>-0.216958</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>hours.per.week</th>\n",
       "      <td>-0.042543</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x4_Trinadad&amp;Tobago</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x4_United-States</th>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x4_Vietnam</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x4_Yugoslavia</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>x4_missing</th>\n",
       "      <td>0.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>86 rows × 1 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                           0\n",
       "age                 0.471457\n",
       "fnlwgt             -0.541792\n",
       "capital.gain       -0.146334\n",
       "capital.loss       -0.216958\n",
       "hours.per.week     -0.042543\n",
       "...                      ...\n",
       "x4_Trinadad&Tobago  0.000000\n",
       "x4_United-States    1.000000\n",
       "x4_Vietnam          0.000000\n",
       "x4_Yugoslavia       0.000000\n",
       "x4_missing          0.000000\n",
       "\n",
       "[86 rows x 1 columns]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_example = X_test.sample(1)\n",
    "print(\"Classes: \", pipe_rf_demo.classes_)\n",
    "print(\"Prediction by random forest: \", pipe_rf_demo.predict(test_example))\n",
    "transformed_example = preprocessor.transform(test_example)\n",
    "pd.DataFrame(data=transformed_example.flatten(), index=feature_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We can look at different trees created by random forest. \n",
    "- Note that each tree looks at different set of features and slightly different data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "Tree 1\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 6.0.1 (0)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"638pt\" height=\"224pt\"\n",
       " viewBox=\"0.00 0.00 638.00 224.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 220)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-220 634,-220 634,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"415.5,-216 218.5,-216 218.5,-163 415.5,-163 415.5,-216\"/>\n",
       "<text text-anchor=\"middle\" x=\"317\" y=\"-200.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1_Married&#45;civ&#45;spouse &lt;= 0.5</text>\n",
       "<text text-anchor=\"middle\" x=\"317\" y=\"-185.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [9951, 3073]</text>\n",
       "<text text-anchor=\"middle\" x=\"317\" y=\"-170.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"304,-127 136,-127 136,-74 304,-74 304,-127\"/>\n",
       "<text text-anchor=\"middle\" x=\"220\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1_Never&#45;married &lt;= 0.5</text>\n",
       "<text text-anchor=\"middle\" x=\"220\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [6570, 384]</text>\n",
       "<text text-anchor=\"middle\" x=\"220\" y=\"-81.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.42,-162.87C278.35,-153.83 266.89,-143.55 256.26,-134.03\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"258.49,-131.32 248.71,-127.25 253.82,-136.53 258.49,-131.32\"/>\n",
       "<text text-anchor=\"middle\" x=\"250.07\" y=\"-148.51\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"505.5,-127 322.5,-127 322.5,-74 505.5,-74 505.5,-127\"/>\n",
       "<text text-anchor=\"middle\" x=\"414\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x2_Exec&#45;managerial &lt;= 0.5</text>\n",
       "<text text-anchor=\"middle\" x=\"414\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [3381, 2689]</text>\n",
       "<text text-anchor=\"middle\" x=\"414\" y=\"-81.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M345.58,-162.87C355.65,-153.83 367.11,-143.55 377.74,-134.03\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"380.18,-136.53 385.29,-127.25 375.51,-131.32 380.18,-136.53\"/>\n",
       "<text text-anchor=\"middle\" x=\"383.93\" y=\"-148.51\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"140,-38 0,-38 0,0 140,0 140,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [2582, 257]</text>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M171.71,-73.91C152.96,-63.97 131.64,-52.67 113.38,-42.99\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"114.89,-39.83 104.41,-38.24 111.61,-46.01 114.89,-39.83\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"298,-38 158,-38 158,0 298,0 298,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"228\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [3988, 127]</text>\n",
       "<text text-anchor=\"middle\" x=\"228\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M222.58,-73.91C223.4,-65.74 224.31,-56.65 225.15,-48.3\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"228.65,-48.54 226.16,-38.24 221.68,-47.84 228.65,-48.54\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"479.5,-38 332.5,-38 332.5,0 479.5,0 479.5,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"406\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [3053, 2000]</text>\n",
       "<text text-anchor=\"middle\" x=\"406\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M411.42,-73.91C410.6,-65.74 409.69,-56.65 408.85,-48.3\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"412.32,-47.84 407.84,-38.24 405.35,-48.54 412.32,-47.84\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"630,-38 498,-38 498,0 630,0 630,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"564\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [328, 689]</text>\n",
       "<text text-anchor=\"middle\" x=\"564\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 1.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>4&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M462.29,-73.91C481.04,-63.97 502.36,-52.67 520.62,-42.99\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"522.39,-46.01 529.59,-38.24 519.11,-39.83 522.39,-46.01\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.sources.Source at 0x1218b1810>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prediction [0.]\n",
      "\n",
      "\n",
      "Tree 2\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 6.0.1 (0)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"616pt\" height=\"224pt\"\n",
       " viewBox=\"0.00 0.00 615.50 224.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 220)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-220 611.5,-220 611.5,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"374,-216 227,-216 227,-163 374,-163 374,-216\"/>\n",
       "<text text-anchor=\"middle\" x=\"300.5\" y=\"-200.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">education &lt;= 12.5</text>\n",
       "<text text-anchor=\"middle\" x=\"300.5\" y=\"-185.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [9841, 3183]</text>\n",
       "<text text-anchor=\"middle\" x=\"300.5\" y=\"-170.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"292,-127 145,-127 145,-74 292,-74 292,-127\"/>\n",
       "<text text-anchor=\"middle\" x=\"218.5\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1_Separated &lt;= 0.5</text>\n",
       "<text text-anchor=\"middle\" x=\"218.5\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [8247, 1746]</text>\n",
       "<text text-anchor=\"middle\" x=\"218.5\" y=\"-81.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M276.34,-162.87C267.99,-154.01 258.51,-143.95 249.68,-134.58\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"252.18,-132.13 242.77,-127.25 247.08,-136.93 252.18,-132.13\"/>\n",
       "<text text-anchor=\"middle\" x=\"242.03\" y=\"-148.54\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"457,-127 310,-127 310,-74 457,-74 457,-127\"/>\n",
       "<text text-anchor=\"middle\" x=\"383.5\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x3_Husband &lt;= 0.5</text>\n",
       "<text text-anchor=\"middle\" x=\"383.5\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [1594, 1437]</text>\n",
       "<text text-anchor=\"middle\" x=\"383.5\" y=\"-81.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M324.95,-162.87C333.41,-154.01 343,-143.95 351.94,-134.58\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"354.56,-136.9 358.93,-127.25 349.5,-132.07 354.56,-136.9\"/>\n",
       "<text text-anchor=\"middle\" x=\"359.52\" y=\"-148.54\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"147,-38 0,-38 0,0 147,0 147,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"73.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [7895, 1738]</text>\n",
       "<text text-anchor=\"middle\" x=\"73.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M171.82,-73.91C153.77,-64.01 133.27,-52.77 115.67,-43.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"117.22,-39.98 106.77,-38.24 113.85,-46.12 117.22,-39.98\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"282,-38 165,-38 165,0 282,0 282,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"223.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [352, 8]</text>\n",
       "<text text-anchor=\"middle\" x=\"223.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M220.11,-73.91C220.62,-65.74 221.2,-56.65 221.72,-48.3\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"225.22,-48.44 222.35,-38.24 218.23,-48 225.22,-48.44\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"449.5,-38 309.5,-38 309.5,0 449.5,0 449.5,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"379.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [1191, 378]</text>\n",
       "<text text-anchor=\"middle\" x=\"379.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M382.21,-73.91C381.8,-65.74 381.34,-56.65 380.92,-48.3\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"384.42,-48.05 380.42,-38.24 377.42,-48.4 384.42,-48.05\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"607.5,-38 467.5,-38 467.5,0 607.5,0 607.5,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"537.5\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [403, 1059]</text>\n",
       "<text text-anchor=\"middle\" x=\"537.5\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 1.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>4&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M433.07,-73.91C452.42,-63.92 474.42,-52.56 493.22,-42.86\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"494.89,-45.94 502.17,-38.24 491.68,-39.72 494.89,-45.94\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.sources.Source at 0x1218b37c0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prediction [0.]\n",
      "\n",
      "\n",
      "Tree 3\n"
     ]
    },
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 6.0.1 (0)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"631pt\" height=\"224pt\"\n",
       " viewBox=\"0.00 0.00 631.00 224.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 220)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-220 627,-220 627,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"400,-216 232,-216 232,-163 400,-163 400,-216\"/>\n",
       "<text text-anchor=\"middle\" x=\"316\" y=\"-200.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1_Never&#45;married &lt;= 0.5</text>\n",
       "<text text-anchor=\"middle\" x=\"316\" y=\"-185.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [9890, 3134]</text>\n",
       "<text text-anchor=\"middle\" x=\"316\" y=\"-170.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"321.5,-127 124.5,-127 124.5,-74 321.5,-74 321.5,-127\"/>\n",
       "<text text-anchor=\"middle\" x=\"223\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x1_Married&#45;civ&#45;spouse &lt;= 0.5</text>\n",
       "<text text-anchor=\"middle\" x=\"223\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [5928, 2976]</text>\n",
       "<text text-anchor=\"middle\" x=\"223\" y=\"-81.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M288.6,-162.87C279.04,-153.92 268.17,-143.75 258.07,-134.3\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"260.22,-131.53 250.53,-127.25 255.44,-136.64 260.22,-131.53\"/>\n",
       "<text text-anchor=\"middle\" x=\"251.36\" y=\"-148.54\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"480,-127 340,-127 340,-74 480,-74 480,-127\"/>\n",
       "<text text-anchor=\"middle\" x=\"410\" y=\"-111.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">x0_Private &lt;= 0.5</text>\n",
       "<text text-anchor=\"middle\" x=\"410\" y=\"-96.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [3962, 158]</text>\n",
       "<text text-anchor=\"middle\" x=\"410\" y=\"-81.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>0&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M343.69,-162.87C353.36,-153.92 364.35,-143.75 374.56,-134.3\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"377.22,-136.61 382.18,-127.25 372.46,-131.47 377.22,-136.61\"/>\n",
       "<text text-anchor=\"middle\" x=\"381.21\" y=\"-148.53\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"140,-38 0,-38 0,0 140,0 140,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [2509, 239]</text>\n",
       "<text text-anchor=\"middle\" x=\"70\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M173.75,-73.91C154.53,-63.92 132.67,-52.56 113.99,-42.86\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"115.59,-39.74 105.1,-38.24 112.36,-45.96 115.59,-39.74\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"305.5,-38 158.5,-38 158.5,0 305.5,0 305.5,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"232\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [3419, 2737]</text>\n",
       "<text text-anchor=\"middle\" x=\"232\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>1&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M225.9,-73.91C226.82,-65.74 227.85,-56.65 228.8,-48.3\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"232.29,-48.57 229.94,-38.24 225.33,-47.78 232.29,-48.57\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"464.5,-38 339.5,-38 339.5,0 464.5,0 464.5,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"402\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [908, 52]</text>\n",
       "<text text-anchor=\"middle\" x=\"402\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>4&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M407.42,-73.91C406.6,-65.74 405.69,-56.65 404.85,-48.3\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"408.32,-47.84 403.84,-38.24 401.35,-48.54 408.32,-47.84\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"623,-38 483,-38 483,0 623,0 623,-38\"/>\n",
       "<text text-anchor=\"middle\" x=\"553\" y=\"-22.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">counts = [3054, 106]</text>\n",
       "<text text-anchor=\"middle\" x=\"553\" y=\"-7.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0.0</text>\n",
       "</g>\n",
       "<!-- 4&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>4&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M456.03,-73.91C473.83,-64.01 494.05,-52.77 511.41,-43.12\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"513.15,-46.16 520.19,-38.24 509.75,-40.04 513.15,-46.16\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.sources.Source at 0x1218b18a0>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "prediction [0.]\n"
     ]
    }
   ],
   "source": [
    "for i, tree in enumerate(\n",
    "    pipe_rf_demo.named_steps[\"randomforestclassifier\"].estimators_\n",
    "):\n",
    "    print(\"\\n\\nTree\", i + 1)\n",
    "    display(display_tree(feature_names, tree, counts=True))\n",
    "    print(\"prediction\", tree.predict(preprocessor.transform(test_example)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Some important hyperparameters:\n",
    "\n",
    "- `n_estimators`: number of decision trees (higher = more complexity)\n",
    "- `max_depth`: max depth of each decision tree (higher = more complexity)\n",
    "- `max_features`: the number of features you get to look at each split (higher = more complexity)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "\n",
    "### Random forests: number of trees (`n_estimators`) and the fundamental tradeoff "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlgAAAHCCAYAAAAzc7dkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMywgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/NK7nSAAAACXBIWXMAAA9hAAAPYQGoP6dpAABhZElEQVR4nO3dd3hUVf7H8fekTXojtJBCQCkrEAhVgkgRRSwIKCi6Cv5siwqiqwi7yKooLFZE0XVXxQJLU1xBLHRBOgSQ3gKEgEACJCGNZHJ/fwwZGVIIyYSZJJ/X8+Qhc+85d74zXpkP9545x2QYhoGIiIiIOIybswsQERERqW4UsEREREQcTAFLRERExMEUsEREREQcTAFLRERExMEUsEREREQcTAFLRERExMEUsEREREQczMPZBdRUBQUFHDt2jICAAEwmk7PLERERkTIwDIOMjAzCw8Nxcyv5OpUClpMcO3aMyMhIZ5chIiIi5ZCUlERERESJ+xWwnCQgIACw/gcKDAx0cjUiIiJSFunp6URGRto+x0uigOUkhbcFAwMDFbBERESqmMsN79EgdxEREREHU8ASERERcTAFLBEREREHU8ASERERcTAFLBEREREHU8ASERERcTCXDVh79uxhypQpDBkyhJYtW+Lh4YHJZGL8+PEVOu7ixYvp06cPYWFh+Pj40KxZM/72t79x7ty5Uvvt37+fIUOGEBERgdlsJiIigiFDhnDw4MEK1SMiIiLVj8sGrA8//JDhw4fz+eefs337diwWS4WP+c4779CrVy9+/PFHrrvuOu644w7S0tJ4/fXXadeuHSkpKcX2+/XXX4mNjeXzzz8nODiYfv36ERwczOeff06rVq1Yu3ZthWsTERGR6sNlA1aLFi3461//yvTp09m1axd//vOfK3S8hIQEnnvuOdzd3fn+++9ZsWIFs2fP5sCBA/Ts2ZM9e/bwxBNPFOmXlZXFwIEDycrKYvTo0Wzfvp2ZM2eyfft2Ro8eTWZmJgMHDiQ7O7tC9YmIiEj14bIzuT/yyCN2j0tbULEsJkyYgGEYDB06lFtvvdW23dfXl08++YRGjRrx9ddfs3v3bpo1a2bbP23aNI4dO0aTJk2K3J4cP348X3/9NXv37uWLL77g8ccfr1CNIiIiUj247BUsRzp//jzff/89AIMHDy6yPzo6mvj4eADmzZtnt6/w8b333lsk5Lm5uTFo0CAAvvnmG4fXLSIiIlVTjQhYe/fuJSsrC4B27doV26Zwe0JCgt32wsdX2k9ERERqLpe9RehIiYmJAAQHB5e4+nVkZKRdW4CMjAxSU1MBiIqKKrXfqVOnyMzMxM/Pz2F1i4jUVIZhYCkwKDCgwPa7QUEBWIzC3637LbbfL2pnUHofWztrH8MAA6OYOkqrsYTtpbymEo91hc9Rcg+52HXhQUSG+jrluWtEwMrIyAAoNfz4+/sDkJ6eXqRfaX0L+xX2Laldbm4uubm5dm1FRJytoMAgK89CZm7+hR8Lmeetv5/LzSfrvMVu+7ncfLJy8zmXayHrfD7ZeRYKCowLIYciIcf2uIQwZFzYdunvIo7wer+WDO5Y/AWSylYjApYrmDBhAi+//LKzyxCRKi7PUmANPOeLD0X2+y78eb6YdhfaZJ2v+BQ4zmIygbvJhJubCbfC3y88dr+wzc1U+LsJN7cLjy/q42ay31fs81yuiCvoU0Lzy/QpR10CQJi/l9Oeu0YErMLbgpmZmSW2KZxoNDAwsEi/0vpePEHpxX0vNXr0aJ599lnb4/T0dNvtRRGpOfIsBew4lk7ymexyhaLz+QWVUpebCfzMHvibPfD1cr/wpwd+Zg/8zO4l7HPHx9PdGmAuhBj3i4NMYbAxcdHvF4WfEvoUBifr7yWEJVPJwUPEFdSIgNWwYUMAzp49S0ZGRrHjsJKSkuzagjVghYaGcvr0aY4cOUJsbGyJ/cLCwkq9BWk2mzGbzRV4FSJSFeXmW9h2NI11B1NZl3iaTYfPOOSqkZeHG35e9sHHz+yBXwmh6I99fwQkf7MHvhcemz3cFFhEHKhGBKymTZvi6+tLVlYWGzdupHv37kXabNy4EYC4uDi77XFxcSxevJiNGzdyxx13lLmfiNRMOXkWNh85w7qDp1mXmErCkbPkXnLVKdjXkyZ1AmwhqDAU+Zvd8TVfCEjFhSKzB/5e1lDk6V4jvgQuUmXViIDl5eXFbbfdxpw5c5gxY0aRgHX48GFWr14NQL9+/ez29evXj8WLFzNz5kzGjRtnNxdWQUEBs2bNAqB///6V/CpExBVl5uaz6fAZ1iWmsj7xNFuT0jhvsQ9UYf5edIgJpWNMLTo2CqVJnQDc3HS1SKQ6q1YB6/333+f999+nQ4cOfPHFF3b7XnzxRebOnctnn33GgAED6N27N2BdCuf//u//sFgsDBgwwG4Wd4AhQ4bw2muvsXfvXsaOHctrr71m2zd27Fj27t1LREQEDz74YOW/QBFxuvScPDYeOs26xNOsO3ia7clp5F/yrbe6gWZbmOoYU4vGtf10+02khnHZgLV582aGDRtme3zgwAEA/vWvf7FgwQLb9nnz5lG/fn0AUlJS2LNnD/Xq1StyvLi4ON566y2effZZ+vTpw4033kidOnVYuXIlx48fp2nTpnz00UdF+vn6+jJ79mxuvvlmXn/9db777jtatGjB9u3b2b59O35+fsyZMwcfHx9HvwUi4gLOZJ5n/aHTrE+03vLbeSydS2cRaBDsQ8dGoXSKqUWHmFCia/kqUInUcC4bsNLT01m3bl2R7UePHuXo0aO2xxfPLXU5I0eOpGXLlrz11lusX7+ezMxMoqKiGD16NKNHjy5xEtL4+Hi2bt3Kq6++yuLFi/n666+pXbs2Dz74IC+99BKNGze+8hcoIi4p5VyuNUxdGJS++/eMIm0a1vKl44Uw1bFRKBEhzpnIUERcl8kobWpZqTTp6ekEBQWRlpZW6vQOIlK5TqTnsPZCmFp3MJUDp4pOyXJNHf8LY6ist/zqBXk7oVIRcQVl/fx22StYIiKV4eiZLNs3/NYlnuZwalaRNs3qBVjDVCPrVaowf02xIiJXRgFLRKotwzA4nJplDVMHrQPTk89m27VxM8GfwgOtg9JjQukQE0qwr/NmfxaR6kEBS0SqDcMwOHDqHGsvhKn1iamcSLcfp+nuZqJlgyDboPS2DUMI9PZ0UsUiUl0pYIlIlVVQYLDnRAbrDqbavumXcu68XRsvdzdiI4Nsg9LbRofgZ9ZffSJSufS3jIhUGZYCg53H0m3jpzYcOs3ZrDy7NmYPN+KiQujYyHq7Ly4qBG9PdydVLCI1lQKWiLisPEsBvyWn2aZN2HjoDBm5+XZtfL3caRsdYhuU3ioiCLOHApWIOJcCloi4jLIsjBxg9qBdwxA6NrIOSm/RIEjr8omIy1HAEhGnKevCyO0bWueg6tSoFs3rB+KudfxExMUpYInIVXPxwsjrDp5m69Gz5Fns5zqu5edlW8NPCyOLSFWlgCUilepQSiaLd51gya6TbDh0+jILI4fSuLa/1vETkSpPAUtEHMpSYLD5yBlbqNp/8pzdfi2MLCI1gQKWiFRYRk4ev+xNYcmuEyzbc5IzF02d4OFmokNMKD2b16Vnszo0DPNzYqUiIleHApaIlEvS6SyW7DrBkt0nWXsw1W4sVZCPJ92a1qZn87rc2KQ2QT6aKV1EahYFLBEpE0uBwZaks9ZQteske05k2O1vFOZHz+Z16Nm8Lu2iQ/DQ1AkiUoMpYIlIiTJz81m57xSLd51k2e6TpGb+sQyNu5uJdtEh3NS8Lj2b16FRbX8nVioi4loUsETEzrGz2SzZdYLFu06y5kAq5y1/zEsV4O3BjU1qc1PzunRrWptgXy8nVioi4roUsERquIICg23JabZQtet4ut3+6Fq+9GxWl5ua16F9TKhmTRcRKQMFLJEaKPu8hVX7U2yD1E9l5Nr2uZmgbXQIPZtbQ5XmpRIRuXIKWCI1xO9pOSzZbR2g/uv+FLslafzNHnRtEkbPZnXp3qwOoX669SciUhEKWCLVlGEY7DiWzqKdJ1iy+wTbk+1v/UWE+NgGqHeMqYWXh279iYg4igKWSDWSk2dh9YEUFu86ydJdJ/k9Pce2z2SCNpHBF2791aVJXd36ExGpLApYIlXcyYwclu46yeILt/6y8yy2fb5e7txwbRg9m9elR7M6hPmbnVipiEjNoYAlUsUYhsGu4xnWb/3tPsnWpLN2+8ODvK3L0jSvQ6dGtfD2dHdOoSIiNZgClkgVkJtvYc2BVJbsOsnS3SdJPptttz82IujCeKq6NK8foFt/IiJOpoAl4sLO5xfwxk+7mb7uCFnn/7j15+3pRpdranNT8zr0aFaHOoHeTqxSREQupYAl4qKSz2bz5PTNbLlwC7BuoNk2N1XnxmG69Sci4sIUsERc0LI9Jxk5awtns/II8vFk0t2tuPlPdXXrT0SkilDAEnEhlgKDdxfvZcrS/QC0igjig8FxRIb6OrkyERG5EgpYIi7iVEYuI2YmsPpAKgB/7hTN329vjtlDtwJFRKoaBSwRF7DuYCpP/zeBkxm5+Hq5M6F/S/q2buDsskREpJwUsEScyDAM/vXLQd74aQ+WAoNr6/jz4QNxXFMnwNmliYhIBShgiThJWlYez83ZyuJdJwDo16YBr/Vrga+X/rcUEanq9De5iBP8djSNYTM2kXQ6Gy93N/5x53Xc1yFS3xIUEakmFLBEriLDMJi+7givzN/JeUsBkaE+fHh/W1o0CHJ2aSIi4kAKWCJXSWZuPn+b9xvfbjkGQK8/1eXNe2IJ8vF0cmUiIuJoClgiV8H+kxk88dVm9p88h7ubiVG9m/LoDY10S1BEpJpSwBKpZP/bkszob34j67yFOgFm3h8cR4eYUGeXJSIilUgBS6SS5OZbeHXBTr5aewSA+GtqMfneNoT5m51cmYiIVDYFLJFKkHQ6i2HTN/NbchomEzzd/RpG3NQEdzfdEhQRqQkUsEQcbPHOEzw7ewvpOfmE+HryzqDWdGtax9lliYjIVaSAJeIg+ZYC3vh5D/9acRCANlHBfDA4jvBgHydXJiIiV5sClogDnEzP4an/JrA+8TQAQ+MbMvrW5nh5uDm5MhERcQYFLJEKWn0gheH/3ULKuVz8zR5MursVfVrWd3ZZIiLiRApYIuVUUGDw4YoDvPXzHgoMaFYvgKn3x9Gotr+zSxMRESdTwBIphzOZ53l29haW7TkFwD1tI3ilbwt8vNydXJmIiLgCBSyRK5Rw5AxPzUgg+Ww2Zg83Xu3bgoHtI51dloiIuBAFLJEyMgyDL9YcZvz3O8mzGDSs5cvU+9vyp/BAZ5cmIiIuRgFLpAzO5eYz6uttfL/tOAC3tqjHP+9uRaC3FmoWEZGiFLBELmP37+kM+2ozB1My8XAzMaZPc4bGN9RCzSIiUiIFLJFSzN10lL9/+xs5eQXUD/Lm/cFxtI0OcXZZIiLi4hSwRIqRk2fhH9/tYOaGJAC6NqnNu4NaE+rn5eTKRESkKlDAErnE4dRM/vLVZnYeT8dkgpE3NeGp7tfgpoWaRUSkjBSwRC7y4/bjPD9nGxm5+dTy82LyvW3ocm2Ys8sSEZEqRgFLBMizFPDPH3bzn1WJALSLDuH9wXHUC/J2cmUiIlIVKWBJjXc8LZunZiSw6fAZAB7r2ojnb2mKp7sWahYRkfJRwJIabeW+U4yYuYXTmecJ8PbgzXtiueW6es4uS0REqjgFLKmRLAUG7y3Zx3tL92EYcF14IFPvjyO6lp+zSxMRkWpAAUtqnNRzuTwzawsr96UAcF+HKMbd8Se8PbVQs4iIOIYCltQomw6f5snpCfyenoOPpzuv9WtB/7gIZ5clIiLVjAKW1AiGYfDJqkQm/rCb/AKDRrX9+OiBtjSpG+Ds0kREpBpSwJJqLz0nj+fnbOWnHScAuCM2nAn9W+Jv1ukvIiKVQ58wUq3tOJbGsOmbOZyahae7ibG3/4k/d4rWQs0iIlKpFLCkWjIMg9kbkxj7vx2czy+gQbAPH9wfR+vIYGeXJiIiNYACllQ72ect/P3b7Xy9+SgAPZrV4e2BsQT7aqFmERG5OhSwpFo5cOocw77azJ4TGbiZ4K+3NOWJro21ULOIiFxVClhSbSzYdoxRc7eRed5CmL+ZKfe14frGtZxdloiI1EAuv9janDlz6NatGyEhIfj5+REbG8ukSZPIy8u74mNlZWUxYcIEWrdujZ+fHwEBAbRv354pU6ZgsViK7bN8+XJMJlOpPx999FFFX6ZUwPn8Av7x3Q6empFA5nkLHWNCWTi8i8KViIg4jUtfwXrmmWeYPHkyHh4e9OjRA39/f5YuXcqoUaOYP38+P//8Mz4+PmU61unTp+nRowdbt24lICCA+Ph43N3dWbt2LcOHD2f+/PksWLAAL6/ix+nUrVuX3r17F7uvadOm5X6NUjFHz2Tx5IwEtiadBWBYt8Y826sJHlqoWUREnMhlA9a3337L5MmT8ff3Z8WKFcTFxQGQkpJCjx49WLVqFWPHjuXNN98s0/GeeOIJtm7dSosWLVi4cCGRkZEAnDhxgjvvvJNFixbx8ssv89prrxXbv1mzZkybNs0hr00cY9mek4yctYWzWXkE+Xjy9sBYejav6+yyREREXPcW4euvvw7Aiy++aAtXAGFhYUydOhWA999/n7S0tMse69ixY8ydOxeAKVOm2MIVWK9M/fvf/wbgnXfeISMjw2GvQSqHpcDgzZ/2MPSzDZzNyqNVRBALnu6icCUiIi7DJQNWcnIyGzZsAGDw4MFF9nfp0oXIyEhyc3NZuHDhZY+3ceNGDMPAy8uLrl27FtnfqlUrateuTXZ2dpmOJ85zKiOXP3+yjveX7Qfgz52imfPE9USG+jq5MhERkT+45C3ChIQEAEJDQ4mJiSm2Tbt27UhKSiIhIYH77ruv1OOdO3cOgODgYNzcis+UYWFhnDp1ik2bNjFo0KAi+0+cOMErr7xCcnIy3t7eNGvWjNtuu42oqKgreWlSAesOpvL0fxM4mZGLr5c7Ewe04s7YcGeXJSIiUoRLBqzExESAUsNL4W2+wralqVOnDgAnT57k3Llz+Pv72+0vKCjg8OHDpR5v9+7djBs3zm6bh4cHTz/9NJMmTcLDwyXfymqhoMDg45UHeeOnPVgKDK6t48+HD8RxTR0t1CwiIq7JJW8RFo6D8vPzK7FNYUhKT0+/7PE6duyIr6/1FtJ//vOfIvu/+OILsrKyij1eUFAQzzzzDCtWrOD48eNkZmaybds2Ro4ciclk4p133mHYsGGXrSE3N5f09HS7H7m8tKw8HvtyIxN/2I2lwKBfmwb876l4hSsREXFpLhmwHC0gIIDnnnsOgNGjR/Pee+9x/PhxTp48yX/+8x+efvppPD09AYrcQmzTpg3vvPMOXbt2pV69evj6+tKyZUvefvttZs6cCcC///1vtmzZUmoNEyZMICgoyPZz8UB7Kd5vR9O4bcpKFu86iZeHG6/3a8nbA2Px9dLVQhERcW0uGbACAqxXJzIzM0tsUziuKjAwsEzHHDduHE888QQ5OTmMGDGC8PBw6taty6OPPkpcXBwPP/wwYB33VVb9+/endevWAMyfP7/UtqNHjyYtLc32k5SUVObnqWkMw+DLtYcZ8OFqjp7JJjLUh2/+0pnBHaMwmbTkjYiIuD6XvBTQsGFDgFJDSOG+wraX4+7uzocffsiwYcP47rvvOHLkCP7+/nTr1o3bbruNBx54AICWLVteUa3Nmzdny5YtHD16tNR2ZrMZs9l8RceuiTJz8xkz7zf+t+UYAL3+VJc374klyMfTyZWJiIiUnUsGrDZt2gCQmppKYmJisd8k3LhxI4DdHFll0bJlyyIhyjAMfv31VwB69ep1RcdLTU0F/rjqJuW370QGf5m+mf0nz+HuZmJU76Y8ekMjXbUSEZEqxyVvEUZERNC+fXsAZsyYUWT/qlWrSEpKwmw206dPnwo/3+zZszly5AjXX389bdu2LXO/5ORkVq5cCUCHDh0qXEdN9r8tydz5/q/sP3mOOgFm/vtoJx7r2ljhSkREqiSXDFgAY8aMAWDixIls3rzZtj01NdX2rb2nnnqKoKAg27558+bRrFkzevbsWeR4x44dK/aW44IFC3jssccwm83FLto8efJkUlJSimzftm0bd9xxB9nZ2TRu3Ji+ffte+YsUcvIs/G3eb4yYuYXsPAvx19Ri4Ygb6BBT9rFwIiIirsYlbxEC3HXXXQwfPpz33nuPTp060bNnT/z8/FiyZAlnz54lPj6eV1991a5PWloae/bsIScnp8jx1q9fT//+/YmNjSUmJgZPT0+2bdvG7t278ff359tvv6VVq1ZF+o0bN47nnnuO1q1bExMTg5ubGwcOHCAhIYGCggKioqKYP3++xleVQ9LpLP4yfRPbk9MxmeDp7tcw4qYmuLvpqpWIiFRtLhuwwHr1KD4+ng8++IDVq1eTl5dH48aNefHFFxk5ciReXl5lPlaLFi148MEHWbNmDYsWLcJisRAVFcXIkSN57rnnaNCgQbH9/va3v/Hrr7+yY8cOFi1aRGZmJoGBgXTu3Jm+ffvy+OOPa/xVOSzaeYLnZm8hPSefEF9P3hnUmm5N6zi7LBEREYcwGYZhOLuImig9PZ2goCDS0tLKPNVEdZBvKeCNn/fwrxUHAWgTFcwHg+MID/ZxcmUiIiKXV9bPb5e+giXVy4n0HJ6ekcD6Q6cBGBrfkNG3NsfLw2WHAoqIiJSLApZcFav3pzB8ZgIp587jb/Zg0t2t6NOyvrPLEhERqRQKWFKpCgoMpi7fz9uL9lJgQLN6AUy9P45Gtf0v31lERKSKUsCSSnMm8zwjZ29h+Z5TANzTNoJX+rbAx8vdyZWJiIhULgUsqRQJR87w1IwEks9mY/Zw49W+LRjYXgtci4hIzaCAJQ5lGAafrz7Eawt3kWcxaFjLl6n3t+VP4TXnm5IiIiIKWOIwGTl5vPj1b3z/23EAbm1Rj3/e3YpAby3ULCIiNYsCljjE7t/TGfbVZg6mZOLhZmJMn+YMjW+otQRFRKRGUsCSCpu76Sh///Y3cvIKqB/kzfuD42gbHeLsskRERJxGAUvKLSfPwrj/7WDWRusi2l2b1ObdQa0J9Sv7EkYiIiLVkQKWlMuhlEz+Mn0zu45bF2oeeVMTnup+DW5aqFlEREQBS67cj9uP8/ycbWTk5lPLz4vJ97ahy7Vhzi5LRETEZShgSZnlWQqY+MNuPlmVCEC76BDeHxxHvSBvJ1cmIiLiWhSwpEyOp2Xz1IwENh0+A8BjXRvx/C1N8XTXQs0iIiKXUsCSy/pl7ymembWF05nnCfD24M17YrnlunrOLktERMRlKWBJiSwFBu8t2cd7S/dhGHBdeCBT748jupafs0sTERFxaQpYUqzUc7k8M2sLK/elAHBfhyjG3fEnvD21ULOIiMjlKGBJERsPneapGQn8np6Dj6c7r/VrQf+4CGeXJSIiUmUoYImNYRh8siqRiT/sJr/AoFFtPz56oC1N6gY4uzQREZEqRQFLAEjPyeP5OVv5accJAO6IDWdC/5b4m3WKiIiIXCl9ego7jqUxbPpmDqdm4eluYuztf+LPnaK1ULOIiEg5KWDVYIZhMGtDEi99t4Pz+QU0CPZh6v1xxEYGO7s0ERGRKk0Bq4bKPm/h799u5+vNRwHo0awObw+MJdhXCzWLiIhUlAJWDXTg1DmGfbWZPScycDPBX29pyhNdG2uhZhEREQdRwKphFmw7xqi528g8byHM38yU+9pwfeNazi5LRESkWlHAqiFy8y28/v0uPl9zGICOMaFMua8NdQK1ULOIiIijKWDVAEfPZPHkjAS2Jp0FYFi3xjzbqwkeWqhZRESkUihgVXPL9pxk5KwtnM3KI8jHk7cHxtKzeV1nlyUiIlKtKWBVU/mWAt5ZvJcPlh0AoFVEEB8MjiMy1NfJlYmIiFR/CljV0MmMHEb8dwtrDqYC8OdO0fz99uaYPbRQs4iIyNWggFXNrDuYylP/TeBURi6+Xu5MHNCKO2PDnV2WiIhIjaKAVY0YhsGUpfs5lZHLtXX8+fCBOK6po4WaRURErjYFrGrEZDLx9qBYpi47wAu9m+Lrpf+8IiIizqBP4GqmToA3/7jzOmeXISIiUqNpIiQRERERB1PAEhEREXEwBSwRERERB1PAEhEREXEwBSwRERERB1PAEhEREXEwBSwRERERB1PAEhEREXEwBSwRERERB1PAEhEREXGwSlkqJzExkW3bthEdHU3r1q0r4ylEREREXFa5r2B999139O/fn/Xr19ttf+ONN2jSpAn9+/enbdu2PPzwwxUuUkRERKQqKXfA+uKLL/jxxx9p3ry5bdvu3bt58cUXMQyD2NhYfH19+fzzz5k/f75DihURERGpCsodsBISEoiNjSUgIMC2bfr06QBMnTqVzZs3s2HDBtzd3fn4448rXqmIiIhIFVHugJWSkkKDBg3sti1fvhwfHx+GDBkCQLNmzejSpQs7duyoUJEiIiIiVUm5A1ZOTg7u7u62xxaLhc2bN9OxY0e8vLxs28PDw/n9998rVqWIiIhIFVLugFWnTh327dtne7x27Vqys7OJj4+3a5ednY2fn1/5KxQRERGpYsodsDp37szWrVuZOXMmaWlpvP7665hMJm666Sa7drt27SI8PLzChYqIiIhUFeUOWKNGjcLDw4P777+f0NBQfvjhB+Li4ujatautTVJSErt376Z9+/YOKVZERESkKih3wIqLi2PhwoXceOONNG/enCFDhrBgwQK7NrNnzyYoKIiePXtWuFARERGRqsJkGIbh7CJqovT0dIKCgkhLSyMwMNDZ5YiIiEgZlPXzW2sRioiIiDhYhdciTE9P56uvvmL16tWcOnWKnj178sILLwCwZ88eDh8+TNeuXfH29q5wsSIiIiJVQYUC1s8//8zgwYM5c+YMhmFgMpnsJh/du3cvd911F//9738ZOHBghYsVERERqQrKfYtw165d9OvXj7S0NP7yl78wa9YsLh3Odcstt+Dr68v//ve/ChcqIiIiUlWU+wrW66+/Tk5ODnPmzKF///4ADBo0yK6Nl5cXrVu3ZuvWrRWrUkRERKQKKfcVrGXLlhEbG2sLVyWJiIjg+PHj5X0aERERkSqn3AHr1KlTNGnS5LLt8vPzyczMLO/TiIiIiFQ55Q5YQUFBJCcnX7bdwYMHqVOnTnmfRkRERKTKqdBM7ps2beLIkSMlttm+fTtbt26lY8eO5X0aERERkSqn3AHrkUceIScnh/vuu4/ff/+9yP6UlBQeeeQRDMPgkUceqVCRIiIiIlVJuQPW3XffzT333MOaNWto3LgxN998MwC//vord955J40aNWL9+vUMHjyYW265xWEFi4iIiLi6Cq1FaLFYeOmll3j33XfJzs622+fl5cXTTz/NxIkTcXd3r3Ch1Y3WIhQREal6rspahO7u7rz22mscPXqUuXPnMmnSJCZOnMj06dM5cuQIb7zxRoXD1Zw5c+jWrRshISH4+fkRGxvLpEmTyMvLu+JjZWVlMWHCBFq3bo2fnx8BAQG0b9+eKVOmYLFYSu27f/9+hgwZQkREBGazmYiICIYMGcLBgwfL+9JERESkmir3FayHH36YsLAwJk2a5OiabJ555hkmT56Mh4cHPXr0wN/fn6VLl3L27Fm6dOnCzz//jI+PT5mOdfr0aXr06MHWrVsJCAigU6dOuLu7s3btWs6ePUuvXr1YsGABXl5eRfr++uuv3HzzzWRlZXHdddfRokULtm/fzo4dO/Dz82Px4sV06tTpil6brmCJiIhUPWX+/DbKydPT07j77rvL2/2y5s2bZwCGv7+/sWnTJtv2U6dOGS1btjQA47nnnivz8e655x4DMFq0aGEcOXLEtv333383OnToYADGmDFjivTLzMw0wsPDDcAYPXq03b7Ro0cbgBEZGWlkZWVd0etLS0szACMtLe2K+omIiIjzlPXzu9y3COvVq4fJZCpv98t6/fXXAXjxxReJi4uzbQ8LC2Pq1KkAvP/++6SlpV32WMeOHWPu3LkATJkyhcjISNu+unXr8u9//xuAd955h4yMDLu+06ZN49ixYzRp0oTx48fb7Rs/fjxNmjQhKSmJL774ohyvUkRERKqjcgesXr168euvv5ZrLNTlJCcns2HDBgAGDx5cZH+XLl2IjIwkNzeXhQsXXvZ4GzduxDAMvLy86Nq1a5H9rVq1onbt2mRnZxc53rx58wC49957cXOzf7vc3Nxs6y9+8803ZXtxIiIiUu2VO2D94x//IDc3l0cffbTIVZ+KSkhIACA0NJSYmJhi27Rr186ubWnOnTsHQHBwcJGQVCgsLAyATZs2FVtL4fNVpA4RERGpGTzK2/Gzzz6jd+/efPHFF3z//ffcdNNNNGzYsNhB5yaTibFjx5b52ImJiQBERUWV2KbwNl9h29IULtVz8uRJzp07h7+/v93+goICDh8+XOR4GRkZpKamllpLYR2nTp0iMzMTPz+/y9YjIiIi1Vu5A9Y//vEP2xis1NRUZs2aVaSNyWTCMIwrDliFV8RKCyuFISk9Pf2yx+vYsSO+vr5kZWXxn//8h2eeecZu/xdffEFWVlaR4118Za6kWi4Oa+np6SW2y83NJTc3166tiIiIVE/lDlgvvfRSpQ5yd6SAgACee+45Xn31VUaPHo2bmxv33HMP7u7ufPfdd4wcORJPT0/y8vJKvIVYURMmTODll1+ulGOLiIiIa6nQFazKEhAQAEBmZmaJbQrHVZV1Dqlx48Zx6tQpPvroI0aMGMGIESNs+7p27Urz5s3517/+RWhoaJE6SqulsI7L1TJ69GieffZZ2+P09HS7bzOKiIhI9VHugFWZGjZsCEBSUlKJbQr3Fba9HHd3dz788EOGDRvGd999x5EjR/D396dbt27cdtttPPDAAwC0bNnS1icgIIDQ0FBOnz7NkSNHiI2NLbGOsLCwUm9pms1mzGZzmWoVERGRqs1hAev333/n6NGjADRo0ID69euX+1ht2rQBrGO7EhMTi/0m4caNGwHs5sgqi5YtW9qFKADDMPj1118B6/QTF4uLi2Px4sVs3LiRO+64w2F1iIiISPVV4QFHn3zyCc2aNaNBgwZ07NiRjh07EhERQfPmzfn000/LdcyIiAjat28PwIwZM4rsX7VqFUlJSZjNZvr06VOh+gFmz57NkSNHuP7662nbtq3dvn79+gEwc+ZMCgoK7PYVFBTYBvf379+/wnWIiIhINVGR6eIfeeQRw83NzTCZTIbJZDJq1apl1KpVy/bYzc3NeOSRR8p17JKWyklJSSlxqZxvvvnGaNq0qdGjR48ix0tOTrZbIqfQ/PnzjcDAQMNsNhtbt24tsv/ipXIuXUpnzJgxBmBERERoqRwREZEaoKyf3+UOWLNnzzZMJpMRGhpqvPXWW8bZs2ftnvztt982atWqZbi5uRlz5swp13MMHz7cAAxPT0+jd+/exoABA4zg4GADMOLj44uEms8++8wAjOjo6CLHmjdvnmEymYzWrVsb/fr1MwYOHGg0a9bMFuJ++OGHEutYtWqV4evra1vL8N577zVatGhhAIafn5+xZs2aK35tClgiIiJVT6UHrJ49exqenp52V5cutXnzZsPT09O46aabyvs0xqxZs4yuXbsagYGBho+Pj9GiRQtj4sSJRm5ubpG2pQWsffv2GQ899JDRpEkTw9/f3/Dx8TGaNm1qjBw50jh69Ohl69i3b5/x4IMPGuHh4Yanp6cRHh5uPPjgg8b+/fvL9boUsERERKqesn5+mwzDMMpza7FWrVrExcWxaNGiUtv16tWLzZs322ZEF6v09HSCgoJIS0sr81QTIiIi4lxl/fwu9yD3zMxM2xI0palTp06p81mJiIiIVDflDlj16tUr0wLHCQkJ1K1bt7xPIyIiIlLllDtgde/enT179jBx4sQS20yYMIE9e/bQs2fP8j6NiIiISJVT7jFYe/bsoU2bNuTm5tKuXTsefPBB24SgBw8e5PPPP2fz5s14e3uzefNmmjZt6tDCqzqNwRIREal6yvr5Xe6Z3Js2bcqcOXO4//772bBhg21G80KGYRAYGMj06dMVrkRERKRGqdBSObfddht79+7l448/ZsWKFSQnJwPWpXK6devGo48+WqaB8CIiIiLVSblvEUrF6BahiIhI1VPp0zSIiIiISPHKHbB27NjBK6+8UupUDZs3b+aVV15h9+7d5X0aERERkSqn3AFr6tSpvPLKK4SFhZXYJiwsjJdffpmPPvqovE8jIiIiUuWUO2AtX76cVq1aERkZWWKbqKgoYmNjWbJkSXmfRkRERKTKKXfAOnr0KI0aNbpsu0aNGtm+XSgiIiJSE5Q7YOXn5+Pmdvnubm5u5OTklPdpRERERKqccgesyMhINmzYcNl2GzZsIDw8vLxPIyIiIlLllDtg9ejRgyNHjjB16tQS23z44YccPnyYHj16lPdpRERERKqccgeskSNH4uXlxfDhwxk5ciQ7d+7EYrFgsVjYuXMnI0eOZPjw4Xh5efHss886smYRERERl1ahmdxnzJjB0KFDyc/PL7LPMAw8PDz45JNP+POf/1yhIqsjzeQuIiJS9VyVmdwHDx7MmjVruPPOO/H19cUwDAzDwMfHh759+7J69WqFKxEREalxHLYWYUFBAampqQDUqlWrTN8wrMl0BUtERKTqKevnt4ejntDNzY3atWsDsGLFCrZs2UJ0dDR33nmnwpaIiIjUKOVOPtOmTSMuLo5Vq1bZbX/qqafo0aMHzz77LAMGDKB3795YLJYKFyoiIiJSVZQ7YM2dO5cDBw7Qvn1727aNGzcydepUvL296du3Lw0aNGDJkiXMnDnTIcWKiIiIVAXlDljbt2+nZcuWmM1m27aZM2diMpn48ssv+eabb1i/fj3e3t58+umnDilWREREpCood8BKTU0lIiLCbtsvv/xCYGAgd911FwD16tXjhhtuYP/+/RUqUkRERKQqKXfAysvLsxtblZuby9atW+ncubPdoPbatWtz8uTJilUpIiIiUoWUO2CFh4ezY8cO2+MVK1aQl5dH586d7doVfp1RREREpKYod8Dq1q0be/bsYeLEiWzdupVx48ZhMpno3bu3Xbvt27cXuZUoIiIiUp2VO2CNGTMGf39//va3vxEXF8e6deu46aabaNu2ra3N3r17SUxMpFOnTg4pVkRERKQqKPdEo9dccw2rV6/mrbfe4uTJk3To0IHnn3/ers2SJUuIjY3ltttuq3ChIiIiIlWFw5bKkSujpXJERESqnquy2LOIiIiIFKWAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDqaAJSIiIuJgClgiIiIiDubyAWvOnDl069aNkJAQ/Pz8iI2NZdKkSeTl5V3xsTIzM5kwYQLt2rUjMDAQT09P6tWrx+233853331XbJ/ly5djMplK/fnoo48q+jJFRESkGvFwdgGleeaZZ5g8eTIeHh706NEDf39/li5dyqhRo5g/fz4///wzPj4+ZTpWamoqXbt2ZefOnfj7+9O5c2eCg4PZv38/33//Pd9//z3Dhw9n8uTJxfavW7cuvXv3LnZf06ZNy/0aRUREpJwKLJCXBeczL/ycu+j3TKgfC6ExTinNZQPWt99+y+TJk/H392fFihXExcUBkJKSQo8ePVi1ahVjx47lzTffLNPxXnnlFXbu3Enbtm35+eefCQ0Nte1buHAhffv25b333uO+++6jU6dORfo3a9aMadOmOeS1iYiI1CiGAfk5cD7rkhB00e95mSUHpfMl7MvPLv1573hPAetSr7/+OgAvvviiLVwBhIWFMXXqVG644Qbef/99xo4dS1BQ0GWPt3TpUgBGjRplF64A+vTpQ/fu3Vm0aBFr1qwpNmCJiEg1VlAAJpP1p6az5P8RYvJKCUSlBZ+8YvYZBZVXs8kNvALAyw+8fC/86Q9+YZX3nJfhkgErOTmZDRs2ADB48OAi+7t06UJkZCRJSUksXLiQ++6777LH9Pb2LtNzh4U57z+GiEiNYRiQnwuWXMg/f+HP3GK2nbde+bh0myXXur3ItuKOUbjt/EX7LtlmWKx1mdzBzeOiH/dLfi9t/yWPTcW1L63PpX9epk9JtVrOFw09tttoxYWlS/ZZciv3v72nr/WnMAR5+V30439RSPIvZvuF3y/t72F2uXDskgErISEBgNDQUGJiir+0165dO5KSkkhISChTwLr11lvZuHEj//znP+nZs2eRW4TLli2jXr163HnnncX2P3HiBK+88grJycl4e3vTrFkzbrvtNqKiosrxCkVEXExBgfVD+OLxLIW/52VZP4TzMu3/LLxFc2lYuTjEFNl2IfRYzjv7FRfPsIDFUvkhoypw8yg+3Hj5lRCQLg1LxWz39LUGwRrAJQNWYmIiQKnhJTIy0q7t5YwaNYr169fz008/ER0dTXx8vG2Q+6ZNm4iPj+eTTz4p8Xbj7t27GTdunN02Dw8Pnn76aSZNmoSHh0u+lSJSnRQUFA05l4ag8+dKDkRFwtNF+y43lqWyuXuBuxk8LvrTw9u63cN8yb4LP3b7Lt124U+7Y5Syzd3LelWtIP+iH4v9Y8NSdFupjy1laHPRNqO4vqU9Lm5bnvX9uPRWWbFhp4QQVPjYw8u550QV55KpICMjAwA/P78S2/j7+wOQnp5epmP6+fkxf/58xowZw1tvvcVPP/1k21erVi1uuukmGjRoUKRfUFAQzzzzDP369aNJkyYEBgZy4MABPvvsM95//33eeecdzp07x8cff1zq8+fm5pKb+8e/iMpat4hUMxm/Q9I6SDtaTAgqITRd7RDkeeHDufAqhafvhcd+f3xw29r4XAgsl4Qj9wtBpsg2c9HA5O4Fbi4/a5DIFXHJgFUZjh8/Tt++fdm2bRvjx4/nvvvuo06dOuzcuZO///3vvPzyy3z77besXLmSgIAAW782bdrQpk0bu2O1bNmSt99+my5dujBgwAD+/e9/M2zYMFq3bl3i80+YMIGXX365sl6eiLgiw4CUvXBkDRxZa/05U7ar7pdVbAjyKyYQ+Raz79I/L2rr4aOwI+IALhmwCgNOZmZmiW3OnTsHQGBgYJmO+dBDD7FhwwYmTZrE888/b9vevn17FixYQNu2bdm6dStvvvlmmYNQ//79ad26NVu2bGH+/PmlBqzRo0fz7LPP2h6np6fbbnOKSDWRnwvHtkDS2j8CVfbpSxqZoG4LqN3kots0vva/lxSCCgf3evq43IBeEbHnkgGrYcOGACQlJZXYpnBfYdvSJCcns2jRIoBiB8R7enpy991389tvv7F48eIrutLUvHlztmzZwtGjR0ttZzabMZvNZT6uiFQB2WcgacMfV6iSNxUdHO3hAxHtIKqT9SeiPXhffmoZEanaXDJgFd6SS01NJTExsdhvEm7cuBHAbo6skhw5csT2e0lXvAoHt58+fem/NkuXmpoKYHdbUUSqIcOAtKQLV6YuBKqTO4u28w37I0xFXQ/1WmmwsEgN5JIBKyIigvbt27NhwwZmzJjB3/72N7v9q1atIikpCbPZTJ8+fS57vIsHr69bt45evXoVabN27VqAEqeFKE5ycjIrV64EoEOHDmXuJyJVQIEFTuywD1QZx4q2q3UNRF4UqGo11u07EXHNgAUwZswY+vXrx8SJE7n11lttV6pSU1MZNmwYAE899ZTdtArz5s1j9OjRNGjQgCVLlti2R0VF2QLbiBEjWLhwod2txa+++opZs2YBRSc2nTx5Mvfff3+RCUi3bdvGkCFDyM7OpnHjxvTt29ehr19ErrLzmdZbfIVjp5LWw/kM+zZuHta1zaKutwaqyI7gX8c59YqISzMZhmE4u4iSjBgxgvfeew9PT0969uyJn58fS5Ys4ezZs8THx7No0SK7xZ6nTZvG0KFDiY6O5tChQ3bH2r59O927dyclJQVvb286depEWFgYu3btYseOHQA88MADfPHFF5gu+tdncHAw586do3Xr1sTExODm5saBAwdISEigoKCAqKgofvzxR5o3b35Fry09PZ2goCDS0tLKPFBfRBzo3MkLQWqd9QrV8a3WuYQu5hUAkR3+CFQN2loHm4tIjVXWz2+XvYIF1qtH8fHxfPDBB6xevZq8vDwaN27Miy++yMiRI/HyKvu4hhYtWrB9+3beeecdfvjhBzZs2EBubi4hISHccsstPPzwwwwcOLBIv7/97W/8+uuv7Nixg0WLFpGZmUlgYCCdO3emb9++PP744xp/JeLqDANSD1w0XcIaOH2gaLuAcIi+3hqoIjtC3etqzKzTIuJYLn0FqzrTFSyRSpR/Hn7fZj//VFbKJY1MUOdPFw1I7wRBkRo/JSKlqhZXsEREyiQnreh0CZfOeu5utk6XENnxwhWq9uAT4px6RaTaU8ASkaon7ehF3+5bBye2A5dcjPcJtZ8uoX6sdYkWEZGrQAFLRFxbgQVO7rKfHT2tmEmIQ2L+GIwedT2EXavbfSLiNApYIuJa8rIhefMft/uS1kNumn0bkzvUb/XHYPSoThBQzzn1iogUQwFLRJwrM/XC1akLgerYFijIs2/j5W9dYqbwll+DdmD2d0q5IiJloYAlIlePYcDpg3+Mn0paByl7i7bzr/fHrb6oTtbFkd3115WIVB36G0tEHC/7DJxOhDOJ1j8Lf0/ZC5mnirav3cw+UAVHa/yUiFRpClgicuUMA86dsF6NsgWpi37PPlNyX3cvCI/7I1BFdgDf0KtXu4jIVaCAJSLFs+Rbv613+mDRK1FnDkFeVun9/epAaCMIjbF+wy80xvq4bgvw9L4qL0FExFkUsERqsrxsa1iyuxJ14WpUWlLRtfkuZnKDoAhraCoMUCEXQlRIQw1CF5EaTQFLpLorHA9luxJ16I9behnHS+/rbraGpeKuRAVFgkfZ1wMVEalJFLBEqjrDgIzfi46DKgxVOWdL728OgtCGxV+JCqgPbm5X41WIiFQrClgiVYElH9KOFP/NvNOJRdfdu5R/3T9C08UBKjTGuh6fvrEnNVxeXh4Wi8XZZchV5ObmhqenJ6ZK+vtPAUvElf3+Gyx9DfYvKsN4qMg/bt9dfCsvpCF4+V21kkWqkvT0dFJSUsjNzXV2KeIE7u7u+Pr6UqdOHby8HDvkQQFLxBWlHoBlr8P2r7EtYuzhbQ1LRa5ExWg8lEg5pKenk5ycjL+/P2FhYZV6NUNci2EYWCwWsrOzSUtL49ChQ0RERODr6+uw51DAEnEl6cdhxT8h4cs/rli1GABdn4ewphoPJeJAKSkp+Pv7ExERoWBVQ/n7+xMaGsrhw4dJSUkhKirKYcdWwBJxBVmnYdU7sP5jyM+xbrumF/QcC/VjnVubSDWUl5dHbm4uYWFhClc1nLu7O6GhoRw/fpz8/Hw8PBwTjRSwRJwp9xys+xB+fQ9y063bIjvBTeMgurNzaxOpxgoHtHt6ejq5EnEFZrMZQAFLpMrLz4VN0+CXN/5Ym69uS+j5ElzbS9/qE7lKdPVKoHLOAwUskaupwALbZsGyCdZpF8A6UL3H3+G6/hpjJSJSTShgiVwNhgG7F8DS8XBqt3VbQH248QVo82dw120KEZHqRP9cFqlsB1fAf3rCrAes4co7GHq9AsMToN3DClciIsVYvnw5JpOJbt26ObuUctEVLJHKkrwJlrwCB5dbH3v6wfXDoPPT4B3k1NJERMqqYcOGHD58mMTERBo2bOjscqoMBSwRRzu1B5a+CrvmWx+7eVqvVHX9K/jXcW5tIiJVRIcOHdi1a5dDJ/+8mhSwRBzl7BFYPhG2/heMAuvyNa3uhW4vQki0s6sTEalSfH19adasmbPLKDeNwRKpqHOn4IdRMKUtbJluDVfNboe/rIF+HypciUiVNG3aNEwmE4cPHwYgJiYGk8lk+1m+fLndOKmsrCxeeuklmjdvjq+vr93txPXr1/PCCy/QoUMH6tWrh5eXF3Xr1uWOO+5g8eLFxT5/SWOwDh06hMlkomHDhhiGwccff0zbtm3x8/MjKCiIm2++mTVr1lTW21JmuoIlUl45abB6CqyZCnmZ1m0xN0LPcRDR1rm1iYhU0DXXXMNDDz3E3LlzyczMZMCAAfj7+9v216tXj99//x2AnJwcunXrxs6dO+natSuxsbGkpqba2o4ZM4Zly5Zx3XXX2cLQgQMHWLBgAQsWLODdd99lxIgRV1zj0KFDmTFjBjfccAO33347W7ZsYdGiRfzyyy+sWLGCjh07VvyNKCcFLJErlZdtXdJm1TuQfca6LTzOOklo4+7OrU1ExEG6dOlCly5dWL58OZmZmbz55ptFBrkXBqx169bRqlUr9u/fT7169Yoc67nnnuPLL7+kfv36dtvXrFlD7969ef7557n77rtp0KBBmes7fPgwy5cvZ/v27TRp0gSwztD/2GOP8emnn/LSSy/x008/XeGrdhwFLJGysuRBwlewYhJkHLNuC2tqnSS0+R2afV2kGjEMg+w8i7PLKBcfT3enzFD//vvvFxuuAG699dZit19//fU8+eSTTJgwgf/9738MGzbsip5zypQptnAF1nUFX3vtNT799FNWrFhBXl6e05ZDUsASuZyCAtjxDSx7DU4ftG4LioTuY6DVIHBzd259IuJw2XkW/vSS865+VMTOV27B1+vqfrzXqVOHG264odQ2qampfP/992zfvp0zZ86Ql5cHwL59+wDYs2fPFT2nh4cHvXv3LrK9Xr16hISEcObMGVJTU0sMfZVNAUukJIYB+xbB0lfg99+s23zDoOvz0G4oeJidW5+IiIu43PxY//73vxk5ciSZmZkltklPT7+i56xfv36JV6cCAwM5c+YMOTk5V3RMR1LAEinO4TWw5GU4cuGbKOZA6DwcOv0FzP6l9xWRKs/H052dr9zi7DLKxcfz6l9V9/HxKXHfpk2bePzxx3F3d+ef//wnd9xxB1FRUfj6+mIymfj44495/PHHMQzjip7TzcXXblXAErnY77/Bkldh34VbAx7e0OEx6DISfEOdW5uIXDUmk+mq32arrubMmYNhGDz99NO88MILRfYX3iKsbnT2iACkHoBlr8P2udbHJneIe9C6GHNguHNrExFxIi8vLwDy8/PL1f/06dMAREcXnRMwJyeHr7/+uvzFuTDXvr4mUtnSj8P8Z+CDDn+EqxYD4KkNcMe7ClciUuNFREQAsGPHjnL1b968OQCff/45GRkZtu05OTkMGzaMxMTEihfpgnQFS2qmrNPWeazWfwz5FwZBXnsz9BgL9Vs5tzYRERcyYMAAli1bxgMPPMDNN99MSEgIAM8//3yZ+g8dOpTJkyeTkJBATEwMN9xwA+7u7qxcuZLs7GxGjBjB5MmTK/MlOIUCltQsuedg7Yew+j3IvfCNlajrrZOERnd2bm0iIi7oL3/5CxkZGXz11VcsXLjQ9s28Bx54oEz9g4OD2bhxI+PGjeOnn37ihx9+oFatWtx8882MGzeOVatWVWb5TmMyrnTYvjhEeno6QUFBpKWlERgY6Oxyqr/8XNj4Gax8EzJPWbfVbWkNVtf20iShIjVMTk4OiYmJxMTE4O3t7exyxMmu5Hwo6+e3rmBJ9VZggW2zYNkESDti3RYSY519/br+4OJf8xURkapJAUuqJ8OA3Qtg6Xg4tdu6LaC+9VuBbf4M7s5ZOkFERGoGBSypfg4uhyWvQPIm62OfEOs8Vh0eA8+SJ8MTERFxFAUsqT6SN8HilyFxhfWxpx9cPww6Pw3eQc6tTUREahQFLKn6Tu6Gpa9abwkCuHtBu4fhhufAv45zaxMRkRpJAUuqrrNHYPlE2PpfMArA5Aax90G3FyE4ytnViYhIDaaAJVXPuVPW6RY2fgqW89ZtzW63ThJap5lzaxMREUEBS6qSnDRYPQXWTIW8TOu2mBuh5ziIaOvc2kRERC6igCWuLy/buqTNqncg+4x1W3gc3DQOGnVzamkiIiLFUcAS12XJg4QvYcUkyDhu3RbWFHqOtd4S1OzrIiLiohSwxPUUFMCOb2DZa3D6oHVbUCR0HwOtBoGbu3PrExERuQwFLHEdhgH7FlknCT3xm3Wbbxh0fR7aDQUPs3PrExERKSMFLHENh9fAkpfhyBrrY3MgdB4Onf4CZn/n1iYiInKFFLDEuY5vs04Suu9n62MPb+uSNl1Ggm+oc2sTEREpJwUscY7UA9YxVtu/tj42uUPcg9bFmAPDnVubiIhIBSlgydWVfsz6rcDNX4BhsW5rcbd1AHutxs6tTURExEEUsOTqyDptncdq/ceQn2Pddu0t1ikX6rV0bm0iIiIOpoAllSv3HKz9EFa/B7np1m1R11tnX4++3rm1iYiIVBIFLKkc+bmw8TPrmoGZp6zb6raEni/Btb00SaiIiFRrbs4uQKqZAgskTIcp7eDHUdZwFdoIBnwCj/8CTW5WuBIRqUKysrJ499136dKlCyEhIZjNZqKjo7njjjuYMWMGFouFiIgITCYTa9euLfE4f/3rXzGZTIwcOfIqVu88uoIljmEYsGs+LB0PKXus2wLqw42joM0D4O7p3PpEROSKJSUl0bt3b3bu3Imvry/x8fHUqlWL5ORkVq5cyW+//cbgwYN58MEHmTBhAtOmTaNTp05FjpOfn89XX30FwMMPP3y1X4ZTKGBJxR1cDotfhmObrY99QqzzWHV4DDx9nFqaiEi5GAbkZTm7ivLx9HXInYKCggL69+/Pzp07ufnmm/nqq6+oXbu2bX9OTg5Lly4FYOjQoUyYMIFZs2bx7rvv4u3tbXeshQsXcuLECdq2bUvLljXji00KWFJ+RzdZZ19PXGF97OkH1w+Dzk+Dd5BzaxMRqYi8LHi9is7JN+YYePlV+DDz589n48aN1K9fn6+//hp/f/tVNby9venTpw8A1157LTfccAMrV65k3rx53HfffXZtP/vsM8AaxGoKjcGSK3dyN8y8H/7Twxqu3L2g4xMwYgv0+LvClYhINfDjjz8CMHjw4CLhqjiF4WnatGl220+dOsX333+P2Wxm8ODBDq/TVekKlpTdmcOwfCJsmwlGAZjcIPY+6PYiBEc5uzoREcfx9LVeCaqKPH0dcpjDhw8D0KxZszK1HzhwIMOHD2fx4sUcPXqUiIgIAL766ivy8vIYNGgQISEhDqmtKlDAkss7dxJ+eRM2fgoFedZtze+A7n+HOmX7H09EpEoxmRxym60m8fPzY+DAgXz66ad88cUXjBkzBvjjilZNuj0IukUopclJs34rcHJrWP8va7hq1A0eXQqDvlK4EhGpxqKirHcmdu/eXeY+l94m3Lx5M9u2bSMiIoJevXo5vEZXpoAlReVlw6+TYXIs/PIG5GVCg7bw4P+sPw3aOrtCERGpZL179wbgv//9L5mZmWXq06VLF5o0acK+ffv49ddfbYPbH3roIdzcalbkcPlXO2fOHLp160ZISAh+fn7ExsYyadIk8vLyrvhYmZmZTJgwgXbt2hEYGIinpyf16tXj9ttv57vvviu17/79+xkyZAgRERGYzWYiIiIYMmQIBw8eLO9Lcz2WPOttwPfawKKXIPsM1G5mvVr1yBLr1SsREakR7rzzTtq0acOxY8e45557SE1Ntdufk5PDDz/8UKRf4VWsjz76iBkzZgAwZMiQSq/X1ZgMwzCcXURJnnnmGSZPnoyHhwc9evTA39+fpUuXcvbsWbp06cLPP/+Mj0/Z5llKTU2la9eu7Ny5E39/fzp37kxwcDD79+9n82br/E3Dhw9n8uTJRfr++uuv3HzzzWRlZXHdddfRokULtm/fzo4dO/Dz82Px4sXFTqxWmvT0dIKCgkhLSyMwMPCK+jpcQQHs+AaWvQanLwTGoCjoPhpaDQI3d+fWJyLiYDk5OSQmJhITE1Nkzib5w+HDh7nlllvYs2cPvr6+dOnSxTbR6NatWwkODubQoUN2fY4dO0ZUVBQWiwWArl27smLFCidUX3ZXcj6U+fPbcFHz5s0zAMPf39/YtGmTbfupU6eMli1bGoDx3HPPlfl4w4cPNwCjbdu2Rmpqqt2+77//3vDw8DAAY82aNXb7MjMzjfDwcAMwRo8ebbdv9OjRBmBERkYaWVlZV/T60tLSDMBIS0u7on4OVVBgGHt+NIyp8YYxLtD6M6mxYaz9yDDycpxXl4hIJcvOzjZ27txpZGdnO7sUl5eRkWH885//NNq3b28EBAQYZrPZiI6ONu68805j5syZxfbp06ePARiA8dlnn13dgsvhSs6Hsn5+u2zAat++vQEY48ePL7Jv5cqVBmCYzWbj7NmzZTpeixYtDMCYPXt2sft79eplAMbbb79tt/2DDz4wAKNJkyaGxWKx22exWIwmTZoYgPHRRx+V8ZVZOT1gHfrVMD655Y9g9XqEYayYZBg5Gc6pR0TkKlLAkotVRsByyTFYycnJbNiwAaDYScm6dOlCZGQkubm5LFy4sEzHLOsl4LCwMLvH8+bNA+Dee+8tMkDPzc2NQYMGAfDNN9+U6fhOd3wbTL8HPrsVjqwBD2/oPBxGbIWuz4P58pPJiYiISOlcMmAlJCQAEBoaSkxMTLFt2rVrZ9f2cm699VYA/vnPf3L69Gm7fQsXLmTZsmXUq1ePO++8s9haCp+vonU4TeoBmPsw/OsG2PczuHlA26EwPAFufhV8Q51doYiISLXhkhONJiYmAn/MwVGcyMhIu7aXM2rUKNavX89PP/1EdHQ08fHxtkHumzZtIj4+nk8++YSgoD+WecnIyLB9a6KkWgrrOHXqFJmZmfj5udjEdOnHYMU/YfOXYFgHHNLibug+Bmo1dm5tIiIi1ZRLBqyMjAyAUsNK4bpI6enpZTqmn58f8+fPZ8yYMbz11lv89NNPtn21atXipptuokGDBsXWUVotF6/PlJ6eXmK73NxccnNz7dpWqqzTsOptWP9vyM+xbrv2Fug5FurVjJXMRUREnMUlbxFWhuPHjxMfH8+UKVMYP348Bw8e5Ny5c6xfv562bdvy8ssv06VLF7tQ5UgTJkwgKCjI9lN45cvhcs/Bijesk4SunmINV1GdYeiPcP9shSsREZGrwCUDVkBAAECpM8eeO3cOoMxzSD300ENs2LCBV199lTFjxhATE4Ofnx/t27dnwYIFtGzZkq1bt/Lmm28WqaO0WgrruFwto0ePJi0tzfaTlJRUprqv2IxBsGw85KZbw9T9c2HoQoi+vnKeT0RERIpwyYDVsGFDgFJDSOG+wralSU5OZtGiRQDcd999RfZ7enpy9913A7B48WLb9oCAAEJDrYO/jxw5UmodYWFhpd7SNJvNBAYG2v1Uio6PQWgjGPAJPPYLXNvLumipiIiIXDUuGbDatGkDWGdfL2kQ+8aNGwGIi4u77PEuDkclBZvCwe2XfsOw8PiFz1eROq6K5nfCk+uh5d1Qw9Z9EhG5UobrLmYiV1FlnAcu+QkcERFB+/btAWzrGF1s1apVJCUlYTab6dOnz2WPd/Hg9XXr1hXbZu3atQBFpoXo168fADNnzqSgoMBuX0FBAbNmzQKgf//+l63jqjCZwN3T2VWIiLg0d3frEmDlWddWqp/CL6F5eDjuu38uGbAAxowZA8DEiRNtawWC9arWsGHDAHjqqafsplWYN28ezZo1o2fPnnbHioqKsgW2ESNGFFk36auvvrIFpUsnNh0yZAjh4eHs3buXsWPH2u0bO3Yse/fuJSIiggcffLACr1ZERK4mT09PzGYzaWlpuopVw1ksFk6fPo2fn59DA5ZLL/Y8YsQI3nvvPTw9PenZsyd+fn4sWbKEs2fPEh8fz6JFi+wWe542bRpDhw4lOjq6SIjavn073bt3JyUlBW9vbzp16kRYWBi7du1ix44dADzwwAN88cUXmC4Zs3TxYs8tWrSwLfa8ffv26rHYs4hIDZSenk5ycjL+/v4EBQXh6elZ5O9/qZ4Mw8BisZCdnU1aWhoFBQVERkbaZYqSlPXz26UDFsDs2bP54IMP2LJlC3l5eTRu3JgHHniAkSNH4uXlZde2tIAFcOLECd555x1++OEHDhw4QG5uLiEhIcTFxfHwww8zcODAEuvYv38/r776KosXL+bUqVPUrl2bm266iZdeeonGja98wk4FLBER50tPTyclJcVunkKpOdzd3fH19aVOnTpFMkVJqk3Aqq4UsEREXEdeXh4Wi8XZZchV5ObmVq6rlmX9/HbJmdxFRESuJk9PTzw99QUhcRyXHeQuIiIiUlUpYImIiIg4mAKWiIiIiIMpYImIiIg4mAKWiIiIiIMpYImIiIg4mAKWiIiIiIMpYImIiIg4mCYadZLCCfTT09OdXImIiIiUVeHn9uUWwlHAcpKMjAwAIiMjnVyJiIiIXKmMjAyCgoJK3K+1CJ2koKCAY8eOERAQUGQdpPbt27Nhw4ZS+5fUJj09ncjISJKSkqrFGodleS+qwnNW9Jjl6X+lfSpy3pW1TXU6P3VuVqz/lfTTuXlldG5WrP/l+hmGQUZGBuHh4bi5lTzSSlewnMTNzY2IiIhi97m7u1/2f/DLtQkMDKzyf0lA2d6LqvCcFT1mefpfaR9HnHdlbVMdzk+dmxXrfyX9dG5eGZ2bFetfln6lXbkqpEHuLujJJ590SJvqwBmvszKes6LHLE//K+3jqPNO52bVek5nnJtX2k/n5pXRuVmx/o56LbpFWM2kp6cTFBREWlpalf9XmFQ/Oj/FVencFEfTFaxqxmw2M27cOMxms7NLESlC56e4Kp2b4mi6giUiIiLiYLqCJSIiIuJgClgiIiIiDqaAJezfv58+ffrg7+9PWFgYw4YNIzMz09llSQ23f/9+nnjiCeLi4vD09KRhw4bOLkkEgLlz59KvXz+ioqLw9fXluuuu46233iIvL8/ZpYkL0TxYNVxaWho9evQgPDycOXPmcPr0aZ599llOnDjB119/7ezypAbbsWMHCxYsoEOHDhiGwZkzZ5xdkggAb775Jg0bNmTSpEnUrVuX1atX8/e//51t27bx+eefO7s8cREa5F7DTZo0iXHjxnH48GHq1KkDwDfffMOAAQPYuHEjbdu2dXKFUlMVFBTYZkl+4okn+PHHHzl06JBzixIBTp06Re3ate22jR8/nrFjx/L7779Tt25dJ1UmrkS3CGu4hQsX0qNHD1u4Arjzzjvx9/dnwYIFTqxMarrSlqAQcaZLwxVg+8fosWPHrnY54qL0N5gL2rNnD1OmTGHIkCG0bNkSDw8PTCYT48ePL1P/OXPm0K1bN0JCQvDz8yM2NpZJkyYVOz5g586dNG/e3G6bh4cHTZo0YdeuXQ55PVJ9XM1zU+RKOPvc/OWXX/Dy8qJx48YVeRlSjWgMlgv68MMPmTx5crn6PvPMM0yePBkPDw969OiBv78/S5cuZdSoUcyfP5+ff/4ZHx8fW/szZ84QHBxc5DghISGcPn26vC9BqqmreW6KXAlnnps7d+5k8uTJPPbYY5oFXmx0BcsFtWjRgr/+9a9Mnz6dXbt28ec//7lM/b799lsmT56Mv78/69at46effuLrr79m3759tGzZklWrVjF27NhKrl6qM52b4qqcdW6mpKRw1113cc011zBx4kRHvRypDgxxeQ899JABGK+++mqp7dq3b28Axvjx44vsW7lypQEYZrPZOHv2rG177dq1jeeee65I+7i4OGPQoEEVL16qtco8Ny/2+OOPG9HR0Y4oWWqIq3FupqenG+3atTOio6ON5ORkh9Uu1YOuYFUTycnJbNiwAYDBgwcX2d+lSxciIyPJzc1l4cKFtu3NmzcvMtbKYrGwd+/eImOzRMqjvOemSGWryLmZm5tL3759OXToED/99BPh4eFXpWapOhSwqomEhAQAQkNDiYmJKbZNu3bt7NoC9OnTh2XLlnHq1Cnbtvnz53Pu3Dluu+22SqxYaorynpsila2856bFYuHee+9lw4YNLFy4kKZNm1Z+sVLlaJB7NZGYmAhAVFRUiW0iIyPt2gI8/vjjTJkyhb59+zJ27FjOnDnDs88+S9++fW1/sYhURHnPzaysLNtVg4MHD5KVlcXcuXMBaN++PdHR0ZVVstQQ5T03n3zySb799lteffVVLBYLa9eute3705/+pIHuAihgVRsZGRkA+Pn5ldjG398fgPT0dNu24OBgli5dyvDhw7n77rvx9vbmnnvu4c0336zcgqXGKO+5efLkSe655x67doWPP/vsM4YMGeLgSqWmKe+5+eOPPwIwduzYIgPgly1bRrdu3RxcqVRFClhCkyZNbH9hiLiKhg0bYmihCXFBWlFAykJjsKqJgIAAgFIXaT537hyALl/LVaVzU1yVzk2pTApY1UTDhg0BSEpKKrFN4b7CtiJXg85NcVU6N6UyKWBVE23atAEgNTXVbjDmxTZu3AhAXFzcVatLROemuCqdm1KZFLCqiYiICNq3bw/AjBkziuxftWoVSUlJmM1m+vTpc7XLkxpM56a4Kp2bUpkUsKqRMWPGADBx4kQ2b95s256amsqwYcMAeOqppwgKCnJKfVJz6dwUV6VzUyqLydDXdFzO5s2bbf9jAxw4cICUlBQiIiJo0KCBbfu8efOoX7++Xd8RI0bw3nvv4enpSc+ePfHz82PJkiWcPXuW+Ph4Fi1apAV1pdx0boqr0rkprkYBywUtX76c7t27X7ZdYmJisQMvZ8+ezQcffMCWLVvIy8ujcePGPPDAA4wcORIvL69KqFhqCp2b4qp0boqrUcASERERcTCNwRIRERFxMAUsEREREQdTwBIRERFxMAUsEREREQdTwBIRERFxMAUsEREREQdTwBIRERFxMAUsEREREQdTwBIRERFxMAUsEREREQdTwBKRKuvQoUOYTKZi15arbnJzcxkzZgzXXnstZrO5xrxukarKw9kFiIjI5Y0dO5Y33niDunXr0rdvX3x9fQkLC7tsv8JFkG+88UaWL19e+YWKCKCAJSJSJcyePRuAlStXcu211zq5GhG5HN0iFBGpAo4cOQKgcCVSRShgidRQJpMJk8kEwNdff02XLl0IDAzEz8+P+Ph4Fi5ceNl+xenWrRsmk6nI7aiLt69du5bbbruNWrVqERAQwI033sjKlSttbX/88Ud69uxJSEgI/v7+9OrVi82bN5f6evLz85k0aRLXXXcdPj4+hIWFMXDgQHbv3l1in+zsbN566y06depEcHAw3t7eNG3alBdeeIHU1NQi7adNm4bJZGLIkCGcPn2aZ555hsaNG2M2m+nWrVup9V3s6NGjPP3001x77bV4e3sTFBREfHw8//rXv7BYLHZtGzZsiMlkwjAM4I/332QyMW3atFKfp1u3bnTv3h2AFStW2PW9ePzWkCFDbMfbvn07gwYNon79+ri7u/OPf/zD1i4/P5///Oc/dOvWjdDQUMxmMzExMfzlL38hKSmpxDqOHTvGs88+S/PmzfH19SUgIID27dvz/vvvk5+fX6R9bm4ub7zxBm3btiUgIAAvLy/q1atH+/bteeGFFzh9+vRl3mER59MtQpEabty4cbz66qt07tyZPn36sHv3blavXs3tt9/O119/Tb9+/Rz6fN9//z3vvvsuLVu2pFevXuzZs4dffvmFXr16sXTpUhISEhg+fDidOnXi5ptvZsuWLSxevJgbb7yRhIQErrnmmmKPO2jQIObPn8+NN95Iq1atWL9+PXPmzOGHH37g559/5vrrr7drf+zYMXr37s1vv/1GaGgo7du3JyAggM2bN/PGG28wZ84cli9fTnR0dJHnSklJoV27dpw9e5YbbriBtm3b4uXlVabXv2HDBnr37s3p06eJiorirrvuIi0tjeXLl7N69WrmzZvHd999Zzve3XffTUpKCp9//jkADz30kO1YJb0XhXr37o23tzc//fQTdevWpXfv3rZ9xY3fWr16NU888QT169ena9euZGdnExAQAEBGRgZ33nkny5cvx9/fn7Zt21K7dm1+++03PvroI+bMmcOiRYto06aN3TF/+eUX7rrrLs6cOUPDhg3p1asXubm5rF+/nqeffpr58+ezYMECPD09ASgoKOC2225jyZIlBAYGcsMNNxAcHMypU6fYt28fb7zxBoMHDyY0NLRM77eI0xgiUiMBBmAEBwcba9eutds3btw4AzCaNGlSYr+S3HjjjQZgLFu2rNjtJpPJ+PLLL+32PfvsswZgNG3a1PD39zcWL15s25efn28MGDDAAIxHHnnErl9iYqKtnrCwMGPr1q12/Z5++mkDMKKjo42cnBzbvoKCAiM+Pt4AjP/7v/8z0tPTbfvy8vKM5557zgCM7t272z3fZ599Znu+nj17GmlpaSW+D8XJyckxoqOjDcB44oknjPPnz9v2HThwwGjYsKEBGGPGjCnS93Lve0mWLVtmAMaNN95YYpuHHnrIdvwXX3zRsFgsRdoMHjzYAIzbb7/dOHHihN2+d955xwCMa6+91sjPz7dtP378uFGrVi3DZDIZU6dOtTtuSkqK0aNHDwMwXn75Zdv2FStWGIDRpk0bu/8uhTZs2GCkpKRcyVsg4hQKWCI1VOEH6nvvvVdkX05OjhEUFGQAxpEjR4rtV5LLBax77rmnSJ/U1FTbcZ9//vki+zdt2mQARkxMjN32iwPWu+++W+zraNCggQEY06dPt23/4YcfDMBo3bq1kZeXV6SfxWIxWrRoYQDGb7/9ZtteGLA8PT2NAwcOlPgelOTLL780ACM8PNwu8BWaO3euARgBAQFGdna23b6rEbCaNGliF5AK7dy50zCZTEZ4eHixoccwDKNPnz4GYMyfP9+2bdSoUQZgPPXUU8X2OXr0qOHp6WnUrl3bKCgoMAzDMGbPnm0AxvDhw6/gVYq4Ho3BEqnh7rjjjiLbzGYzjRo1AiA5Odmhz9enT58i20JDQ6lVq1aJ+wsHdh87dqzE415866yQ2Wxm0KBBAHZjwr7//nsABgwYgIdH0ZESbm5udO3aFbDeNrtUmzZtbO/PlSis4d5778VsNhfZ379/f0JCQsjIyGDTpk1XfPyKuuuuu3B3dy+yfeHChRiGwa233mq7ZXipwjFoF79fhe9z4X+DSzVo0IBrr73WdvsPIC4uDnd3dz799FM++OADjh8/XpGXJOI0ClgiNVxUVFSx2wMDAwHIycm5Ks/n7+9f4v7CD/Xc3Nxi+wYHBxMcHFzsvpiYGMA6sLzQwYMHAevcUhcP/L74Z+rUqQCcOnWqyDHLO8FnYVgtrOlSJpPJts/RwbYsSnpdhe/XJ598UuL79cILLwD271dhvxtuuKHEfjt37rTr17hxY9555x3y8vJ46qmnCA8Pp2HDhtx3331Mnz6d8+fPV9bLF3EoDXIXqeHc3Bz776yCgoIKPZ+j6ylkXPgWHvxRY5cuXWjcuHGp/a677roi23x8fBxbnIso6XUVvl+tW7cmNja21GN07NixSL+7774bPz+/UvsVXsEEePrppxk4cCDfffcdq1atYtWqVcycOZOZM2cybtw4Vq5cSf369cv0mkScRQFLRK6Ip6cneXl5ZGRkFHu76PDhw1e9prNnz3L27Nlir2IdOnQIgIiICNu2yMhIAPr27ctf//rXq1EiYL0lBn9c2SlOYmKiXVtXUPh+xcfH8/77719Rv3379jFq1CjatWt3Rc9Zt25dHn30UR599FEAdu/ezcMPP8yaNWt48cUXbd+qFHFVukUoIlek8IN/165dRfZt27at1PmQKtOXX35ZZNv58+eZNWsW8McYIYBbb70VgDlz5thd2apshTXMmjWr2Fuv8+bN48yZMwQEBNC2bVuHPGfhdA/FzTdVVoXv13fffXdFt4wL+xXOQl8RzZo1Y9SoUQBs2bKlwscTqWwKWCJyRW666SYAXn75ZbsxUYcOHeKhhx66qoHlYq+++irbt2+3PS4oKGDUqFEcPXqUyMhIBgwYYNvXt29f2rdvz/r16xk6dGix46zOnDnDRx99VKFgcql77rmHqKgo28SbFx87MTGR5557DrDeIvP29nbIcxZeudu3bx95eXnlOkabNm0YMGAASUlJ9O/f33ZV8GKZmZlMnz6dEydO2LY9//zzBAcH8/bbb/PWW28VO34qMTGRr776yvZ46dKlLFy4sEithmGwYMECgGLnJhNxNbpFKCJXZMyYMcydO5eFCxfSpEkT2rdvz6lTp9iwYQPx8fF07ty52G/eVaaoqCjatm1LXFwc3bp1o1atWmzYsIEDBw7g5+fHjBkz7AKLm5sb3377Lbfddhuff/45c+fOJTY2lqioKM6fP8/Bgwf57bffsFgsDBkypNhvGpaH2Wxm7ty59O7dmw8//JCFCxfSqVMnMjIyWLp0KTk5Odxyyy2MGzfOIc8H1vemXbt2bNy4kZYtW9KuXTu8vb0JCwtj4sSJZT7OZ599xtmzZ/nhhx9o2rQpsbGxxMTEYBgGhw4dYuvWrZw/f55du3ZRt25dwBru/ve//zFgwAD++te/MmnSJFq0aEH9+vVJS0tj165dHDhwgI4dO/LAAw8A1qugI0eOJDAwkLi4OMLDw8nOzmbz5s0cPnyYoKAgXnnlFYe9PyKVxplzRIiI81DO+awMwzovUv/+/Y2QkBDDbDYbTZs2NcaPH2+cP3/+svNgFXc8wzBsE3AmJiaWud7CebCio6ONvLw847XXXjOaNWtmmM1mIzQ01BgwYICxY8eOEl9jTk6O8dFHHxndu3c3atWqZXh4eBh16tQxWrdubTz55JPGTz/9ZNe+cB6shx56qMRjlsWRI0eMJ5980mjUqJHh5eVlBAQEGNdff73x4YcfFjsvV0mvv6wOHz5sDB482Khfv77h4eFhe88KFc6D9dlnn5V6HIvFYsyYMcPo06ePUbduXcPT09OoVauW0aJFC2Po0KHGvHnz7CZPLXTixAlj7NixRlxcnBEQEGB4eXkZERERRufOnY1x48YZ27Zts7Xdv3+/8Y9//MPo2bOnERUVZXh7exshISFGq1atjBdffNFISkoq13sgcrWZDMNJ1/NFREREqimNwRIRERFxMAUsEREREQdTwBIRERFxMAUsEREREQdTwBIRERFxMAUsEREREQdTwBIRERFxMAUsEREREQdTwBIRERFxMAUsEREREQdTwBIRERFxMAUsEREREQdTwBIRERFxsP8He65XJazq3DcAAAAASUVORK5CYII=\n",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "make_num_tree_plot(\n",
    "    preprocessor, X_train, y_train, X_test, y_test, [1, 5, 10, 25, 50, 100, 200, 500]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Number of trees and fundamental trade-off\n",
    "\n",
    "- Above: seems like we're beating the fundamental \"tradeoff\" by increasing training score and not decreasing validation score much.\n",
    "- This is the promise of ensembles, though it's not guaranteed to work so nicely.\n",
    "\n",
    "More trees are always better! We pick less trees for speed. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Strengths and weaknesses\n",
    "\n",
    "- Usually one of the best performing off-the-shelf classifiers without heavy tuning of hyperparameters\n",
    "- Don't require scaling of data \n",
    "- Less likely to overfit \n",
    "- Slower than decision trees because we are fitting multiple trees but can easily parallelize training because all trees are independent of each other (that said, sklearn implementation is kind of slow)\n",
    "- In general, able to capture a much broader picture of the data compared to a single decision tree. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Weaknesses\n",
    "\n",
    "- Require more memory \n",
    "- Hard to interpret\n",
    "- Tend not to perform well on high dimensional sparse data such as text data\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{important}\n",
    "Make sure to set the `random_state` for reproducibility. Changing the `random_state` can have a big impact on the model and the results due to the random nature of these models. Having more trees can get you a more robust estimate. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{seealso}\n",
    "[The original random forests paper](https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf) by Leo Breiman. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ❓❓ Questions for you"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### (iClicker) Exercise 11.1 \n",
    "\n",
    "**iClicker cloud join link: https://join.iclicker.com/3DP5H**\n",
    "\n",
    "**Select all of the following statements which are TRUE.**\n",
    "\n",
    "- (A) Every tree in a random forest uses a different bootstrap sample of the training set.\n",
    "- (B) To train a tree in a random forest, we first randomly select a subset of features. The tree is then restricted to only using those features.\n",
    "- (C) A reasonable implementation of `predict_proba` for random forests would be for each tree to \"vote\" and then normalize these vote counts into probabilities.\n",
    "- (D) Increasing the hyperparameter max_features (the number of features to consider for a split) makes the model more complex and moves the fundamental tradeoff toward lower training error. \n",
    "- (E) A random forest with only one tree is likely to get a higher training error than a decision tree of the same depth."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "How would you carry out \"soft voting\" with `predict_proba` output instead of hard voting for random forests? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Gradient boosted trees "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Another popular and effective class of tree-based models is gradient boosted trees. \n",
    "\n",
    "- No randomization.\n",
    "- The key idea is combining many simple models called weak learners to create a strong learner. \n",
    "- They combine multiple shallow (depth 1 to 5) decision trees  \n",
    "- They build trees in a serial manner, where each tree tries to correct the mistakes of the previous one. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Important hyperparameters\n",
    "\n",
    "- `n_estimators`\n",
    "    - control the number of trees to build\n",
    "- `learning_rate`\n",
    "    - controls how strongly each tree tries to correct the mistakes of the previous trees\n",
    "    - higher learning rate means each tree can make stronger corrections, which means more complex model "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We'll not go into the details. We'll look at brief examples of using the following three gradient boosted tree models. \n",
    "\n",
    "- [XGBoost](https://xgboost.readthedocs.io/en/latest/)\n",
    "- [LightGBM](https://lightgbm.readthedocs.io/en/latest/Python-Intro.html)\n",
    "- [CatBoost](https://catboost.ai/docs/concepts/python-quickstart.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### [XGBoost](https://xgboost.ai/about) \n",
    "\n",
    "- Not part of `sklearn` but has similar interface. \n",
    "- Install it in your conda environment: `conda install -c conda-forge xgboost`\n",
    "- Supports missing values\n",
    "- GPU training, networked parallel training\n",
    "- Supports sparse data\n",
    "- Typically better scores than random forests    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### [LightGBM](https://lightgbm.readthedocs.io/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Not part of `sklearn` but has similar interface. \n",
    "- Install it in your conda environment: `conda install -c conda-forge lightgbm`\n",
    "- Small model size\n",
    "- Faster \n",
    "- Typically better scores than random forests    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### [CatBoost](https://catboost.ai/)\n",
    "\n",
    "- Not part of `sklearn` but has similar interface. \n",
    "- Install it in your conda environment: `conda install -c conda-forge catboost`\n",
    "- Usually better scores but slower compared to `XGBoost` and `LightGBM`     "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/xgboost/compat.py:36: FutureWarning: pandas.Int64Index is deprecated and will be removed from pandas in a future version. Use pandas.Index with the appropriate dtype instead.\n",
      "  from pandas import MultiIndex, Int64Index\n"
     ]
    }
   ],
   "source": [
    "from catboost import CatBoostClassifier\n",
    "from lightgbm.sklearn import LGBMClassifier\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from xgboost import XGBClassifier\n",
    "\n",
    "pipe_lr = make_pipeline(\n",
    "    preprocessor, LogisticRegression(max_iter=2000, random_state=123)\n",
    ")\n",
    "pipe_dt = make_pipeline(preprocessor, DecisionTreeClassifier(random_state=123))\n",
    "pipe_rf = make_pipeline(preprocessor, RandomForestClassifier(random_state=123))\n",
    "# pipe_xgb = make_pipeline(\n",
    "#     preprocessor, XGBClassifier(random_state=123, eval_metric=\"logloss\", verbosity=0)\n",
    "# )\n",
    "pipe_lgbm = make_pipeline(preprocessor, LGBMClassifier(random_state=123))\n",
    "pipe_catboost = make_pipeline(\n",
    "    preprocessor, CatBoostClassifier(verbose=0, random_state=123)\n",
    ")\n",
    "classifiers = {\n",
    "    \"logistic regression\": pipe_lr,\n",
    "    \"decision tree\": pipe_dt,\n",
    "    \"random forest\": pipe_rf,\n",
    "    #\"XGBoost\": pipe_xgb,\n",
    "    \"LightGBM\": pipe_lgbm,\n",
    "    \"CatBoost\": pipe_catboost,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import warnings\n",
    "\n",
    "warnings.simplefilter(action=\"ignore\", category=DeprecationWarning)\n",
    "warnings.simplefilter(action=\"ignore\", category=UserWarning)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "dummy = DummyClassifier(strategy=\"stratified\")\n",
    "results[\"Dummy\"] = mean_std_cross_val_scores(\n",
    "    dummy, X_train, y_train, return_train_score=True, scoring=scoring_metric\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "for (name, model) in classifiers.items():\n",
    "    results[name] = mean_std_cross_val_scores(\n",
    "        model, X_train, y_train, return_train_score=True, scoring=scoring_metric\n",
    "    )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Dummy</th>\n",
       "      <td>0.003 (+/- 0.000)</td>\n",
       "      <td>0.002 (+/- 0.000)</td>\n",
       "      <td>0.636 (+/- 0.008)</td>\n",
       "      <td>0.633 (+/- 0.004)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.412 (+/- 0.063)</td>\n",
       "      <td>0.008 (+/- 0.001)</td>\n",
       "      <td>0.849 (+/- 0.004)</td>\n",
       "      <td>0.851 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>0.064 (+/- 0.001)</td>\n",
       "      <td>0.007 (+/- 0.000)</td>\n",
       "      <td>0.811 (+/- 0.002)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>0.571 (+/- 0.024)</td>\n",
       "      <td>0.039 (+/- 0.001)</td>\n",
       "      <td>0.852 (+/- 0.004)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>0.115 (+/- 0.025)</td>\n",
       "      <td>0.012 (+/- 0.001)</td>\n",
       "      <td>0.868 (+/- 0.003)</td>\n",
       "      <td>0.907 (+/- 0.002)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CatBoost</th>\n",
       "      <td>3.084 (+/- 0.270)</td>\n",
       "      <td>0.055 (+/- 0.019)</td>\n",
       "      <td>0.870 (+/- 0.001)</td>\n",
       "      <td>0.906 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              fit_time         score_time         test_score  \\\n",
       "Dummy                0.003 (+/- 0.000)  0.002 (+/- 0.000)  0.636 (+/- 0.008)   \n",
       "logistic regression  0.412 (+/- 0.063)  0.008 (+/- 0.001)  0.849 (+/- 0.004)   \n",
       "decision tree        0.064 (+/- 0.001)  0.007 (+/- 0.000)  0.811 (+/- 0.002)   \n",
       "random forest        0.571 (+/- 0.024)  0.039 (+/- 0.001)  0.852 (+/- 0.004)   \n",
       "LightGBM             0.115 (+/- 0.025)  0.012 (+/- 0.001)  0.868 (+/- 0.003)   \n",
       "CatBoost             3.084 (+/- 0.270)  0.055 (+/- 0.019)  0.870 (+/- 0.001)   \n",
       "\n",
       "                           train_score  \n",
       "Dummy                0.633 (+/- 0.004)  \n",
       "logistic regression  0.851 (+/- 0.001)  \n",
       "decision tree        1.000 (+/- 0.000)  \n",
       "random forest        1.000 (+/- 0.000)  \n",
       "LightGBM             0.907 (+/- 0.002)  \n",
       "CatBoost             0.906 (+/- 0.001)  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Some observations**\n",
    "- Keep in mind that all these results are with default hyperparameters\n",
    "- Ideally we would carry out hyperparameter optimization for all of them and then compare the results. \n",
    "- We are using a particular scoring metric (accuracy in this case)\n",
    "- We are scaling numeric features but it shouldn't matter for these tree-based models. \n",
    "- Look at the std. Doesn't look very high. \n",
    "    - The scores look more or less stable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Dummy</th>\n",
       "      <td>0.003 (+/- 0.000)</td>\n",
       "      <td>0.002 (+/- 0.000)</td>\n",
       "      <td>0.636 (+/- 0.008)</td>\n",
       "      <td>0.633 (+/- 0.004)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.412 (+/- 0.063)</td>\n",
       "      <td>0.008 (+/- 0.001)</td>\n",
       "      <td>0.849 (+/- 0.004)</td>\n",
       "      <td>0.851 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>0.064 (+/- 0.001)</td>\n",
       "      <td>0.007 (+/- 0.000)</td>\n",
       "      <td>0.811 (+/- 0.002)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>0.571 (+/- 0.024)</td>\n",
       "      <td>0.039 (+/- 0.001)</td>\n",
       "      <td>0.852 (+/- 0.004)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>0.115 (+/- 0.025)</td>\n",
       "      <td>0.012 (+/- 0.001)</td>\n",
       "      <td>0.868 (+/- 0.003)</td>\n",
       "      <td>0.907 (+/- 0.002)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CatBoost</th>\n",
       "      <td>3.084 (+/- 0.270)</td>\n",
       "      <td>0.055 (+/- 0.019)</td>\n",
       "      <td>0.870 (+/- 0.001)</td>\n",
       "      <td>0.906 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              fit_time         score_time         test_score  \\\n",
       "Dummy                0.003 (+/- 0.000)  0.002 (+/- 0.000)  0.636 (+/- 0.008)   \n",
       "logistic regression  0.412 (+/- 0.063)  0.008 (+/- 0.001)  0.849 (+/- 0.004)   \n",
       "decision tree        0.064 (+/- 0.001)  0.007 (+/- 0.000)  0.811 (+/- 0.002)   \n",
       "random forest        0.571 (+/- 0.024)  0.039 (+/- 0.001)  0.852 (+/- 0.004)   \n",
       "LightGBM             0.115 (+/- 0.025)  0.012 (+/- 0.001)  0.868 (+/- 0.003)   \n",
       "CatBoost             3.084 (+/- 0.270)  0.055 (+/- 0.019)  0.870 (+/- 0.001)   \n",
       "\n",
       "                           train_score  \n",
       "Dummy                0.633 (+/- 0.004)  \n",
       "logistic regression  0.851 (+/- 0.001)  \n",
       "decision tree        1.000 (+/- 0.000)  \n",
       "random forest        1.000 (+/- 0.000)  \n",
       "LightGBM             0.907 (+/- 0.002)  \n",
       "CatBoost             0.906 (+/- 0.001)  "
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Decision trees and random forests overfit\n",
    "    - Other models do not seem to overfit much. \n",
    "- Fit times\n",
    "    - Decision trees are fast but not very accurate\n",
    "    - LightGBM is faster than decision trees and more accurate! \n",
    "    - CatBoost fit time is highest followed by random forests.  \n",
    "    - There is not much difference between the validation scores of XGBoost, LightGBM, and CatBoost but it is about 48x slower than LightGBM!\n",
    "    - XGBoost and LightGBM are faster and more accurate than random forest!    \n",
    "- Scores times  \n",
    "    - Prediction times are much smaller in all cases. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What classifier should I use?\n",
    "\n",
    "**Simple answer**\n",
    "- Whichever gets the highest CV score making sure that you're not overusing the validation set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Interpretability**\n",
    "- This is an area of growing interest and concern in ML.\n",
    "- How important is interpretability for you? \n",
    "- In the next class we'll talk about interpretability of non-linear models. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "**Speed/code maintenance**\n",
    "- Other considerations could be speed (fit and/or predict), maintainability of the code.\n",
    "\n",
    "Finally, you could use all of them!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Averaging "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Earlier we looked at a bunch of classifiers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['logistic regression', 'decision tree', 'random forest', 'LightGBM', 'CatBoost'])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "classifiers.keys()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What if we use all these models and let them vote during prediction time? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import VotingClassifier\n",
    "\n",
    "averaging_model = VotingClassifier(\n",
    "    list(classifiers.items()), voting=\"soft\"\n",
    ")  # need the list() here for cross_val to work!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import set_config\n",
    "\n",
    "set_config(display=\"diagram\")  # global setting"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>VotingClassifier(estimators=[(&#x27;logistic regression&#x27;,\n",
       "                              Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                                               ColumnTransformer(transformers=[(&#x27;pipeline-1&#x27;,\n",
       "                                                                                Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                                                                                 StandardScaler())]),\n",
       "                                                                                [&#x27;age&#x27;,\n",
       "                                                                                 &#x27;fnlwgt&#x27;,\n",
       "                                                                                 &#x27;capital.gain&#x27;,\n",
       "                                                                                 &#x27;capital.loss&#x27;,\n",
       "                                                                                 &#x27;hours.per.week&#x27;]),\n",
       "                                                                               (&#x27;pipeline-2&#x27;,\n",
       "                                                                                Pipeline(steps=[(&#x27;ordinalencoder&#x27;,\n",
       "                                                                                                 OrdinalEncoder(categories=[[&#x27;Preschool&#x27;,\n",
       "                                                                                                                             &#x27;1st-4th&#x27;...\n",
       "                                                                                Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                                                                 SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                                                               strategy=&#x27;constant&#x27;)),\n",
       "                                                                                                (&#x27;onehotencoder&#x27;,\n",
       "                                                                                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                                                               sparse=False))]),\n",
       "                                                                                [&#x27;workclass&#x27;,\n",
       "                                                                                 &#x27;marital.status&#x27;,\n",
       "                                                                                 &#x27;occupation&#x27;,\n",
       "                                                                                 &#x27;relationship&#x27;,\n",
       "                                                                                 &#x27;native.country&#x27;]),\n",
       "                                                                               (&#x27;drop&#x27;,\n",
       "                                                                                &#x27;drop&#x27;,\n",
       "                                                                                [&#x27;race&#x27;,\n",
       "                                                                                 &#x27;education.num&#x27;])])),\n",
       "                                              (&#x27;catboostclassifier&#x27;,\n",
       "                                               &lt;catboost.core.CatBoostClassifier object at 0x1218b2320&gt;)]))],\n",
       "                 voting=&#x27;soft&#x27;)</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">VotingClassifier</label><div class=\"sk-toggleable__content\"><pre>VotingClassifier(estimators=[(&#x27;logistic regression&#x27;,\n",
       "                              Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                                               ColumnTransformer(transformers=[(&#x27;pipeline-1&#x27;,\n",
       "                                                                                Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                                                                                 StandardScaler())]),\n",
       "                                                                                [&#x27;age&#x27;,\n",
       "                                                                                 &#x27;fnlwgt&#x27;,\n",
       "                                                                                 &#x27;capital.gain&#x27;,\n",
       "                                                                                 &#x27;capital.loss&#x27;,\n",
       "                                                                                 &#x27;hours.per.week&#x27;]),\n",
       "                                                                               (&#x27;pipeline-2&#x27;,\n",
       "                                                                                Pipeline(steps=[(&#x27;ordinalencoder&#x27;,\n",
       "                                                                                                 OrdinalEncoder(categories=[[&#x27;Preschool&#x27;,\n",
       "                                                                                                                             &#x27;1st-4th&#x27;...\n",
       "                                                                                Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                                                                 SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                                                               strategy=&#x27;constant&#x27;)),\n",
       "                                                                                                (&#x27;onehotencoder&#x27;,\n",
       "                                                                                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                                                               sparse=False))]),\n",
       "                                                                                [&#x27;workclass&#x27;,\n",
       "                                                                                 &#x27;marital.status&#x27;,\n",
       "                                                                                 &#x27;occupation&#x27;,\n",
       "                                                                                 &#x27;relationship&#x27;,\n",
       "                                                                                 &#x27;native.country&#x27;]),\n",
       "                                                                               (&#x27;drop&#x27;,\n",
       "                                                                                &#x27;drop&#x27;,\n",
       "                                                                                [&#x27;race&#x27;,\n",
       "                                                                                 &#x27;education.num&#x27;])])),\n",
       "                                              (&#x27;catboostclassifier&#x27;,\n",
       "                                               &lt;catboost.core.CatBoostClassifier object at 0x1218b2320&gt;)]))],\n",
       "                 voting=&#x27;soft&#x27;)</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>logistic regression</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline-1&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;age&#x27;, &#x27;fnlwgt&#x27;, &#x27;capital.gain&#x27;,\n",
       "                                  &#x27;capital.loss&#x27;, &#x27;hours.per.week&#x27;]),\n",
       "                                (&#x27;pipeline-2&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;ordinalencoder&#x27;,\n",
       "                                                  OrdinalEncoder(categories=[[&#x27;Preschool&#x27;,\n",
       "                                                                              &#x27;1st-4th&#x27;,\n",
       "                                                                              &#x27;5th-6th&#x27;,\n",
       "                                                                              &#x27;7th-8th&#x27;,\n",
       "                                                                              &#x27;9th&#x27;,\n",
       "                                                                              &#x27;10th&#x27;,\n",
       "                                                                              &#x27;11th&#x27;,\n",
       "                                                                              &#x27;12th&#x27;,\n",
       "                                                                              &#x27;HS-grad&#x27;,\n",
       "                                                                              &#x27;Prof-school&#x27;,\n",
       "                                                                              &#x27;Assoc-voc&#x27;,\n",
       "                                                                              &#x27;Ass...\n",
       "                                                  OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                                                dtype=&lt;class &#x27;int&#x27;&gt;))]),\n",
       "                                 [&#x27;sex&#x27;]),\n",
       "                                (&#x27;pipeline-4&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                strategy=&#x27;constant&#x27;)),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;workclass&#x27;, &#x27;marital.status&#x27;, &#x27;occupation&#x27;,\n",
       "                                  &#x27;relationship&#x27;, &#x27;native.country&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;race&#x27;, &#x27;education.num&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-1</label><div class=\"sk-toggleable__content\"><pre>[&#x27;age&#x27;, &#x27;fnlwgt&#x27;, &#x27;capital.gain&#x27;, &#x27;capital.loss&#x27;, &#x27;hours.per.week&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-2</label><div class=\"sk-toggleable__content\"><pre>[&#x27;education&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Preschool&#x27;, &#x27;1st-4th&#x27;, &#x27;5th-6th&#x27;, &#x27;7th-8th&#x27;, &#x27;9th&#x27;,\n",
       "                            &#x27;10th&#x27;, &#x27;11th&#x27;, &#x27;12th&#x27;, &#x27;HS-grad&#x27;, &#x27;Prof-school&#x27;,\n",
       "                            &#x27;Assoc-voc&#x27;, &#x27;Assoc-acdm&#x27;, &#x27;Some-college&#x27;,\n",
       "                            &#x27;Bachelors&#x27;, &#x27;Masters&#x27;, &#x27;Doctorate&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-3</label><div class=\"sk-toggleable__content\"><pre>[&#x27;sex&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(drop=&#x27;if_binary&#x27;, dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-4</label><div class=\"sk-toggleable__content\"><pre>[&#x27;workclass&#x27;, &#x27;marital.status&#x27;, &#x27;occupation&#x27;, &#x27;relationship&#x27;, &#x27;native.country&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-12\" type=\"checkbox\" ><label for=\"sk-estimator-id-12\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-13\" type=\"checkbox\" ><label for=\"sk-estimator-id-13\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;race&#x27;, &#x27;education.num&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=2000, random_state=123)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>decision tree</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline-1&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;age&#x27;, &#x27;fnlwgt&#x27;, &#x27;capital.gain&#x27;,\n",
       "                                  &#x27;capital.loss&#x27;, &#x27;hours.per.week&#x27;]),\n",
       "                                (&#x27;pipeline-2&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;ordinalencoder&#x27;,\n",
       "                                                  OrdinalEncoder(categories=[[&#x27;Preschool&#x27;,\n",
       "                                                                              &#x27;1st-4th&#x27;,\n",
       "                                                                              &#x27;5th-6th&#x27;,\n",
       "                                                                              &#x27;7th-8th&#x27;,\n",
       "                                                                              &#x27;9th&#x27;,\n",
       "                                                                              &#x27;10th&#x27;,\n",
       "                                                                              &#x27;11th&#x27;,\n",
       "                                                                              &#x27;12th&#x27;,\n",
       "                                                                              &#x27;HS-grad&#x27;,\n",
       "                                                                              &#x27;Prof-school&#x27;,\n",
       "                                                                              &#x27;Assoc-voc&#x27;,\n",
       "                                                                              &#x27;Ass...\n",
       "                                                  OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                                                dtype=&lt;class &#x27;int&#x27;&gt;))]),\n",
       "                                 [&#x27;sex&#x27;]),\n",
       "                                (&#x27;pipeline-4&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                strategy=&#x27;constant&#x27;)),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;workclass&#x27;, &#x27;marital.status&#x27;, &#x27;occupation&#x27;,\n",
       "                                  &#x27;relationship&#x27;, &#x27;native.country&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;race&#x27;, &#x27;education.num&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-17\" type=\"checkbox\" ><label for=\"sk-estimator-id-17\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-1</label><div class=\"sk-toggleable__content\"><pre>[&#x27;age&#x27;, &#x27;fnlwgt&#x27;, &#x27;capital.gain&#x27;, &#x27;capital.loss&#x27;, &#x27;hours.per.week&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" ><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-2</label><div class=\"sk-toggleable__content\"><pre>[&#x27;education&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-20\" type=\"checkbox\" ><label for=\"sk-estimator-id-20\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Preschool&#x27;, &#x27;1st-4th&#x27;, &#x27;5th-6th&#x27;, &#x27;7th-8th&#x27;, &#x27;9th&#x27;,\n",
       "                            &#x27;10th&#x27;, &#x27;11th&#x27;, &#x27;12th&#x27;, &#x27;HS-grad&#x27;, &#x27;Prof-school&#x27;,\n",
       "                            &#x27;Assoc-voc&#x27;, &#x27;Assoc-acdm&#x27;, &#x27;Some-college&#x27;,\n",
       "                            &#x27;Bachelors&#x27;, &#x27;Masters&#x27;, &#x27;Doctorate&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-21\" type=\"checkbox\" ><label for=\"sk-estimator-id-21\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-3</label><div class=\"sk-toggleable__content\"><pre>[&#x27;sex&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-22\" type=\"checkbox\" ><label for=\"sk-estimator-id-22\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-23\" type=\"checkbox\" ><label for=\"sk-estimator-id-23\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(drop=&#x27;if_binary&#x27;, dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-24\" type=\"checkbox\" ><label for=\"sk-estimator-id-24\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-4</label><div class=\"sk-toggleable__content\"><pre>[&#x27;workclass&#x27;, &#x27;marital.status&#x27;, &#x27;occupation&#x27;, &#x27;relationship&#x27;, &#x27;native.country&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-25\" type=\"checkbox\" ><label for=\"sk-estimator-id-25\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-26\" type=\"checkbox\" ><label for=\"sk-estimator-id-26\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-27\" type=\"checkbox\" ><label for=\"sk-estimator-id-27\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;race&#x27;, &#x27;education.num&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-28\" type=\"checkbox\" ><label for=\"sk-estimator-id-28\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-29\" type=\"checkbox\" ><label for=\"sk-estimator-id-29\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">DecisionTreeClassifier</label><div class=\"sk-toggleable__content\"><pre>DecisionTreeClassifier(random_state=123)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>random forest</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-30\" type=\"checkbox\" ><label for=\"sk-estimator-id-30\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline-1&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;age&#x27;, &#x27;fnlwgt&#x27;, &#x27;capital.gain&#x27;,\n",
       "                                  &#x27;capital.loss&#x27;, &#x27;hours.per.week&#x27;]),\n",
       "                                (&#x27;pipeline-2&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;ordinalencoder&#x27;,\n",
       "                                                  OrdinalEncoder(categories=[[&#x27;Preschool&#x27;,\n",
       "                                                                              &#x27;1st-4th&#x27;,\n",
       "                                                                              &#x27;5th-6th&#x27;,\n",
       "                                                                              &#x27;7th-8th&#x27;,\n",
       "                                                                              &#x27;9th&#x27;,\n",
       "                                                                              &#x27;10th&#x27;,\n",
       "                                                                              &#x27;11th&#x27;,\n",
       "                                                                              &#x27;12th&#x27;,\n",
       "                                                                              &#x27;HS-grad&#x27;,\n",
       "                                                                              &#x27;Prof-school&#x27;,\n",
       "                                                                              &#x27;Assoc-voc&#x27;,\n",
       "                                                                              &#x27;Ass...\n",
       "                                                  OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                                                dtype=&lt;class &#x27;int&#x27;&gt;))]),\n",
       "                                 [&#x27;sex&#x27;]),\n",
       "                                (&#x27;pipeline-4&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                strategy=&#x27;constant&#x27;)),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;workclass&#x27;, &#x27;marital.status&#x27;, &#x27;occupation&#x27;,\n",
       "                                  &#x27;relationship&#x27;, &#x27;native.country&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;race&#x27;, &#x27;education.num&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-31\" type=\"checkbox\" ><label for=\"sk-estimator-id-31\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-1</label><div class=\"sk-toggleable__content\"><pre>[&#x27;age&#x27;, &#x27;fnlwgt&#x27;, &#x27;capital.gain&#x27;, &#x27;capital.loss&#x27;, &#x27;hours.per.week&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-32\" type=\"checkbox\" ><label for=\"sk-estimator-id-32\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-33\" type=\"checkbox\" ><label for=\"sk-estimator-id-33\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-2</label><div class=\"sk-toggleable__content\"><pre>[&#x27;education&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-34\" type=\"checkbox\" ><label for=\"sk-estimator-id-34\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Preschool&#x27;, &#x27;1st-4th&#x27;, &#x27;5th-6th&#x27;, &#x27;7th-8th&#x27;, &#x27;9th&#x27;,\n",
       "                            &#x27;10th&#x27;, &#x27;11th&#x27;, &#x27;12th&#x27;, &#x27;HS-grad&#x27;, &#x27;Prof-school&#x27;,\n",
       "                            &#x27;Assoc-voc&#x27;, &#x27;Assoc-acdm&#x27;, &#x27;Some-college&#x27;,\n",
       "                            &#x27;Bachelors&#x27;, &#x27;Masters&#x27;, &#x27;Doctorate&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-35\" type=\"checkbox\" ><label for=\"sk-estimator-id-35\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-3</label><div class=\"sk-toggleable__content\"><pre>[&#x27;sex&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-36\" type=\"checkbox\" ><label for=\"sk-estimator-id-36\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-37\" type=\"checkbox\" ><label for=\"sk-estimator-id-37\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(drop=&#x27;if_binary&#x27;, dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-38\" type=\"checkbox\" ><label for=\"sk-estimator-id-38\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-4</label><div class=\"sk-toggleable__content\"><pre>[&#x27;workclass&#x27;, &#x27;marital.status&#x27;, &#x27;occupation&#x27;, &#x27;relationship&#x27;, &#x27;native.country&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-39\" type=\"checkbox\" ><label for=\"sk-estimator-id-39\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-40\" type=\"checkbox\" ><label for=\"sk-estimator-id-40\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-41\" type=\"checkbox\" ><label for=\"sk-estimator-id-41\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;race&#x27;, &#x27;education.num&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-42\" type=\"checkbox\" ><label for=\"sk-estimator-id-42\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-43\" type=\"checkbox\" ><label for=\"sk-estimator-id-43\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">RandomForestClassifier</label><div class=\"sk-toggleable__content\"><pre>RandomForestClassifier(random_state=123)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>LightGBM</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-44\" type=\"checkbox\" ><label for=\"sk-estimator-id-44\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline-1&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;age&#x27;, &#x27;fnlwgt&#x27;, &#x27;capital.gain&#x27;,\n",
       "                                  &#x27;capital.loss&#x27;, &#x27;hours.per.week&#x27;]),\n",
       "                                (&#x27;pipeline-2&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;ordinalencoder&#x27;,\n",
       "                                                  OrdinalEncoder(categories=[[&#x27;Preschool&#x27;,\n",
       "                                                                              &#x27;1st-4th&#x27;,\n",
       "                                                                              &#x27;5th-6th&#x27;,\n",
       "                                                                              &#x27;7th-8th&#x27;,\n",
       "                                                                              &#x27;9th&#x27;,\n",
       "                                                                              &#x27;10th&#x27;,\n",
       "                                                                              &#x27;11th&#x27;,\n",
       "                                                                              &#x27;12th&#x27;,\n",
       "                                                                              &#x27;HS-grad&#x27;,\n",
       "                                                                              &#x27;Prof-school&#x27;,\n",
       "                                                                              &#x27;Assoc-voc&#x27;,\n",
       "                                                                              &#x27;Ass...\n",
       "                                                  OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                                                dtype=&lt;class &#x27;int&#x27;&gt;))]),\n",
       "                                 [&#x27;sex&#x27;]),\n",
       "                                (&#x27;pipeline-4&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                strategy=&#x27;constant&#x27;)),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;workclass&#x27;, &#x27;marital.status&#x27;, &#x27;occupation&#x27;,\n",
       "                                  &#x27;relationship&#x27;, &#x27;native.country&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;race&#x27;, &#x27;education.num&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-45\" type=\"checkbox\" ><label for=\"sk-estimator-id-45\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-1</label><div class=\"sk-toggleable__content\"><pre>[&#x27;age&#x27;, &#x27;fnlwgt&#x27;, &#x27;capital.gain&#x27;, &#x27;capital.loss&#x27;, &#x27;hours.per.week&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-46\" type=\"checkbox\" ><label for=\"sk-estimator-id-46\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-47\" type=\"checkbox\" ><label for=\"sk-estimator-id-47\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-2</label><div class=\"sk-toggleable__content\"><pre>[&#x27;education&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-48\" type=\"checkbox\" ><label for=\"sk-estimator-id-48\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Preschool&#x27;, &#x27;1st-4th&#x27;, &#x27;5th-6th&#x27;, &#x27;7th-8th&#x27;, &#x27;9th&#x27;,\n",
       "                            &#x27;10th&#x27;, &#x27;11th&#x27;, &#x27;12th&#x27;, &#x27;HS-grad&#x27;, &#x27;Prof-school&#x27;,\n",
       "                            &#x27;Assoc-voc&#x27;, &#x27;Assoc-acdm&#x27;, &#x27;Some-college&#x27;,\n",
       "                            &#x27;Bachelors&#x27;, &#x27;Masters&#x27;, &#x27;Doctorate&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-49\" type=\"checkbox\" ><label for=\"sk-estimator-id-49\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-3</label><div class=\"sk-toggleable__content\"><pre>[&#x27;sex&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-50\" type=\"checkbox\" ><label for=\"sk-estimator-id-50\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-51\" type=\"checkbox\" ><label for=\"sk-estimator-id-51\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(drop=&#x27;if_binary&#x27;, dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-52\" type=\"checkbox\" ><label for=\"sk-estimator-id-52\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-4</label><div class=\"sk-toggleable__content\"><pre>[&#x27;workclass&#x27;, &#x27;marital.status&#x27;, &#x27;occupation&#x27;, &#x27;relationship&#x27;, &#x27;native.country&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-53\" type=\"checkbox\" ><label for=\"sk-estimator-id-53\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-54\" type=\"checkbox\" ><label for=\"sk-estimator-id-54\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-55\" type=\"checkbox\" ><label for=\"sk-estimator-id-55\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;race&#x27;, &#x27;education.num&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-56\" type=\"checkbox\" ><label for=\"sk-estimator-id-56\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-57\" type=\"checkbox\" ><label for=\"sk-estimator-id-57\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LGBMClassifier</label><div class=\"sk-toggleable__content\"><pre>LGBMClassifier(random_state=123)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><label>CatBoost</label></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-58\" type=\"checkbox\" ><label for=\"sk-estimator-id-58\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline-1&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;age&#x27;, &#x27;fnlwgt&#x27;, &#x27;capital.gain&#x27;,\n",
       "                                  &#x27;capital.loss&#x27;, &#x27;hours.per.week&#x27;]),\n",
       "                                (&#x27;pipeline-2&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;ordinalencoder&#x27;,\n",
       "                                                  OrdinalEncoder(categories=[[&#x27;Preschool&#x27;,\n",
       "                                                                              &#x27;1st-4th&#x27;,\n",
       "                                                                              &#x27;5th-6th&#x27;,\n",
       "                                                                              &#x27;7th-8th&#x27;,\n",
       "                                                                              &#x27;9th&#x27;,\n",
       "                                                                              &#x27;10th&#x27;,\n",
       "                                                                              &#x27;11th&#x27;,\n",
       "                                                                              &#x27;12th&#x27;,\n",
       "                                                                              &#x27;HS-grad&#x27;,\n",
       "                                                                              &#x27;Prof-school&#x27;,\n",
       "                                                                              &#x27;Assoc-voc&#x27;,\n",
       "                                                                              &#x27;Ass...\n",
       "                                                  OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                                                dtype=&lt;class &#x27;int&#x27;&gt;))]),\n",
       "                                 [&#x27;sex&#x27;]),\n",
       "                                (&#x27;pipeline-4&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(fill_value=&#x27;missing&#x27;,\n",
       "                                                                strategy=&#x27;constant&#x27;)),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;,\n",
       "                                                                sparse=False))]),\n",
       "                                 [&#x27;workclass&#x27;, &#x27;marital.status&#x27;, &#x27;occupation&#x27;,\n",
       "                                  &#x27;relationship&#x27;, &#x27;native.country&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;race&#x27;, &#x27;education.num&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-59\" type=\"checkbox\" ><label for=\"sk-estimator-id-59\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-1</label><div class=\"sk-toggleable__content\"><pre>[&#x27;age&#x27;, &#x27;fnlwgt&#x27;, &#x27;capital.gain&#x27;, &#x27;capital.loss&#x27;, &#x27;hours.per.week&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-60\" type=\"checkbox\" ><label for=\"sk-estimator-id-60\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-61\" type=\"checkbox\" ><label for=\"sk-estimator-id-61\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-2</label><div class=\"sk-toggleable__content\"><pre>[&#x27;education&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-62\" type=\"checkbox\" ><label for=\"sk-estimator-id-62\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Preschool&#x27;, &#x27;1st-4th&#x27;, &#x27;5th-6th&#x27;, &#x27;7th-8th&#x27;, &#x27;9th&#x27;,\n",
       "                            &#x27;10th&#x27;, &#x27;11th&#x27;, &#x27;12th&#x27;, &#x27;HS-grad&#x27;, &#x27;Prof-school&#x27;,\n",
       "                            &#x27;Assoc-voc&#x27;, &#x27;Assoc-acdm&#x27;, &#x27;Some-college&#x27;,\n",
       "                            &#x27;Bachelors&#x27;, &#x27;Masters&#x27;, &#x27;Doctorate&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-63\" type=\"checkbox\" ><label for=\"sk-estimator-id-63\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-3</label><div class=\"sk-toggleable__content\"><pre>[&#x27;sex&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-64\" type=\"checkbox\" ><label for=\"sk-estimator-id-64\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-65\" type=\"checkbox\" ><label for=\"sk-estimator-id-65\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(drop=&#x27;if_binary&#x27;, dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-66\" type=\"checkbox\" ><label for=\"sk-estimator-id-66\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline-4</label><div class=\"sk-toggleable__content\"><pre>[&#x27;workclass&#x27;, &#x27;marital.status&#x27;, &#x27;occupation&#x27;, &#x27;relationship&#x27;, &#x27;native.country&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-67\" type=\"checkbox\" ><label for=\"sk-estimator-id-67\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(fill_value=&#x27;missing&#x27;, strategy=&#x27;constant&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-68\" type=\"checkbox\" ><label for=\"sk-estimator-id-68\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;, sparse=False)</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-69\" type=\"checkbox\" ><label for=\"sk-estimator-id-69\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;race&#x27;, &#x27;education.num&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-70\" type=\"checkbox\" ><label for=\"sk-estimator-id-70\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-71\" type=\"checkbox\" ><label for=\"sk-estimator-id-71\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CatBoostClassifier</label><div class=\"sk-toggleable__content\"><pre>&lt;catboost.core.CatBoostClassifier object at 0x1218b2320&gt;</pre></div></div></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "VotingClassifier(estimators=[('logistic regression',\n",
       "                              Pipeline(steps=[('columntransformer',\n",
       "                                               ColumnTransformer(transformers=[('pipeline-1',\n",
       "                                                                                Pipeline(steps=[('standardscaler',\n",
       "                                                                                                 StandardScaler())]),\n",
       "                                                                                ['age',\n",
       "                                                                                 'fnlwgt',\n",
       "                                                                                 'capital.gain',\n",
       "                                                                                 'capital.loss',\n",
       "                                                                                 'hours.per.week']),\n",
       "                                                                               ('pipeline-2',\n",
       "                                                                                Pipeline(steps=[('ordinalencoder',\n",
       "                                                                                                 OrdinalEncoder(categories=[['Preschool',\n",
       "                                                                                                                             '1st-4th'...\n",
       "                                                                                Pipeline(steps=[('simpleimputer',\n",
       "                                                                                                 SimpleImputer(fill_value='missing',\n",
       "                                                                                                               strategy='constant')),\n",
       "                                                                                                ('onehotencoder',\n",
       "                                                                                                 OneHotEncoder(handle_unknown='ignore',\n",
       "                                                                                                               sparse=False))]),\n",
       "                                                                                ['workclass',\n",
       "                                                                                 'marital.status',\n",
       "                                                                                 'occupation',\n",
       "                                                                                 'relationship',\n",
       "                                                                                 'native.country']),\n",
       "                                                                               ('drop',\n",
       "                                                                                'drop',\n",
       "                                                                                ['race',\n",
       "                                                                                 'education.num'])])),\n",
       "                                              ('catboostclassifier',\n",
       "                                               <catboost.core.CatBoostClassifier object at 0x1218b2320>)]))],\n",
       "                 voting='soft')"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "averaging_model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This `VotingClassifier` will take a _vote_ using the predictions of the constituent classifier pipelines."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Main parameter: `voting`\n",
    "- `voting='hard'` \n",
    "    - it uses the output of `predict` and actually votes.\n",
    "- `voting='soft'`\n",
    "    - with `voting='soft'` it averages the output of `predict_proba` and then thresholds / takes the larger."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- The choice depends on whether you trust `predict_proba` from your base classifiers - if so, it's nice to access that information."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "averaging_model.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What happens when you `fit` a `VotingClassifier`?\n",
    "    - It will fit all constituent models.    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "It seems sklearn requires us to actually call `fit` on the `VotingClassifier`, instead of passing in pre-fit models. This is an implementation choice rather than a conceptual limitation.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's look at particular test examples where `income` is \">50k\" (y=1):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_g50k = (\n",
    "    test_df.query(\"income == '>50K'\").sample(4, random_state=2).drop(columns=[\"income\"])\n",
    ")\n",
    "test_l50k = (\n",
    "    test_df.query(\"income == '<=50K'\")\n",
    "    .sample(4, random_state=2)\n",
    "    .drop(columns=[\"income\"])\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['<=50K', '>50K'], dtype=object)"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "averaging_model.classes_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Voting classifier</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&gt;50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;=50K</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Voting classifier\n",
       "0             <=50K\n",
       "1              >50K\n",
       "2             <=50K\n",
       "3             <=50K"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = {\"Voting classifier\": averaging_model.predict(test_l50k)}\n",
    "pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For hard voting, these are the votes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Voting classifier</th>\n",
       "      <th>logistic regression</th>\n",
       "      <th>decision tree</th>\n",
       "      <th>random forest</th>\n",
       "      <th>LightGBM</th>\n",
       "      <th>CatBoost</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>&gt;50K</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>&lt;=50K</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Voting classifier  logistic regression  decision tree  random forest  \\\n",
       "0             <=50K                    0              0              0   \n",
       "1              >50K                    1              1              1   \n",
       "2             <=50K                    0              0              0   \n",
       "3             <=50K                    0              0              0   \n",
       "\n",
       "   LightGBM  CatBoost  \n",
       "0         0         0  \n",
       "1         1         0  \n",
       "2         0         0  \n",
       "3         0         0  "
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1 = {\n",
    "    name: classifier.predict(test_l50k)\n",
    "    for name, classifier in averaging_model.named_estimators_.items()\n",
    "}\n",
    "data.update(r1)\n",
    "pd.DataFrame(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "For soft voting, these are the scores:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'logistic regression': array([[0.99808377, 0.00191623],\n",
       "        [0.46828782, 0.53171218],\n",
       "        [0.9752668 , 0.0247332 ],\n",
       "        [0.95695262, 0.04304738]]),\n",
       " 'decision tree': array([[1., 0.],\n",
       "        [0., 1.],\n",
       "        [1., 0.],\n",
       "        [1., 0.]]),\n",
       " 'random forest': array([[1.  , 0.  ],\n",
       "        [0.33, 0.67],\n",
       "        [0.99, 0.01],\n",
       "        [0.81, 0.19]]),\n",
       " 'LightGBM': array([[9.99463389e-01, 5.36610801e-04],\n",
       "        [4.69000802e-01, 5.30999198e-01],\n",
       "        [9.51290934e-01, 4.87090656e-02],\n",
       "        [9.07706919e-01, 9.22930811e-02]]),\n",
       " 'CatBoost': array([[0.99888791, 0.00111209],\n",
       "        [0.55857933, 0.44142067],\n",
       "        [0.96274841, 0.03725159],\n",
       "        [0.90925194, 0.09074806]])}"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r1 = {\n",
    "    name: classifier.predict_proba(test_l50k)\n",
    "    for name, classifier in averaging_model.named_estimators_.items()\n",
    "}\n",
    "r1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(Aside: the probability scores from `DecisionTreeClassifier` are pretty bad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's see how well this model performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "results[\"Voting\"] = mean_std_cross_val_scores(averaging_model, X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Dummy</th>\n",
       "      <td>0.003 (+/- 0.000)</td>\n",
       "      <td>0.002 (+/- 0.000)</td>\n",
       "      <td>0.636 (+/- 0.008)</td>\n",
       "      <td>0.633 (+/- 0.004)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.412 (+/- 0.063)</td>\n",
       "      <td>0.008 (+/- 0.001)</td>\n",
       "      <td>0.849 (+/- 0.004)</td>\n",
       "      <td>0.851 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>0.064 (+/- 0.001)</td>\n",
       "      <td>0.007 (+/- 0.000)</td>\n",
       "      <td>0.811 (+/- 0.002)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>0.571 (+/- 0.024)</td>\n",
       "      <td>0.039 (+/- 0.001)</td>\n",
       "      <td>0.852 (+/- 0.004)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>0.115 (+/- 0.025)</td>\n",
       "      <td>0.012 (+/- 0.001)</td>\n",
       "      <td>0.868 (+/- 0.003)</td>\n",
       "      <td>0.907 (+/- 0.002)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CatBoost</th>\n",
       "      <td>3.084 (+/- 0.270)</td>\n",
       "      <td>0.055 (+/- 0.019)</td>\n",
       "      <td>0.870 (+/- 0.001)</td>\n",
       "      <td>0.906 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Voting</th>\n",
       "      <td>4.414 (+/- 0.489)</td>\n",
       "      <td>0.113 (+/- 0.009)</td>\n",
       "      <td>0.858 (+/- 0.004)</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              fit_time         score_time         test_score  \\\n",
       "Dummy                0.003 (+/- 0.000)  0.002 (+/- 0.000)  0.636 (+/- 0.008)   \n",
       "logistic regression  0.412 (+/- 0.063)  0.008 (+/- 0.001)  0.849 (+/- 0.004)   \n",
       "decision tree        0.064 (+/- 0.001)  0.007 (+/- 0.000)  0.811 (+/- 0.002)   \n",
       "random forest        0.571 (+/- 0.024)  0.039 (+/- 0.001)  0.852 (+/- 0.004)   \n",
       "LightGBM             0.115 (+/- 0.025)  0.012 (+/- 0.001)  0.868 (+/- 0.003)   \n",
       "CatBoost             3.084 (+/- 0.270)  0.055 (+/- 0.019)  0.870 (+/- 0.001)   \n",
       "Voting               4.414 (+/- 0.489)  0.113 (+/- 0.009)  0.858 (+/- 0.004)   \n",
       "\n",
       "                           train_score  \n",
       "Dummy                0.633 (+/- 0.004)  \n",
       "logistic regression  0.851 (+/- 0.001)  \n",
       "decision tree        1.000 (+/- 0.000)  \n",
       "random forest        1.000 (+/- 0.000)  \n",
       "LightGBM             0.907 (+/- 0.002)  \n",
       "CatBoost             0.906 (+/- 0.001)  \n",
       "Voting                             NaN  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It appears that here we didn't do much better than our best classifier :(. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try removing decision tree classifier. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifiers_ndt = classifiers.copy()\n",
    "del classifiers_ndt[\"decision tree\"]\n",
    "averaging_model_ndt = VotingClassifier(\n",
    "    list(classifiers_ndt.items()), voting=\"soft\"\n",
    ")  # need the list() here for cross_val to work!\n",
    "\n",
    "results[\"Voting_ndt\"] = mean_std_cross_val_scores(\n",
    "    averaging_model_ndt,\n",
    "    X_train,\n",
    "    y_train,\n",
    "    return_train_score=True,\n",
    "    scoring=scoring_metric,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Dummy</th>\n",
       "      <td>0.003 (+/- 0.000)</td>\n",
       "      <td>0.002 (+/- 0.000)</td>\n",
       "      <td>0.636 (+/- 0.008)</td>\n",
       "      <td>0.633 (+/- 0.004)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.412 (+/- 0.063)</td>\n",
       "      <td>0.008 (+/- 0.001)</td>\n",
       "      <td>0.849 (+/- 0.004)</td>\n",
       "      <td>0.851 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>0.064 (+/- 0.001)</td>\n",
       "      <td>0.007 (+/- 0.000)</td>\n",
       "      <td>0.811 (+/- 0.002)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>0.571 (+/- 0.024)</td>\n",
       "      <td>0.039 (+/- 0.001)</td>\n",
       "      <td>0.852 (+/- 0.004)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>0.115 (+/- 0.025)</td>\n",
       "      <td>0.012 (+/- 0.001)</td>\n",
       "      <td>0.868 (+/- 0.003)</td>\n",
       "      <td>0.907 (+/- 0.002)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CatBoost</th>\n",
       "      <td>3.084 (+/- 0.270)</td>\n",
       "      <td>0.055 (+/- 0.019)</td>\n",
       "      <td>0.870 (+/- 0.001)</td>\n",
       "      <td>0.906 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Voting</th>\n",
       "      <td>4.414 (+/- 0.489)</td>\n",
       "      <td>0.113 (+/- 0.009)</td>\n",
       "      <td>0.858 (+/- 0.004)</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Voting_ndt</th>\n",
       "      <td>4.040 (+/- 0.144)</td>\n",
       "      <td>0.099 (+/- 0.004)</td>\n",
       "      <td>0.868 (+/- 0.005)</td>\n",
       "      <td>0.929 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              fit_time         score_time         test_score  \\\n",
       "Dummy                0.003 (+/- 0.000)  0.002 (+/- 0.000)  0.636 (+/- 0.008)   \n",
       "logistic regression  0.412 (+/- 0.063)  0.008 (+/- 0.001)  0.849 (+/- 0.004)   \n",
       "decision tree        0.064 (+/- 0.001)  0.007 (+/- 0.000)  0.811 (+/- 0.002)   \n",
       "random forest        0.571 (+/- 0.024)  0.039 (+/- 0.001)  0.852 (+/- 0.004)   \n",
       "LightGBM             0.115 (+/- 0.025)  0.012 (+/- 0.001)  0.868 (+/- 0.003)   \n",
       "CatBoost             3.084 (+/- 0.270)  0.055 (+/- 0.019)  0.870 (+/- 0.001)   \n",
       "Voting               4.414 (+/- 0.489)  0.113 (+/- 0.009)  0.858 (+/- 0.004)   \n",
       "Voting_ndt           4.040 (+/- 0.144)  0.099 (+/- 0.004)  0.868 (+/- 0.005)   \n",
       "\n",
       "                           train_score  \n",
       "Dummy                0.633 (+/- 0.004)  \n",
       "logistic regression  0.851 (+/- 0.001)  \n",
       "decision tree        1.000 (+/- 0.000)  \n",
       "random forest        1.000 (+/- 0.000)  \n",
       "LightGBM             0.907 (+/- 0.002)  \n",
       "CatBoost             0.906 (+/- 0.001)  \n",
       "Voting                             NaN  \n",
       "Voting_ndt           0.929 (+/- 0.001)  "
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Still the results are not better than the best performing model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- It didn't happen here but how could the average do better than the best model???\n",
    "  - From the perspective of the best estimator (in this case CatBoost), why are you adding on worse estimators??\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here's how this can work:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| Example | log reg    | rand forest    | cat boost    | Averaged model |\n",
    "|--------|--------|--------|---------|---------------|\n",
    "|  1     | ✅    |   ✅    | ❌     | ✅✅❌=>✅  |\n",
    "|  2     | ✅    |   ❌    | ✅     | ✅❌✅=>✅  |\n",
    "|  3     | ❌    |   ✅    | ✅     | ❌✅✅=>✅  |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In short, as long as the different models make different mistakes, this can work."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Why not always do this?\n",
    "\n",
    "1. `fit`/`predict` time.\n",
    "2. Reduction in interpretability.\n",
    "3. Reduction in code maintainability (e.g. Netflix prize)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What kind of estimators can we combine? \n",
    "\n",
    "- You can combine \n",
    "    - completely different estimators, or similar estimators.\n",
    "    - estimators trained on different samples.\n",
    "    - estimators with different hyperparameter values."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Stacking "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "- Another type of ensemble is stacking.\n",
    "- Instead of averaging the outputs of each estimator, instead use their outputs as _inputs to another model_.\n",
    "- By default for classification, it uses logistic regression.\n",
    "  - We don't need a complex model here necessarily, more of a weighted average.\n",
    "  - The features going into the logistic regression are the classifier outputs, _not_ the original features!\n",
    "  - So the number of coefficients = the number of base estimators!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import StackingClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The code starts to get too slow here; so we'll remove CatBoost. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "classifiers_nocat = classifiers.copy()\n",
    "del classifiers_nocat[\"CatBoost\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [],
   "source": [
    "stacking_model = StackingClassifier(list(classifiers_nocat.items()))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "stacking_model.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What's going on in here? \n",
    "\n",
    "- It is doing cross-validation by itself by default (see [documentation](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.StackingClassifier.html))\n",
    "  - It is fitting the base estimators on the training fold\n",
    "  - And the predicting on the validation fold\n",
    "  - And then fitting the meta-estimator on that output (on the validation fold)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    " > Note that estimators_ are fitted on the full X while final_estimator_ is trained using cross-validated predictions of the base estimators using cross_val_predict."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Here is the input features (X) to the meta-model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "valid_sample = train_df.sample(4, random_state=2).drop(columns=[\"income\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'logistic regression': array([[0.75536799, 0.24463201],\n",
       "        [0.99487482, 0.00512518],\n",
       "        [0.82482052, 0.17517948],\n",
       "        [0.958843  , 0.041157  ]]),\n",
       " 'decision tree': array([[1., 0.],\n",
       "        [1., 0.],\n",
       "        [1., 0.],\n",
       "        [1., 0.]]),\n",
       " 'random forest': array([[0.89, 0.11],\n",
       "        [1.  , 0.  ],\n",
       "        [0.91, 0.09],\n",
       "        [0.97, 0.03]]),\n",
       " 'LightGBM': array([[6.39667886e-01, 3.60332114e-01],\n",
       "        [9.99382317e-01, 6.17683085e-04],\n",
       "        [8.15871800e-01, 1.84128200e-01],\n",
       "        [9.83034210e-01, 1.69657901e-02]])}"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r3 = {\n",
    "    name: pipe.predict_proba(valid_sample)\n",
    "    for (name, pipe) in stacking_model.named_estimators_.items()\n",
    "}\n",
    "r3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Our meta-model is logistic regression (which it is by default).\n",
    "- Let's look at the learned coefficients. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Coefficient</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>1.668165</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>-0.069145</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>0.359349</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>4.577977</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                     Coefficient\n",
       "logistic regression     1.668165\n",
       "decision tree          -0.069145\n",
       "random forest           0.359349\n",
       "LightGBM                4.577977"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(\n",
    "    data=stacking_model.final_estimator_.coef_[0],\n",
    "    index=classifiers_nocat.keys(),\n",
    "    columns=[\"Coefficient\"],\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([-3.26552654])"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacking_model.final_estimator_.intercept_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- It seems that the LightGBM is being trusted the most. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['<=50K', '>50K', '<=50K', '<=50K'], dtype=object)"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacking_model.predict(test_g50k)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[0.8092899 , 0.1907101 ],\n",
       "       [0.37962077, 0.62037923],\n",
       "       [0.77018894, 0.22981106],\n",
       "       [0.60534688, 0.39465312]])"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stacking_model.predict_proba(test_g50k)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "(This is the `predict_proba` from logistic regression)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's see how well this model performs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "results[\"Stacking_nocat\"] = mean_std_cross_val_scores(\n",
    "    stacking_model, X_train, y_train, return_train_score=True, scoring=scoring_metric\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Dummy</th>\n",
       "      <td>0.003 (+/- 0.000)</td>\n",
       "      <td>0.002 (+/- 0.000)</td>\n",
       "      <td>0.636 (+/- 0.008)</td>\n",
       "      <td>0.633 (+/- 0.004)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.412 (+/- 0.063)</td>\n",
       "      <td>0.008 (+/- 0.001)</td>\n",
       "      <td>0.849 (+/- 0.004)</td>\n",
       "      <td>0.851 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>decision tree</th>\n",
       "      <td>0.064 (+/- 0.001)</td>\n",
       "      <td>0.007 (+/- 0.000)</td>\n",
       "      <td>0.811 (+/- 0.002)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>random forest</th>\n",
       "      <td>0.571 (+/- 0.024)</td>\n",
       "      <td>0.039 (+/- 0.001)</td>\n",
       "      <td>0.852 (+/- 0.004)</td>\n",
       "      <td>1.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LightGBM</th>\n",
       "      <td>0.115 (+/- 0.025)</td>\n",
       "      <td>0.012 (+/- 0.001)</td>\n",
       "      <td>0.868 (+/- 0.003)</td>\n",
       "      <td>0.907 (+/- 0.002)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CatBoost</th>\n",
       "      <td>3.084 (+/- 0.270)</td>\n",
       "      <td>0.055 (+/- 0.019)</td>\n",
       "      <td>0.870 (+/- 0.001)</td>\n",
       "      <td>0.906 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Voting</th>\n",
       "      <td>4.414 (+/- 0.489)</td>\n",
       "      <td>0.113 (+/- 0.009)</td>\n",
       "      <td>0.858 (+/- 0.004)</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Voting_ndt</th>\n",
       "      <td>4.040 (+/- 0.144)</td>\n",
       "      <td>0.099 (+/- 0.004)</td>\n",
       "      <td>0.868 (+/- 0.005)</td>\n",
       "      <td>0.929 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Stacking_nocat</th>\n",
       "      <td>6.650 (+/- 0.274)</td>\n",
       "      <td>0.062 (+/- 0.003)</td>\n",
       "      <td>0.870 (+/- 0.004)</td>\n",
       "      <td>0.902 (+/- 0.004)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              fit_time         score_time         test_score  \\\n",
       "Dummy                0.003 (+/- 0.000)  0.002 (+/- 0.000)  0.636 (+/- 0.008)   \n",
       "logistic regression  0.412 (+/- 0.063)  0.008 (+/- 0.001)  0.849 (+/- 0.004)   \n",
       "decision tree        0.064 (+/- 0.001)  0.007 (+/- 0.000)  0.811 (+/- 0.002)   \n",
       "random forest        0.571 (+/- 0.024)  0.039 (+/- 0.001)  0.852 (+/- 0.004)   \n",
       "LightGBM             0.115 (+/- 0.025)  0.012 (+/- 0.001)  0.868 (+/- 0.003)   \n",
       "CatBoost             3.084 (+/- 0.270)  0.055 (+/- 0.019)  0.870 (+/- 0.001)   \n",
       "Voting               4.414 (+/- 0.489)  0.113 (+/- 0.009)  0.858 (+/- 0.004)   \n",
       "Voting_ndt           4.040 (+/- 0.144)  0.099 (+/- 0.004)  0.868 (+/- 0.005)   \n",
       "Stacking_nocat       6.650 (+/- 0.274)  0.062 (+/- 0.003)  0.870 (+/- 0.004)   \n",
       "\n",
       "                           train_score  \n",
       "Dummy                0.633 (+/- 0.004)  \n",
       "logistic regression  0.851 (+/- 0.001)  \n",
       "decision tree        1.000 (+/- 0.000)  \n",
       "random forest        1.000 (+/- 0.000)  \n",
       "LightGBM             0.907 (+/- 0.002)  \n",
       "CatBoost             0.906 (+/- 0.001)  \n",
       "Voting                             NaN  \n",
       "Voting_ndt           0.929 (+/- 0.001)  \n",
       "Stacking_nocat       0.902 (+/- 0.004)  "
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The situation here is a bit mind-boggling.\n",
    "- On each fold of cross-validation it is doing cross-validation.\n",
    "- This is really loops within loops within loops within loops..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- We can also try a different final estimator:\n",
    "- Let's `DecisionTreeClassifier` as a final estimator. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "stacking_model_tree = StackingClassifier(\n",
    "    list(classifiers_nocat.items()), final_estimator=DecisionTreeClassifier(max_depth=3)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results are not very good. But we can look at the tree:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "stacking_model_tree.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "image/svg+xml": [
       "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"no\"?>\n",
       "<!DOCTYPE svg PUBLIC \"-//W3C//DTD SVG 1.1//EN\"\n",
       " \"http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd\">\n",
       "<!-- Generated by graphviz version 6.0.1 (0)\n",
       " -->\n",
       "<!-- Title: Tree Pages: 1 -->\n",
       "<svg width=\"761pt\" height=\"266pt\"\n",
       " viewBox=\"0.00 0.00 761.00 266.00\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n",
       "<g id=\"graph0\" class=\"graph\" transform=\"scale(1 1) rotate(0) translate(4 262)\">\n",
       "<title>Tree</title>\n",
       "<polygon fill=\"white\" stroke=\"none\" points=\"-4,4 -4,-262 757,-262 757,4 -4,4\"/>\n",
       "<!-- 0 -->\n",
       "<g id=\"node1\" class=\"node\">\n",
       "<title>0</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"447.5,-258 311.5,-258 311.5,-220 447.5,-220 447.5,-258\"/>\n",
       "<text text-anchor=\"middle\" x=\"379.5\" y=\"-242.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">LightGBM &lt;= 0.428</text>\n",
       "<text text-anchor=\"middle\" x=\"379.5\" y=\"-227.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0</text>\n",
       "</g>\n",
       "<!-- 1 -->\n",
       "<g id=\"node2\" class=\"node\">\n",
       "<title>1</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"343.5,-184 215.5,-184 215.5,-146 343.5,-146 343.5,-184\"/>\n",
       "<text text-anchor=\"middle\" x=\"279.5\" y=\"-168.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">LightGBM &lt;= 0.12</text>\n",
       "<text text-anchor=\"middle\" x=\"279.5\" y=\"-153.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;1 -->\n",
       "<g id=\"edge1\" class=\"edge\">\n",
       "<title>0&#45;&gt;1</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M354.27,-219.83C341.71,-210.79 326.37,-199.75 312.86,-190.02\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"314.89,-187.17 304.73,-184.16 310.8,-192.85 314.89,-187.17\"/>\n",
       "<text text-anchor=\"middle\" x=\"308.74\" y=\"-205.14\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">True</text>\n",
       "</g>\n",
       "<!-- 8 -->\n",
       "<g id=\"node9\" class=\"node\">\n",
       "<title>8</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"547.5,-184 411.5,-184 411.5,-146 547.5,-146 547.5,-184\"/>\n",
       "<text text-anchor=\"middle\" x=\"479.5\" y=\"-168.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">LightGBM &lt;= 0.776</text>\n",
       "<text text-anchor=\"middle\" x=\"479.5\" y=\"-153.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 1</text>\n",
       "</g>\n",
       "<!-- 0&#45;&gt;8 -->\n",
       "<g id=\"edge8\" class=\"edge\">\n",
       "<title>0&#45;&gt;8</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M404.73,-219.83C417.29,-210.79 432.63,-199.75 446.14,-190.02\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"448.2,-192.85 454.27,-184.16 444.11,-187.17 448.2,-192.85\"/>\n",
       "<text text-anchor=\"middle\" x=\"450.26\" y=\"-205.14\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">False</text>\n",
       "</g>\n",
       "<!-- 2 -->\n",
       "<g id=\"node3\" class=\"node\">\n",
       "<title>2</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"170.5,-110 34.5,-110 34.5,-72 170.5,-72 170.5,-110\"/>\n",
       "<text text-anchor=\"middle\" x=\"102.5\" y=\"-94.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">LightGBM &lt;= 0.032</text>\n",
       "<text text-anchor=\"middle\" x=\"102.5\" y=\"-79.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;2 -->\n",
       "<g id=\"edge2\" class=\"edge\">\n",
       "<title>1&#45;&gt;2</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M234.84,-145.83C210.95,-136.12 181.37,-124.08 156.24,-113.86\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"157.43,-110.56 146.84,-110.04 154.79,-117.05 157.43,-110.56\"/>\n",
       "</g>\n",
       "<!-- 5 -->\n",
       "<g id=\"node6\" class=\"node\">\n",
       "<title>5</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"370.5,-110 188.5,-110 188.5,-72 370.5,-72 370.5,-110\"/>\n",
       "<text text-anchor=\"middle\" x=\"279.5\" y=\"-94.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">logistic regression &lt;= 0.237</text>\n",
       "<text text-anchor=\"middle\" x=\"279.5\" y=\"-79.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0</text>\n",
       "</g>\n",
       "<!-- 1&#45;&gt;5 -->\n",
       "<g id=\"edge5\" class=\"edge\">\n",
       "<title>1&#45;&gt;5</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M279.5,-145.83C279.5,-138.13 279.5,-128.97 279.5,-120.42\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"283,-120.41 279.5,-110.41 276,-120.41 283,-120.41\"/>\n",
       "</g>\n",
       "<!-- 3 -->\n",
       "<g id=\"node4\" class=\"node\">\n",
       "<title>3</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"71,-36 0,-36 0,0 71,0 71,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"35.5\" y=\"-14.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;3 -->\n",
       "<g id=\"edge3\" class=\"edge\">\n",
       "<title>2&#45;&gt;3</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M85.25,-71.72C77.13,-63.11 67.33,-52.73 58.58,-43.46\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"60.95,-40.87 51.55,-36 55.86,-45.68 60.95,-40.87\"/>\n",
       "</g>\n",
       "<!-- 4 -->\n",
       "<g id=\"node5\" class=\"node\">\n",
       "<title>4</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"160,-36 89,-36 89,0 160,0 160,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"124.5\" y=\"-14.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0</text>\n",
       "</g>\n",
       "<!-- 2&#45;&gt;4 -->\n",
       "<g id=\"edge4\" class=\"edge\">\n",
       "<title>2&#45;&gt;4</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M108.17,-71.72C110.62,-63.8 113.54,-54.38 116.22,-45.71\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"119.61,-46.59 119.23,-36 112.93,-44.52 119.61,-46.59\"/>\n",
       "</g>\n",
       "<!-- 6 -->\n",
       "<g id=\"node7\" class=\"node\">\n",
       "<title>6</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"293,-36 222,-36 222,0 293,0 293,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"257.5\" y=\"-14.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;6 -->\n",
       "<g id=\"edge6\" class=\"edge\">\n",
       "<title>5&#45;&gt;6</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M273.83,-71.72C271.38,-63.8 268.46,-54.38 265.78,-45.71\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"269.07,-44.52 262.77,-36 262.39,-46.59 269.07,-44.52\"/>\n",
       "</g>\n",
       "<!-- 7 -->\n",
       "<g id=\"node8\" class=\"node\">\n",
       "<title>7</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"382,-36 311,-36 311,0 382,0 382,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"346.5\" y=\"-14.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0</text>\n",
       "</g>\n",
       "<!-- 5&#45;&gt;7 -->\n",
       "<g id=\"edge7\" class=\"edge\">\n",
       "<title>5&#45;&gt;7</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M296.75,-71.72C304.87,-63.11 314.67,-52.73 323.42,-43.46\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"326.14,-45.68 330.45,-36 321.05,-40.87 326.14,-45.68\"/>\n",
       "</g>\n",
       "<!-- 9 -->\n",
       "<g id=\"node10\" class=\"node\">\n",
       "<title>9</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"570.5,-110 388.5,-110 388.5,-72 570.5,-72 570.5,-110\"/>\n",
       "<text text-anchor=\"middle\" x=\"479.5\" y=\"-94.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">logistic regression &lt;= 0.404</text>\n",
       "<text text-anchor=\"middle\" x=\"479.5\" y=\"-79.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 1</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;9 -->\n",
       "<g id=\"edge9\" class=\"edge\">\n",
       "<title>8&#45;&gt;9</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M479.5,-145.83C479.5,-138.13 479.5,-128.97 479.5,-120.42\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"483,-120.41 479.5,-110.41 476,-120.41 483,-120.41\"/>\n",
       "</g>\n",
       "<!-- 12 -->\n",
       "<g id=\"node13\" class=\"node\">\n",
       "<title>12</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"724.5,-110 588.5,-110 588.5,-72 724.5,-72 724.5,-110\"/>\n",
       "<text text-anchor=\"middle\" x=\"656.5\" y=\"-94.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">LightGBM &lt;= 0.911</text>\n",
       "<text text-anchor=\"middle\" x=\"656.5\" y=\"-79.8\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 1</text>\n",
       "</g>\n",
       "<!-- 8&#45;&gt;12 -->\n",
       "<g id=\"edge12\" class=\"edge\">\n",
       "<title>8&#45;&gt;12</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M524.16,-145.83C548.05,-136.12 577.63,-124.08 602.76,-113.86\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"604.21,-117.05 612.16,-110.04 601.57,-110.56 604.21,-117.05\"/>\n",
       "</g>\n",
       "<!-- 10 -->\n",
       "<g id=\"node11\" class=\"node\">\n",
       "<title>10</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"481,-36 410,-36 410,0 481,0 481,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"445.5\" y=\"-14.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 0</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;10 -->\n",
       "<g id=\"edge10\" class=\"edge\">\n",
       "<title>9&#45;&gt;10</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M470.74,-71.72C466.87,-63.63 462.25,-53.97 458.02,-45.14\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"461.12,-43.51 453.64,-36 454.8,-46.53 461.12,-43.51\"/>\n",
       "</g>\n",
       "<!-- 11 -->\n",
       "<g id=\"node12\" class=\"node\">\n",
       "<title>11</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"570,-36 499,-36 499,0 570,0 570,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"534.5\" y=\"-14.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 1</text>\n",
       "</g>\n",
       "<!-- 9&#45;&gt;11 -->\n",
       "<g id=\"edge11\" class=\"edge\">\n",
       "<title>9&#45;&gt;11</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M493.66,-71.72C500.19,-63.29 508.05,-53.15 515.12,-44.02\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"517.97,-46.05 521.33,-36 512.44,-41.77 517.97,-46.05\"/>\n",
       "</g>\n",
       "<!-- 13 -->\n",
       "<g id=\"node14\" class=\"node\">\n",
       "<title>13</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"664,-36 593,-36 593,0 664,0 664,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"628.5\" y=\"-14.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 1</text>\n",
       "</g>\n",
       "<!-- 12&#45;&gt;13 -->\n",
       "<g id=\"edge13\" class=\"edge\">\n",
       "<title>12&#45;&gt;13</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M649.29,-71.72C646.13,-63.72 642.37,-54.18 638.92,-45.42\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"642.13,-44.02 635.21,-36 635.62,-46.59 642.13,-44.02\"/>\n",
       "</g>\n",
       "<!-- 14 -->\n",
       "<g id=\"node15\" class=\"node\">\n",
       "<title>14</title>\n",
       "<polygon fill=\"none\" stroke=\"black\" points=\"753,-36 682,-36 682,0 753,0 753,-36\"/>\n",
       "<text text-anchor=\"middle\" x=\"717.5\" y=\"-14.3\" font-family=\"Helvetica,sans-Serif\" font-size=\"14.00\">class = 1</text>\n",
       "</g>\n",
       "<!-- 12&#45;&gt;14 -->\n",
       "<g id=\"edge14\" class=\"edge\">\n",
       "<title>12&#45;&gt;14</title>\n",
       "<path fill=\"none\" stroke=\"black\" d=\"M672.21,-71.72C679.53,-63.2 688.34,-52.94 696.24,-43.74\"/>\n",
       "<polygon fill=\"black\" stroke=\"black\" points=\"699.03,-45.87 702.89,-36 693.72,-41.31 699.03,-45.87\"/>\n",
       "</g>\n",
       "</g>\n",
       "</svg>\n"
      ],
      "text/plain": [
       "<graphviz.sources.Source at 0x121de8580>"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "display_tree(list(classifiers_nocat.keys()), stacking_model_tree.final_estimator_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### An effective strategy\n",
    "\n",
    "- Randomly generate a bunch of models with different hyperparameter configurations, and then stack all the models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- What is an advantage of ensembling multiple models as opposed to just choosing one of them?\n",
    "    - You may get a better score.\n",
    "- What is an disadvantage of ensembling multiple models as opposed to just choosing one of them?\n",
    "    - Slower, more code maintenance issues."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Summary \n",
    "\n",
    "- You have a number of models in your toolbox now.  \n",
    "- Ensembles are usually pretty effective.\n",
    "  - Tree-based classifiers are particularly popular and effective on a wide range of problems. \n",
    "  - But they trade off code complexity and speed for prediction accuracy.\n",
    "  - Don't forget that hyperparameter optimization multiplies the slowness of the code!\n",
    "- Stacking is a bit slower than voting, but generally higher accuracy.\n",
    "  - As a bonus, you get to see the coefficients for each base classifier.\n",
    "- All the above models have equivalent regression models.   "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Relevant papers\n",
    "\n",
    "- [Fernandez-Delgado et al. 2014](http://jmlr.org/papers/volume15/delgado14a/delgado14a.pdf) compared 179 classifiers on 121 datasets:\n",
    "    - First best class of methods was Random Forest and second best class of methods was (RBF) SVMs.\n",
    "\n",
    "- If you like to read original papers [here](https://www.stat.berkeley.edu/~breiman/randomforest2001.pdf) is the original paper on Random Forests by Leo Breiman. "
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}