lectures/06_column-transformer-text-feats.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 6: `sklearn` `ColumnTransformer` and Text Features\n",
    "\n",
    "UBC 2022-23\n",
    "\n",
    "Instructor: Varada Kolhatkar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports, Announcements, and LO"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "from IPython.display import HTML\n",
    "\n",
    "sys.path.append(\"code/.\")\n",
    "from plotting_functions import *\n",
    "from utils import *\n",
    "\n",
    "pd.set_option(\"display.max_colwidth\", 200)\n",
    "\n",
    "from sklearn.compose import ColumnTransformer, make_column_transformer\n",
    "from sklearn.dummy import DummyClassifier, DummyRegressor\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.model_selection import cross_val_score, cross_validate, train_test_split\n",
    "from sklearn.neighbors import KNeighborsClassifier\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler\n",
    "from sklearn.svm import SVC\n",
    "from sklearn.tree import DecisionTreeClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Announcements\n",
    "\n",
    "- Homework 3 is due on Oct 3rd. \n",
    "- Friday is a holiday. So no tutorial this Friday. \n",
    "- Scheduled extra office hours this week. \n",
    "    - Check the [Calendar](https://htmlpreview.github.io/?https://github.com/UBC-CS/cpsc330/blob/master/docs/calendar.html).\n",
    "- We're working on Homework 2 grading. The grades will be released later this week.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    },
    "tags": []
   },
   "source": [
    "### Learning outcomes \n",
    "\n",
    "From this lecture, you will be able to \n",
    "\n",
    "- use `ColumnTransformer` to build all our transformations together into one object and use it with `sklearn` pipelines;  \n",
    "- define `ColumnTransformer` where transformers contain more than one steps;\n",
    "- explain `handle_unknown=\"ignore\"` hyperparameter of `scikit-learn`'s `OneHotEncoder`;\n",
    "- explain `drop=\"if_binary\"` argument of `OneHotEncoder`;\n",
    "- identify when it's appropriate to apply ordinal encoding vs one-hot encoding;\n",
    "- explain strategies to deal with categorical variables with too many categories; \n",
    "- explain why text data needs a different treatment than categorical variables;\n",
    "- use `scikit-learn`'s `CountVectorizer` to encode text data;\n",
    "- explain different hyperparameters of `CountVectorizer`.\n",
    "- incorporate text features in a machine learning pipeline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "## sklearn's [`ColumnTransformer`](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- In most applications, some features are categorical, some are continuous, some are binary, and some are ordinal. \n",
    "\n",
    "- When we want to develop supervised machine learning pipelines on real-world datasets, very often we want to apply different transformation on different columns. \n",
    "\n",
    "- Enter `sklearn`'s `ColumnTransformer`!! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "- Let's look at a toy example: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>quiz2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Economics</td>\n",
       "      <td>Good</td>\n",
       "      <td>5</td>\n",
       "      <td>70</td>\n",
       "      <td>73.0</td>\n",
       "      <td>68</td>\n",
       "      <td>74</td>\n",
       "      <td>71</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>80</td>\n",
       "      <td>88.0</td>\n",
       "      <td>89</td>\n",
       "      <td>88</td>\n",
       "      <td>91</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>95</td>\n",
       "      <td>93.0</td>\n",
       "      <td>69</td>\n",
       "      <td>79</td>\n",
       "      <td>75</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Linguistics</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>97</td>\n",
       "      <td>90.0</td>\n",
       "      <td>94</td>\n",
       "      <td>82</td>\n",
       "      <td>80</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Average</td>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>82.0</td>\n",
       "      <td>94</td>\n",
       "      <td>94</td>\n",
       "      <td>85</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>3</td>\n",
       "      <td>98</td>\n",
       "      <td>86.0</td>\n",
       "      <td>95</td>\n",
       "      <td>95</td>\n",
       "      <td>78</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Physics</td>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "      <td>95</td>\n",
       "      <td>88.0</td>\n",
       "      <td>93</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Physics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>2</td>\n",
       "      <td>98</td>\n",
       "      <td>96.0</td>\n",
       "      <td>96</td>\n",
       "      <td>99</td>\n",
       "      <td>100</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>94.0</td>\n",
       "      <td>96</td>\n",
       "      <td>95</td>\n",
       "      <td>100</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>95</td>\n",
       "      <td>90.0</td>\n",
       "      <td>93</td>\n",
       "      <td>95</td>\n",
       "      <td>70</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Good</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>85.0</td>\n",
       "      <td>67</td>\n",
       "      <td>94</td>\n",
       "      <td>92</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Average</td>\n",
       "      <td>5</td>\n",
       "      <td>75</td>\n",
       "      <td>91.0</td>\n",
       "      <td>93</td>\n",
       "      <td>86</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Economics</td>\n",
       "      <td>Average</td>\n",
       "      <td>3</td>\n",
       "      <td>86</td>\n",
       "      <td>89.0</td>\n",
       "      <td>65</td>\n",
       "      <td>86</td>\n",
       "      <td>87</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Biology</td>\n",
       "      <td>Good</td>\n",
       "      <td>2</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>90</td>\n",
       "      <td>88</td>\n",
       "      <td>82</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Poor</td>\n",
       "      <td>2</td>\n",
       "      <td>77</td>\n",
       "      <td>94.0</td>\n",
       "      <td>87</td>\n",
       "      <td>81</td>\n",
       "      <td>89</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Linguistics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>96</td>\n",
       "      <td>92.0</td>\n",
       "      <td>92</td>\n",
       "      <td>96</td>\n",
       "      <td>87</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0           yes              1        Computer Science        Excellent   \n",
       "1           yes              1  Mechanical Engineering          Average   \n",
       "2           yes              0             Mathematics             Poor   \n",
       "3            no              0             Mathematics        Excellent   \n",
       "4           yes              0              Psychology             Good   \n",
       "5            no              1               Economics             Good   \n",
       "6           yes              1        Computer Science        Excellent   \n",
       "7            no              0  Mechanical Engineering             Poor   \n",
       "8            no              0             Linguistics          Average   \n",
       "9           yes              1             Mathematics          Average   \n",
       "10          yes              0              Psychology             Good   \n",
       "11          yes              1                 Physics          Average   \n",
       "12          yes              1                 Physics        Excellent   \n",
       "13          yes              0  Mechanical Engineering        Excellent   \n",
       "14           no              0             Mathematics             Poor   \n",
       "15           no              1        Computer Science             Good   \n",
       "16          yes              0        Computer Science          Average   \n",
       "17          yes              1               Economics          Average   \n",
       "18           no              1                 Biology             Good   \n",
       "19           no              0              Psychology             Poor   \n",
       "20          yes              1             Linguistics        Excellent   \n",
       "\n",
       "    university_years  lab1  lab2  lab3  lab4  quiz1   quiz2  \n",
       "0                  3    92  93.0    84    91     92      A+  \n",
       "1                  2    94  90.0    80    83     91  not A+  \n",
       "2                  3    78  85.0    83    80     80  not A+  \n",
       "3                  3    91   NaN    92    91     89      A+  \n",
       "4                  4    77  83.0    90    92     85      A+  \n",
       "5                  5    70  73.0    68    74     71  not A+  \n",
       "6                  4    80  88.0    89    88     91      A+  \n",
       "7                  3    95  93.0    69    79     75  not A+  \n",
       "8                  2    97  90.0    94    82     80  not A+  \n",
       "9                  4    95  82.0    94    94     85  not A+  \n",
       "10                 3    98  86.0    95    95     78      A+  \n",
       "11                 1    95  88.0    93    92     85      A+  \n",
       "12                 2    98  96.0    96    99    100      A+  \n",
       "13                 4    95  94.0    96    95    100      A+  \n",
       "14                 3    95  90.0    93    95     70  not A+  \n",
       "15                 3    92  85.0    67    94     92  not A+  \n",
       "16                 5    75  91.0    93    86     85      A+  \n",
       "17                 3    86  89.0    65    86     87  not A+  \n",
       "18                 2    91   NaN    90    88     82  not A+  \n",
       "19                 2    77  94.0    87    81     89  not A+  \n",
       "20                 4    96  92.0    92    96     87      A+  "
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"data/quiz2-grade-toy-col-transformer.csv\")\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 21 entries, 0 to 20\n",
      "Data columns (total 11 columns):\n",
      " #   Column            Non-Null Count  Dtype  \n",
      "---  ------            --------------  -----  \n",
      " 0   enjoy_course      21 non-null     object \n",
      " 1   ml_experience     21 non-null     int64  \n",
      " 2   major             21 non-null     object \n",
      " 3   class_attendance  21 non-null     object \n",
      " 4   university_years  21 non-null     int64  \n",
      " 5   lab1              21 non-null     int64  \n",
      " 6   lab2              19 non-null     float64\n",
      " 7   lab3              21 non-null     int64  \n",
      " 8   lab4              21 non-null     int64  \n",
      " 9   quiz1             21 non-null     int64  \n",
      " 10  quiz2             21 non-null     object \n",
      "dtypes: float64(1), int64(6), object(4)\n",
      "memory usage: 1.9+ KB\n"
     ]
    }
   ],
   "source": [
    "df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Transformations on the toy data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>quiz2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "      <td>not A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "      <td>A+</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1   quiz2  \n",
       "0                 3    92  93.0    84    91     92      A+  \n",
       "1                 2    94  90.0    80    83     91  not A+  \n",
       "2                 3    78  85.0    83    80     80  not A+  \n",
       "3                 3    91   NaN    92    91     89      A+  \n",
       "4                 4    77  83.0    90    92     85      A+  "
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Scaling on numeric features\n",
    "- One-hot encoding on the categorical feature `major` and binary feature `enjoy_class`\n",
    "- Ordinal encoding on the ordinal feature `class_attendance`\n",
    "- Imputation on the `lab2` feature\n",
    "- None on the `ml_experience` feature"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `ColumnTransformer` example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['enjoy_course', 'ml_experience', 'major', 'class_attendance',\n",
       "       'university_years', 'lab1', 'lab2', 'lab3', 'lab4', 'quiz1'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X = df.drop(columns=[\"quiz2\"])\n",
    "y = df[\"quiz2\"]\n",
    "X.columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Identify the transformations we want to apply"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1  \n",
       "0                 3    92  93.0    84    91     92  \n",
       "1                 2    94  90.0    80    83     91  \n",
       "2                 3    78  85.0    83    80     80  \n",
       "3                 3    91   NaN    92    91     89  \n",
       "4                 4    77  83.0    90    92     85  "
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_feats = [\"university_years\", \"lab1\", \"lab3\", \"lab4\", \"quiz1\"]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = [\n",
    "    \"lab2\",\n",
    "    \"class_attendance\",\n",
    "    \"enjoy_course\",\n",
    "]  # do not include these features in modeling"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For simplicity, let's only focus on scaling and one-hot encoding first. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Create a column transformer"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Each transformation is specified by a name, a transformer object, and the columns this transformer should be applied to. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "ct = ColumnTransformer(\n",
    "    [\n",
    "        (\"scaling\", StandardScaler(), numeric_feats),\n",
    "        (\"onehot\", OneHotEncoder(sparse=False), categorical_feats),\n",
    "    ]\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Convenient `make_column_transformer` syntax\n",
    "\n",
    "- Similar to `make_pipeline` syntax, there is convenient `make_column_transformer` syntax. \n",
    "- The syntax automatically names each step based on its class. \n",
    "- We'll be mostly using this syntax. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.compose import make_column_transformer\n",
    "\n",
    "ct = make_column_transformer(    \n",
    "    (StandardScaler(), numeric_feats),  # scaling on numeric features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features    \n",
    "    (OneHotEncoder(), categorical_feats),  # OHE on categorical features\n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;standardscaler&#x27;, StandardScaler(),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;,\n",
       "                                  &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;lab2&#x27;, &#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;standardscaler&#x27;, StandardScaler(),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;,\n",
       "                                  &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;lab2&#x27;, &#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">standardscaler</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;lab2&#x27;, &#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('standardscaler', StandardScaler(),\n",
       "                                 ['university_years', 'lab1', 'lab3', 'lab4',\n",
       "                                  'quiz1']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('drop', 'drop',\n",
       "                                 ['lab2', 'class_attendance', 'enjoy_course'])])"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "transformed = ct.fit_transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "source": [
    "- When we `fit_transform`, each transformer is applied to the specified columns and the result of the transformations are concatenated horizontally. \n",
    "- A big advantage here is that we build all our transformations together into one object, and that way we're sure we do the same operations to all splits of the data.\n",
    "- Otherwise we might, for example, do the OHE on both train and test but forget to scale the test data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Let's examine the transformed data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "numpy.ndarray"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(transformed[:2])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([[-0.09345386,  0.3589134 , -0.21733442,  0.36269995,  0.84002795,\n",
       "         1.        ,  0.        ,  1.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-1.07471942,  0.59082668, -0.61420598, -0.85597188,  0.71219761,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  1.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386, -1.26447953, -0.31655231, -1.31297381, -0.69393613,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386,  0.24295676,  0.57640869,  0.36269995,  0.45653693,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [ 0.8878117 , -1.38043616,  0.37797291,  0.51503393, -0.05478443,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [ 1.86907725, -2.19213263, -1.80482065, -2.22697768, -1.84440919,\n",
       "         1.        ,  0.        ,  0.        ,  1.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [ 0.8878117 , -1.03256625,  0.27875502, -0.09430199,  0.71219761,\n",
       "         1.        ,  0.        ,  1.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386,  0.70678332, -1.70560276, -1.46530779, -1.33308783,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  1.        ,  0.        ,  0.        ],\n",
       "       [-1.07471942,  0.93869659,  0.77484447, -1.00830586, -0.69393613,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  1.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [ 0.8878117 ,  0.70678332,  0.77484447,  0.81970188, -0.05478443,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386,  1.05465323,  0.87406235,  0.97203586, -0.94959681,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [-2.05598498,  0.70678332,  0.67562658,  0.51503393, -0.05478443,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  1.        ,  0.        ],\n",
       "       [-1.07471942,  1.05465323,  0.97328024,  1.58137177,  1.86267067,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  1.        ,  0.        ],\n",
       "       [ 0.8878117 ,  0.70678332,  0.97328024,  0.97203586,  1.86267067,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  1.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386,  0.70678332,  0.67562658,  0.97203586, -1.97223953,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386,  0.3589134 , -1.90403853,  0.81970188,  0.84002795,\n",
       "         1.        ,  0.        ,  1.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [ 1.86907725, -1.61234944,  0.67562658, -0.39896994, -0.05478443,\n",
       "         0.        ,  0.        ,  1.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-0.09345386, -0.33682642, -2.10247431, -0.39896994,  0.20087625,\n",
       "         1.        ,  0.        ,  0.        ,  1.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-1.07471942,  0.24295676,  0.37797291, -0.09430199, -0.43827545,\n",
       "         1.        ,  1.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ],\n",
       "       [-1.07471942, -1.38043616,  0.08031924, -1.16063983,  0.45653693,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ,  0.        ,\n",
       "         0.        ,  0.        ,  0.        ,  1.        ],\n",
       "       [ 0.8878117 ,  0.82273995,  0.57640869,  1.12436984,  0.20087625,\n",
       "         1.        ,  0.        ,  0.        ,  0.        ,  1.        ,\n",
       "         0.        ,  0.        ,  0.        ,  0.        ]])"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformed"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "Note that the returned object is not a dataframe. So there are no column names. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### Viewing the transformed data as a dataframe\n",
    "\n",
    "- How can we view our transformed data as a dataframe? \n",
    "- We are adding more columns. \n",
    "- So the original columns won't directly map to the transformed data. \n",
    "- Let's create column names for the transformed data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['university_years',\n",
       " 'lab1',\n",
       " 'lab3',\n",
       " 'lab4',\n",
       " 'quiz1',\n",
       " 'ml_experience',\n",
       " 'major_Biology',\n",
       " 'major_Computer Science',\n",
       " 'major_Economics',\n",
       " 'major_Linguistics',\n",
       " 'major_Mathematics',\n",
       " 'major_Mechanical Engineering',\n",
       " 'major_Physics',\n",
       " 'major_Psychology']"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = (\n",
    "    numeric_feats\n",
    "    + passthrough_feats    \n",
    "    + ct.named_transformers_[\"onehotencoder\"].get_feature_names_out().tolist()\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'standardscaler': StandardScaler(),\n",
       " 'passthrough': 'passthrough',\n",
       " 'onehotencoder': OneHotEncoder(),\n",
       " 'drop': 'drop'}"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct.named_transformers_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```{note}\n",
    "Note that the order of the columns in the transformed data depends upon the order of the features we pass to the `ColumnTransformer` and can be different than the order of the features in the original dataframe.  \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major_Biology</th>\n",
       "      <th>major_Computer Science</th>\n",
       "      <th>major_Economics</th>\n",
       "      <th>major_Linguistics</th>\n",
       "      <th>major_Mathematics</th>\n",
       "      <th>major_Mechanical Engineering</th>\n",
       "      <th>major_Physics</th>\n",
       "      <th>major_Psychology</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>-0.217334</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.590827</td>\n",
       "      <td>-0.614206</td>\n",
       "      <td>-0.855972</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-1.264480</td>\n",
       "      <td>-0.316552</td>\n",
       "      <td>-1.312974</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-2.192133</td>\n",
       "      <td>-1.804821</td>\n",
       "      <td>-2.226978</td>\n",
       "      <td>-1.844409</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.032566</td>\n",
       "      <td>0.278755</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-1.705603</td>\n",
       "      <td>-1.465308</td>\n",
       "      <td>-1.333088</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.938697</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>-1.008306</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>0.874062</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-0.949597</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-2.055985</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>1.581372</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-1.972240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>-1.904039</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-1.612349</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-0.336826</td>\n",
       "      <td>-2.102474</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>-0.438275</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>0.080319</td>\n",
       "      <td>-1.160640</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.822740</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>1.124370</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    university_years      lab1      lab3      lab4     quiz1  ml_experience  \\\n",
       "0          -0.093454  0.358913 -0.217334  0.362700  0.840028            1.0   \n",
       "1          -1.074719  0.590827 -0.614206 -0.855972  0.712198            1.0   \n",
       "2          -0.093454 -1.264480 -0.316552 -1.312974 -0.693936            0.0   \n",
       "3          -0.093454  0.242957  0.576409  0.362700  0.456537            0.0   \n",
       "4           0.887812 -1.380436  0.377973  0.515034 -0.054784            0.0   \n",
       "5           1.869077 -2.192133 -1.804821 -2.226978 -1.844409            1.0   \n",
       "6           0.887812 -1.032566  0.278755 -0.094302  0.712198            1.0   \n",
       "7          -0.093454  0.706783 -1.705603 -1.465308 -1.333088            0.0   \n",
       "8          -1.074719  0.938697  0.774844 -1.008306 -0.693936            0.0   \n",
       "9           0.887812  0.706783  0.774844  0.819702 -0.054784            1.0   \n",
       "10         -0.093454  1.054653  0.874062  0.972036 -0.949597            0.0   \n",
       "11         -2.055985  0.706783  0.675627  0.515034 -0.054784            1.0   \n",
       "12         -1.074719  1.054653  0.973280  1.581372  1.862671            1.0   \n",
       "13          0.887812  0.706783  0.973280  0.972036  1.862671            0.0   \n",
       "14         -0.093454  0.706783  0.675627  0.972036 -1.972240            0.0   \n",
       "15         -0.093454  0.358913 -1.904039  0.819702  0.840028            1.0   \n",
       "16          1.869077 -1.612349  0.675627 -0.398970 -0.054784            0.0   \n",
       "17         -0.093454 -0.336826 -2.102474 -0.398970  0.200876            1.0   \n",
       "18         -1.074719  0.242957  0.377973 -0.094302 -0.438275            1.0   \n",
       "19         -1.074719 -1.380436  0.080319 -1.160640  0.456537            0.0   \n",
       "20          0.887812  0.822740  0.576409  1.124370  0.200876            1.0   \n",
       "\n",
       "    major_Biology  major_Computer Science  major_Economics  major_Linguistics  \\\n",
       "0             0.0                     1.0              0.0                0.0   \n",
       "1             0.0                     0.0              0.0                0.0   \n",
       "2             0.0                     0.0              0.0                0.0   \n",
       "3             0.0                     0.0              0.0                0.0   \n",
       "4             0.0                     0.0              0.0                0.0   \n",
       "5             0.0                     0.0              1.0                0.0   \n",
       "6             0.0                     1.0              0.0                0.0   \n",
       "7             0.0                     0.0              0.0                0.0   \n",
       "8             0.0                     0.0              0.0                1.0   \n",
       "9             0.0                     0.0              0.0                0.0   \n",
       "10            0.0                     0.0              0.0                0.0   \n",
       "11            0.0                     0.0              0.0                0.0   \n",
       "12            0.0                     0.0              0.0                0.0   \n",
       "13            0.0                     0.0              0.0                0.0   \n",
       "14            0.0                     0.0              0.0                0.0   \n",
       "15            0.0                     1.0              0.0                0.0   \n",
       "16            0.0                     1.0              0.0                0.0   \n",
       "17            0.0                     0.0              1.0                0.0   \n",
       "18            1.0                     0.0              0.0                0.0   \n",
       "19            0.0                     0.0              0.0                0.0   \n",
       "20            0.0                     0.0              0.0                1.0   \n",
       "\n",
       "    major_Mathematics  major_Mechanical Engineering  major_Physics  \\\n",
       "0                 0.0                           0.0            0.0   \n",
       "1                 0.0                           1.0            0.0   \n",
       "2                 1.0                           0.0            0.0   \n",
       "3                 1.0                           0.0            0.0   \n",
       "4                 0.0                           0.0            0.0   \n",
       "5                 0.0                           0.0            0.0   \n",
       "6                 0.0                           0.0            0.0   \n",
       "7                 0.0                           1.0            0.0   \n",
       "8                 0.0                           0.0            0.0   \n",
       "9                 1.0                           0.0            0.0   \n",
       "10                0.0                           0.0            0.0   \n",
       "11                0.0                           0.0            1.0   \n",
       "12                0.0                           0.0            1.0   \n",
       "13                0.0                           1.0            0.0   \n",
       "14                1.0                           0.0            0.0   \n",
       "15                0.0                           0.0            0.0   \n",
       "16                0.0                           0.0            0.0   \n",
       "17                0.0                           0.0            0.0   \n",
       "18                0.0                           0.0            0.0   \n",
       "19                0.0                           0.0            0.0   \n",
       "20                0.0                           0.0            0.0   \n",
       "\n",
       "    major_Psychology  \n",
       "0                0.0  \n",
       "1                0.0  \n",
       "2                0.0  \n",
       "3                0.0  \n",
       "4                1.0  \n",
       "5                0.0  \n",
       "6                0.0  \n",
       "7                0.0  \n",
       "8                0.0  \n",
       "9                0.0  \n",
       "10               1.0  \n",
       "11               0.0  \n",
       "12               0.0  \n",
       "13               0.0  \n",
       "14               0.0  \n",
       "15               0.0  \n",
       "16               0.0  \n",
       "17               0.0  \n",
       "18               0.0  \n",
       "19               1.0  \n",
       "20               0.0  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(transformed, columns=column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "#### `ColumnTransformer`: Transformed data\n",
    "\n",
    "<br>\n",
    "\n",
    "![](img/column-transformer.png)\n",
    "<!-- <img src='./img/column-transformer.png' width=\"1500\"> -->\n",
    "\n",
    "[Adapted from here.](https://amueller.github.io/COMS4995-s20/slides/aml-04-preprocessing/#37)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Training models with transformed data\n",
    "- We can now pass the `ColumnTransformer` object as a step in a pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['A+', 'not A+', 'not A+', 'A+', 'A+', 'not A+', 'A+', 'not A+',\n",
       "       'not A+', 'A+', 'A+', 'A+', 'A+', 'A+', 'not A+', 'not A+', 'A+',\n",
       "       'not A+', 'not A+', 'not A+', 'A+'], dtype=object)"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe = make_pipeline(ct, SVC())\n",
    "pipe.fit(X, y)\n",
    "pipe.predict(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-2 {color: black;background-color: white;}#sk-container-id-2 pre{padding: 0;}#sk-container-id-2 div.sk-toggleable {background-color: white;}#sk-container-id-2 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-2 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-2 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-2 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-2 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-2 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-2 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-2 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-2 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-2 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-2 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-2 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-2 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-2 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-2 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-2 div.sk-item {position: relative;z-index: 1;}#sk-container-id-2 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-2 div.sk-item::before, #sk-container-id-2 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-2 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-2 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-2 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-2 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-2 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-2 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-2 div.sk-label-container {text-align: center;}#sk-container-id-2 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-2 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-2\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler(),\n",
       "                                                  [&#x27;university_years&#x27;, &#x27;lab1&#x27;,\n",
       "                                                   &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                                 (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                                  [&#x27;ml_experience&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                                 (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                                  [&#x27;lab2&#x27;, &#x27;class_attendance&#x27;,\n",
       "                                                   &#x27;enjoy_course&#x27;])])),\n",
       "                (&#x27;svc&#x27;, SVC())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler(),\n",
       "                                                  [&#x27;university_years&#x27;, &#x27;lab1&#x27;,\n",
       "                                                   &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                                 (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                                  [&#x27;ml_experience&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                                 (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                                  [&#x27;lab2&#x27;, &#x27;class_attendance&#x27;,\n",
       "                                                   &#x27;enjoy_course&#x27;])])),\n",
       "                (&#x27;svc&#x27;, SVC())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;standardscaler&#x27;, StandardScaler(),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;,\n",
       "                                  &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;lab2&#x27;, &#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-12\" type=\"checkbox\" ><label for=\"sk-estimator-id-12\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">standardscaler</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-13\" type=\"checkbox\" ><label for=\"sk-estimator-id-13\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-14\" type=\"checkbox\" ><label for=\"sk-estimator-id-14\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-15\" type=\"checkbox\" ><label for=\"sk-estimator-id-15\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-16\" type=\"checkbox\" ><label for=\"sk-estimator-id-16\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-17\" type=\"checkbox\" ><label for=\"sk-estimator-id-17\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-18\" type=\"checkbox\" ><label for=\"sk-estimator-id-18\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;lab2&#x27;, &#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-19\" type=\"checkbox\" ><label for=\"sk-estimator-id-19\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-20\" type=\"checkbox\" ><label for=\"sk-estimator-id-20\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('columntransformer',\n",
       "                 ColumnTransformer(transformers=[('standardscaler',\n",
       "                                                  StandardScaler(),\n",
       "                                                  ['university_years', 'lab1',\n",
       "                                                   'lab3', 'lab4', 'quiz1']),\n",
       "                                                 ('passthrough', 'passthrough',\n",
       "                                                  ['ml_experience']),\n",
       "                                                 ('onehotencoder',\n",
       "                                                  OneHotEncoder(), ['major']),\n",
       "                                                 ('drop', 'drop',\n",
       "                                                  ['lab2', 'class_attendance',\n",
       "                                                   'enjoy_course'])])),\n",
       "                ('svc', SVC())])"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ❓❓ Questions for you "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "source": [
    "### (iClicker) Exercise 6.1 \n",
    "\n",
    "**iClicker cloud join link: https://join.iclicker.com/3DP5H**\n",
    "\n",
    "**Select all of the following statements which are TRUE.**\n",
    "\n",
    "1. You could carry out cross-validation by passing a `ColumnTransformer` object to `cross_validate`. \n",
    "2. After applying column transformer, the order of the columns in the transformed data has to be the same as the order of the columns in the original data. \n",
    "3. After applying a column transformer, the transformed data is always going to be of different shape than the original data. \n",
    "4. When you call `fit_transform` on a `ColumnTransformer` object, you get a numpy ndarray. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## More on feature transformations"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### `sklearn` `set_config`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- With multiple transformations in a column transformer, it can get tricky to keep track of everything happening inside it.  \n",
    "- We can use `set_config` to display a diagram of this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn import set_config\n",
    "\n",
    "set_config(display=\"diagram\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-3 {color: black;background-color: white;}#sk-container-id-3 pre{padding: 0;}#sk-container-id-3 div.sk-toggleable {background-color: white;}#sk-container-id-3 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-3 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-3 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-3 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-3 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-3 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-3 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-3 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-3 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-3 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-3 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-3 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-3 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-3 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-3 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-3 div.sk-item {position: relative;z-index: 1;}#sk-container-id-3 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-3 div.sk-item::before, #sk-container-id-3 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-3 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-3 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-3 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-3 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-3 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-3 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-3 div.sk-label-container {text-align: center;}#sk-container-id-3 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-3 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-3\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;standardscaler&#x27;, StandardScaler(),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;,\n",
       "                                  &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;lab2&#x27;, &#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-21\" type=\"checkbox\" ><label for=\"sk-estimator-id-21\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;standardscaler&#x27;, StandardScaler(),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;,\n",
       "                                  &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;lab2&#x27;, &#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-22\" type=\"checkbox\" ><label for=\"sk-estimator-id-22\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">standardscaler</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-23\" type=\"checkbox\" ><label for=\"sk-estimator-id-23\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-24\" type=\"checkbox\" ><label for=\"sk-estimator-id-24\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-25\" type=\"checkbox\" ><label for=\"sk-estimator-id-25\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-26\" type=\"checkbox\" ><label for=\"sk-estimator-id-26\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-27\" type=\"checkbox\" ><label for=\"sk-estimator-id-27\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-28\" type=\"checkbox\" ><label for=\"sk-estimator-id-28\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;lab2&#x27;, &#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-29\" type=\"checkbox\" ><label for=\"sk-estimator-id-29\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('standardscaler', StandardScaler(),\n",
       "                                 ['university_years', 'lab1', 'lab3', 'lab4',\n",
       "                                  'quiz1']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('drop', 'drop',\n",
       "                                 ['lab2', 'class_attendance', 'enjoy_course'])])"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "ColumnTransformer(transformers=[('standardscaler', StandardScaler(),\n",
      "                                 ['university_years', 'lab1', 'lab3', 'lab4',\n",
      "                                  'quiz1']),\n",
      "                                ('passthrough', 'passthrough',\n",
      "                                 ['ml_experience']),\n",
      "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
      "                                ('drop', 'drop',\n",
      "                                 ['lab2', 'class_attendance', 'enjoy_course'])])\n"
     ]
    }
   ],
   "source": [
    "print(ct)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Multiple transformations in a transformer\n",
    "\n",
    "- Recall that `lab2` has missing values. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Economics</td>\n",
       "      <td>Good</td>\n",
       "      <td>5</td>\n",
       "      <td>70</td>\n",
       "      <td>73.0</td>\n",
       "      <td>68</td>\n",
       "      <td>74</td>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>80</td>\n",
       "      <td>88.0</td>\n",
       "      <td>89</td>\n",
       "      <td>88</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>95</td>\n",
       "      <td>93.0</td>\n",
       "      <td>69</td>\n",
       "      <td>79</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Linguistics</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>97</td>\n",
       "      <td>90.0</td>\n",
       "      <td>94</td>\n",
       "      <td>82</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Average</td>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>82.0</td>\n",
       "      <td>94</td>\n",
       "      <td>94</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "5           no              1               Economics             Good   \n",
       "6          yes              1        Computer Science        Excellent   \n",
       "7           no              0  Mechanical Engineering             Poor   \n",
       "8           no              0             Linguistics          Average   \n",
       "9          yes              1             Mathematics          Average   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1  \n",
       "0                 3    92  93.0    84    91     92  \n",
       "1                 2    94  90.0    80    83     91  \n",
       "2                 3    78  85.0    83    80     80  \n",
       "3                 3    91   NaN    92    91     89  \n",
       "4                 4    77  83.0    90    92     85  \n",
       "5                 5    70  73.0    68    74     71  \n",
       "6                 4    80  88.0    89    88     91  \n",
       "7                 3    95  93.0    69    79     75  \n",
       "8                 2    97  90.0    94    82     80  \n",
       "9                 4    95  82.0    94    94     85  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- So we would like to apply more than one transformations on it: imputation and scaling.  \n",
    "- We can treat `lab2` separately, but we can also include it into `numeric_feats` and apply both transformations on all numeric columns."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "numeric_feats = [\n",
    "    \"university_years\",\n",
    "    \"lab1\",\n",
    "    \"lab2\",\n",
    "    \"lab3\",\n",
    "    \"lab4\",\n",
    "    \"quiz1\",\n",
    "]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = [\"class_attendance\", \"enjoy_course\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- To apply more than one transformations we can define a pipeline inside a column transformer to chain different transformations. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features    \n",
    "    (OneHotEncoder(), categorical_feats),  # OHE on categorical features\n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-4 {color: black;background-color: white;}#sk-container-id-4 pre{padding: 0;}#sk-container-id-4 div.sk-toggleable {background-color: white;}#sk-container-id-4 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-4 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-4 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-4 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-4 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-4 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-4 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-4 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-4 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-4 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-4 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-4 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-4 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-4 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-4 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-4 div.sk-item {position: relative;z-index: 1;}#sk-container-id-4 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-4 div.sk-item::before, #sk-container-id-4 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-4 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-4 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-4 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-4 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-4 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-4 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-4 div.sk-label-container {text-align: center;}#sk-container-id-4 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-4 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-4\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-30\" type=\"checkbox\" ><label for=\"sk-estimator-id-30\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-31\" type=\"checkbox\" ><label for=\"sk-estimator-id-31\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-32\" type=\"checkbox\" ><label for=\"sk-estimator-id-32\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-33\" type=\"checkbox\" ><label for=\"sk-estimator-id-33\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-34\" type=\"checkbox\" ><label for=\"sk-estimator-id-34\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-35\" type=\"checkbox\" ><label for=\"sk-estimator-id-35\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-36\" type=\"checkbox\" ><label for=\"sk-estimator-id-36\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-37\" type=\"checkbox\" ><label for=\"sk-estimator-id-37\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-38\" type=\"checkbox\" ><label for=\"sk-estimator-id-38\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;, &#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-39\" type=\"checkbox\" ><label for=\"sk-estimator-id-39\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('drop', 'drop',\n",
       "                                 ['class_attendance', 'enjoy_course'])])"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_transformed = ct.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['university_years',\n",
       " 'lab1',\n",
       " 'lab2',\n",
       " 'lab3',\n",
       " 'lab4',\n",
       " 'quiz1',\n",
       " 'ml_experience',\n",
       " 'major_Biology',\n",
       " 'major_Computer Science',\n",
       " 'major_Economics',\n",
       " 'major_Linguistics',\n",
       " 'major_Mathematics',\n",
       " 'major_Mechanical Engineering',\n",
       " 'major_Physics',\n",
       " 'major_Psychology']"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = (\n",
    "    numeric_feats\n",
    "    + passthrough_feats    \n",
    "    + ct.named_transformers_[\"onehotencoder\"].get_feature_names_out().tolist()\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major_Biology</th>\n",
       "      <th>major_Computer Science</th>\n",
       "      <th>major_Economics</th>\n",
       "      <th>major_Linguistics</th>\n",
       "      <th>major_Mathematics</th>\n",
       "      <th>major_Mechanical Engineering</th>\n",
       "      <th>major_Physics</th>\n",
       "      <th>major_Psychology</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-0.217334</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.590827</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>-0.614206</td>\n",
       "      <td>-0.855972</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-1.264480</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-0.316552</td>\n",
       "      <td>-1.312974</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>-1.103439</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-2.192133</td>\n",
       "      <td>-3.100139</td>\n",
       "      <td>-1.804821</td>\n",
       "      <td>-2.226978</td>\n",
       "      <td>-1.844409</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.032566</td>\n",
       "      <td>-0.105089</td>\n",
       "      <td>0.278755</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-1.705603</td>\n",
       "      <td>-1.465308</td>\n",
       "      <td>-1.333088</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.938697</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>-1.008306</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-1.303109</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>-0.504429</td>\n",
       "      <td>0.874062</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-0.949597</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-2.055985</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-0.105089</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>1.492270</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>1.581372</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>1.092930</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-1.972240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-1.904039</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-1.612349</td>\n",
       "      <td>0.493921</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-0.336826</td>\n",
       "      <td>0.094581</td>\n",
       "      <td>-2.102474</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>-0.438275</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>1.092930</td>\n",
       "      <td>0.080319</td>\n",
       "      <td>-1.160640</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.822740</td>\n",
       "      <td>0.693590</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>1.124370</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    university_years      lab1      lab2      lab3      lab4     quiz1  \\\n",
       "0          -0.093454  0.358913  0.893260 -0.217334  0.362700  0.840028   \n",
       "1          -1.074719  0.590827  0.294251 -0.614206 -0.855972  0.712198   \n",
       "2          -0.093454 -1.264480 -0.704099 -0.316552 -1.312974 -0.693936   \n",
       "3          -0.093454  0.242957  0.000000  0.576409  0.362700  0.456537   \n",
       "4           0.887812 -1.380436 -1.103439  0.377973  0.515034 -0.054784   \n",
       "5           1.869077 -2.192133 -3.100139 -1.804821 -2.226978 -1.844409   \n",
       "6           0.887812 -1.032566 -0.105089  0.278755 -0.094302  0.712198   \n",
       "7          -0.093454  0.706783  0.893260 -1.705603 -1.465308 -1.333088   \n",
       "8          -1.074719  0.938697  0.294251  0.774844 -1.008306 -0.693936   \n",
       "9           0.887812  0.706783 -1.303109  0.774844  0.819702 -0.054784   \n",
       "10         -0.093454  1.054653 -0.504429  0.874062  0.972036 -0.949597   \n",
       "11         -2.055985  0.706783 -0.105089  0.675627  0.515034 -0.054784   \n",
       "12         -1.074719  1.054653  1.492270  0.973280  1.581372  1.862671   \n",
       "13          0.887812  0.706783  1.092930  0.973280  0.972036  1.862671   \n",
       "14         -0.093454  0.706783  0.294251  0.675627  0.972036 -1.972240   \n",
       "15         -0.093454  0.358913 -0.704099 -1.904039  0.819702  0.840028   \n",
       "16          1.869077 -1.612349  0.493921  0.675627 -0.398970 -0.054784   \n",
       "17         -0.093454 -0.336826  0.094581 -2.102474 -0.398970  0.200876   \n",
       "18         -1.074719  0.242957  0.000000  0.377973 -0.094302 -0.438275   \n",
       "19         -1.074719 -1.380436  1.092930  0.080319 -1.160640  0.456537   \n",
       "20          0.887812  0.822740  0.693590  0.576409  1.124370  0.200876   \n",
       "\n",
       "    ml_experience  major_Biology  major_Computer Science  major_Economics  \\\n",
       "0             1.0            0.0                     1.0              0.0   \n",
       "1             1.0            0.0                     0.0              0.0   \n",
       "2             0.0            0.0                     0.0              0.0   \n",
       "3             0.0            0.0                     0.0              0.0   \n",
       "4             0.0            0.0                     0.0              0.0   \n",
       "5             1.0            0.0                     0.0              1.0   \n",
       "6             1.0            0.0                     1.0              0.0   \n",
       "7             0.0            0.0                     0.0              0.0   \n",
       "8             0.0            0.0                     0.0              0.0   \n",
       "9             1.0            0.0                     0.0              0.0   \n",
       "10            0.0            0.0                     0.0              0.0   \n",
       "11            1.0            0.0                     0.0              0.0   \n",
       "12            1.0            0.0                     0.0              0.0   \n",
       "13            0.0            0.0                     0.0              0.0   \n",
       "14            0.0            0.0                     0.0              0.0   \n",
       "15            1.0            0.0                     1.0              0.0   \n",
       "16            0.0            0.0                     1.0              0.0   \n",
       "17            1.0            0.0                     0.0              1.0   \n",
       "18            1.0            1.0                     0.0              0.0   \n",
       "19            0.0            0.0                     0.0              0.0   \n",
       "20            1.0            0.0                     0.0              0.0   \n",
       "\n",
       "    major_Linguistics  major_Mathematics  major_Mechanical Engineering  \\\n",
       "0                 0.0                0.0                           0.0   \n",
       "1                 0.0                0.0                           1.0   \n",
       "2                 0.0                1.0                           0.0   \n",
       "3                 0.0                1.0                           0.0   \n",
       "4                 0.0                0.0                           0.0   \n",
       "5                 0.0                0.0                           0.0   \n",
       "6                 0.0                0.0                           0.0   \n",
       "7                 0.0                0.0                           1.0   \n",
       "8                 1.0                0.0                           0.0   \n",
       "9                 0.0                1.0                           0.0   \n",
       "10                0.0                0.0                           0.0   \n",
       "11                0.0                0.0                           0.0   \n",
       "12                0.0                0.0                           0.0   \n",
       "13                0.0                0.0                           1.0   \n",
       "14                0.0                1.0                           0.0   \n",
       "15                0.0                0.0                           0.0   \n",
       "16                0.0                0.0                           0.0   \n",
       "17                0.0                0.0                           0.0   \n",
       "18                0.0                0.0                           0.0   \n",
       "19                0.0                0.0                           0.0   \n",
       "20                1.0                0.0                           0.0   \n",
       "\n",
       "    major_Physics  major_Psychology  \n",
       "0             0.0               0.0  \n",
       "1             0.0               0.0  \n",
       "2             0.0               0.0  \n",
       "3             0.0               0.0  \n",
       "4             0.0               1.0  \n",
       "5             0.0               0.0  \n",
       "6             0.0               0.0  \n",
       "7             0.0               0.0  \n",
       "8             0.0               0.0  \n",
       "9             0.0               0.0  \n",
       "10            0.0               1.0  \n",
       "11            1.0               0.0  \n",
       "12            1.0               0.0  \n",
       "13            0.0               0.0  \n",
       "14            0.0               0.0  \n",
       "15            0.0               0.0  \n",
       "16            0.0               0.0  \n",
       "17            0.0               0.0  \n",
       "18            0.0               0.0  \n",
       "19            0.0               1.0  \n",
       "20            0.0               0.0  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X_transformed, columns=column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Incorporating ordinal feature `class_attendance` "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The `class_attendance` column is different than the `major` column in that there is some ordering of the values. \n",
    "    - Excellent > Good > Average > poor    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0          yes              1        Computer Science        Excellent   \n",
       "1          yes              1  Mechanical Engineering          Average   \n",
       "2          yes              0             Mathematics             Poor   \n",
       "3           no              0             Mathematics        Excellent   \n",
       "4          yes              0              Psychology             Good   \n",
       "\n",
       "   university_years  lab1  lab2  lab3  lab4  quiz1  \n",
       "0                 3    92  93.0    84    91     92  \n",
       "1                 2    94  90.0    80    83     91  \n",
       "2                 3    78  85.0    83    80     80  \n",
       "3                 3    91   NaN    92    91     89  \n",
       "4                 4    77  83.0    90    92     85  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's try applying `OrdinalEncoder` on this column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_toy = X[[\"class_attendance\"]]\n",
    "enc = OrdinalEncoder()\n",
    "enc.fit(X_toy)\n",
    "X_toy_ord = enc.transform(X_toy)\n",
    "df = pd.DataFrame(\n",
    "    data=X_toy_ord,\n",
    "    columns=[\"class_attendance_enc\"],\n",
    "    index=X_toy.index,\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>class_attendance_enc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Poor</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Good</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Good</td>\n",
       "      <td>2.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Poor</td>\n",
       "      <td>3.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Average</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  class_attendance  class_attendance_enc\n",
       "0        Excellent                   1.0\n",
       "1          Average                   0.0\n",
       "2             Poor                   3.0\n",
       "3        Excellent                   1.0\n",
       "4             Good                   2.0\n",
       "5             Good                   2.0\n",
       "6        Excellent                   1.0\n",
       "7             Poor                   3.0\n",
       "8          Average                   0.0\n",
       "9          Average                   0.0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.concat([X_toy, df], axis=1).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- What's the problem here? \n",
    "    - The encoder doesn't know the order. \n",
    "- We can examine unique categories manually, order them based on our intuitions, and then provide this human knowledge to the transformer. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "What are the unique categories of `class_attendance`? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['Excellent', 'Average', 'Poor', 'Good'], dtype=object)"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_toy[\"class_attendance\"].unique()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's order them manually. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "class_attendance_levels = [\"Poor\", \"Average\", \"Good\", \"Excellent\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{note}\n",
    "Note that if you use the reverse order of the categories, it wouldn't matter. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's make sure that we have included all categories in our manual ordering.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "assert set(class_attendance_levels) == set(X_toy[\"class_attendance\"].unique())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[array(['Poor', 'Average', 'Good', 'Excellent'], dtype=object)]\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>class_attendance_enc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Poor</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Good</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Good</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Poor</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  class_attendance  class_attendance_enc\n",
       "0        Excellent                     3\n",
       "1          Average                     1\n",
       "2             Poor                     0\n",
       "3        Excellent                     3\n",
       "4             Good                     2\n",
       "5             Good                     2\n",
       "6        Excellent                     3\n",
       "7             Poor                     0\n",
       "8          Average                     1\n",
       "9          Average                     1"
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "oe = OrdinalEncoder(categories=[class_attendance_levels], dtype=int)\n",
    "oe.fit(X_toy[[\"class_attendance\"]])\n",
    "ca_transformed = oe.transform(X_toy[[\"class_attendance\"]])\n",
    "df = pd.DataFrame(\n",
    "    data=ca_transformed, columns=[\"class_attendance_enc\"], index=X_toy.index\n",
    ")\n",
    "print(oe.categories_)\n",
    "pd.concat([X_toy, df], axis=1).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The encoded categories are looking better now! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### More than one ordinal columns?\n",
    "\n",
    "- We can pass the manually ordered categories when we create an `OrdinalEncoder` object as a list of lists. \n",
    "- If you have more than one ordinal columns\n",
    "    - manually create a list of ordered categories for each column\n",
    "    - pass a list of lists to `OrdinalEncoder`, where each inner list corresponds to manually created list of ordered categories for a corresponding ordinal column. \n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Now let's incorporate ordinal encoding of `class_attendance` in our column transformer. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>93.0</td>\n",
       "      <td>84</td>\n",
       "      <td>91</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>94</td>\n",
       "      <td>90.0</td>\n",
       "      <td>80</td>\n",
       "      <td>83</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>78</td>\n",
       "      <td>85.0</td>\n",
       "      <td>83</td>\n",
       "      <td>80</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>3</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>92</td>\n",
       "      <td>91</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>4</td>\n",
       "      <td>77</td>\n",
       "      <td>83.0</td>\n",
       "      <td>90</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Economics</td>\n",
       "      <td>Good</td>\n",
       "      <td>5</td>\n",
       "      <td>70</td>\n",
       "      <td>73.0</td>\n",
       "      <td>68</td>\n",
       "      <td>74</td>\n",
       "      <td>71</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>80</td>\n",
       "      <td>88.0</td>\n",
       "      <td>89</td>\n",
       "      <td>88</td>\n",
       "      <td>91</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>95</td>\n",
       "      <td>93.0</td>\n",
       "      <td>69</td>\n",
       "      <td>79</td>\n",
       "      <td>75</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Linguistics</td>\n",
       "      <td>Average</td>\n",
       "      <td>2</td>\n",
       "      <td>97</td>\n",
       "      <td>90.0</td>\n",
       "      <td>94</td>\n",
       "      <td>82</td>\n",
       "      <td>80</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Average</td>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>82.0</td>\n",
       "      <td>94</td>\n",
       "      <td>94</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Good</td>\n",
       "      <td>3</td>\n",
       "      <td>98</td>\n",
       "      <td>86.0</td>\n",
       "      <td>95</td>\n",
       "      <td>95</td>\n",
       "      <td>78</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Physics</td>\n",
       "      <td>Average</td>\n",
       "      <td>1</td>\n",
       "      <td>95</td>\n",
       "      <td>88.0</td>\n",
       "      <td>93</td>\n",
       "      <td>92</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Physics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>2</td>\n",
       "      <td>98</td>\n",
       "      <td>96.0</td>\n",
       "      <td>96</td>\n",
       "      <td>99</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Mechanical Engineering</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>95</td>\n",
       "      <td>94.0</td>\n",
       "      <td>96</td>\n",
       "      <td>95</td>\n",
       "      <td>100</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Mathematics</td>\n",
       "      <td>Poor</td>\n",
       "      <td>3</td>\n",
       "      <td>95</td>\n",
       "      <td>90.0</td>\n",
       "      <td>93</td>\n",
       "      <td>95</td>\n",
       "      <td>70</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Good</td>\n",
       "      <td>3</td>\n",
       "      <td>92</td>\n",
       "      <td>85.0</td>\n",
       "      <td>67</td>\n",
       "      <td>94</td>\n",
       "      <td>92</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>yes</td>\n",
       "      <td>0</td>\n",
       "      <td>Computer Science</td>\n",
       "      <td>Average</td>\n",
       "      <td>5</td>\n",
       "      <td>75</td>\n",
       "      <td>91.0</td>\n",
       "      <td>93</td>\n",
       "      <td>86</td>\n",
       "      <td>85</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Economics</td>\n",
       "      <td>Average</td>\n",
       "      <td>3</td>\n",
       "      <td>86</td>\n",
       "      <td>89.0</td>\n",
       "      <td>65</td>\n",
       "      <td>86</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>no</td>\n",
       "      <td>1</td>\n",
       "      <td>Biology</td>\n",
       "      <td>Good</td>\n",
       "      <td>2</td>\n",
       "      <td>91</td>\n",
       "      <td>NaN</td>\n",
       "      <td>90</td>\n",
       "      <td>88</td>\n",
       "      <td>82</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "      <td>Psychology</td>\n",
       "      <td>Poor</td>\n",
       "      <td>2</td>\n",
       "      <td>77</td>\n",
       "      <td>94.0</td>\n",
       "      <td>87</td>\n",
       "      <td>81</td>\n",
       "      <td>89</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "      <td>Linguistics</td>\n",
       "      <td>Excellent</td>\n",
       "      <td>4</td>\n",
       "      <td>96</td>\n",
       "      <td>92.0</td>\n",
       "      <td>92</td>\n",
       "      <td>96</td>\n",
       "      <td>87</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   enjoy_course  ml_experience                   major class_attendance  \\\n",
       "0           yes              1        Computer Science        Excellent   \n",
       "1           yes              1  Mechanical Engineering          Average   \n",
       "2           yes              0             Mathematics             Poor   \n",
       "3            no              0             Mathematics        Excellent   \n",
       "4           yes              0              Psychology             Good   \n",
       "5            no              1               Economics             Good   \n",
       "6           yes              1        Computer Science        Excellent   \n",
       "7            no              0  Mechanical Engineering             Poor   \n",
       "8            no              0             Linguistics          Average   \n",
       "9           yes              1             Mathematics          Average   \n",
       "10          yes              0              Psychology             Good   \n",
       "11          yes              1                 Physics          Average   \n",
       "12          yes              1                 Physics        Excellent   \n",
       "13          yes              0  Mechanical Engineering        Excellent   \n",
       "14           no              0             Mathematics             Poor   \n",
       "15           no              1        Computer Science             Good   \n",
       "16          yes              0        Computer Science          Average   \n",
       "17          yes              1               Economics          Average   \n",
       "18           no              1                 Biology             Good   \n",
       "19           no              0              Psychology             Poor   \n",
       "20          yes              1             Linguistics        Excellent   \n",
       "\n",
       "    university_years  lab1  lab2  lab3  lab4  quiz1  \n",
       "0                  3    92  93.0    84    91     92  \n",
       "1                  2    94  90.0    80    83     91  \n",
       "2                  3    78  85.0    83    80     80  \n",
       "3                  3    91   NaN    92    91     89  \n",
       "4                  4    77  83.0    90    92     85  \n",
       "5                  5    70  73.0    68    74     71  \n",
       "6                  4    80  88.0    89    88     91  \n",
       "7                  3    95  93.0    69    79     75  \n",
       "8                  2    97  90.0    94    82     80  \n",
       "9                  4    95  82.0    94    94     85  \n",
       "10                 3    98  86.0    95    95     78  \n",
       "11                 1    95  88.0    93    92     85  \n",
       "12                 2    98  96.0    96    99    100  \n",
       "13                 4    95  94.0    96    95    100  \n",
       "14                 3    95  90.0    93    95     70  \n",
       "15                 3    92  85.0    67    94     92  \n",
       "16                 5    75  91.0    93    86     85  \n",
       "17                 3    86  89.0    65    86     87  \n",
       "18                 2    91   NaN    90    88     82  \n",
       "19                 2    77  94.0    87    81     89  \n",
       "20                 4    96  92.0    92    96     87  "
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_feats = [\n",
    "    \"university_years\",\n",
    "    \"lab1\",\n",
    "    \"lab2\",\n",
    "    \"lab3\",\n",
    "    \"lab4\",\n",
    "    \"quiz1\",\n",
    "]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "ordinal_feats = [\"class_attendance\"]  # apply ordinal encoding\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = [\"enjoy_course\"]  # do not include these features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (\n",
    "        OrdinalEncoder(categories=[class_attendance_levels], dtype=int),\n",
    "        ordinal_feats,\n",
    "    ),  # Ordinal encoding on ordinal features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    "    (OneHotEncoder(), categorical_feats),  # OHE on categorical features    \n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-5 {color: black;background-color: white;}#sk-container-id-5 pre{padding: 0;}#sk-container-id-5 div.sk-toggleable {background-color: white;}#sk-container-id-5 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-5 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-5 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-5 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-5 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-5 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-5 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-5 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-5 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-5 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-5 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-5 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-5 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-5 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-5 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-5 div.sk-item {position: relative;z-index: 1;}#sk-container-id-5 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-5 div.sk-item::before, #sk-container-id-5 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-5 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-5 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-5 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-5 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-5 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-5 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-5 div.sk-label-container {text-align: center;}#sk-container-id-5 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-5 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-5\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-40\" type=\"checkbox\" ><label for=\"sk-estimator-id-40\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;, OneHotEncoder(), [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-41\" type=\"checkbox\" ><label for=\"sk-estimator-id-41\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-42\" type=\"checkbox\" ><label for=\"sk-estimator-id-42\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-43\" type=\"checkbox\" ><label for=\"sk-estimator-id-43\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-44\" type=\"checkbox\" ><label for=\"sk-estimator-id-44\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ordinalencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-45\" type=\"checkbox\" ><label for=\"sk-estimator-id-45\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;, &#x27;Good&#x27;, &#x27;Excellent&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-46\" type=\"checkbox\" ><label for=\"sk-estimator-id-46\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-47\" type=\"checkbox\" ><label for=\"sk-estimator-id-47\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-48\" type=\"checkbox\" ><label for=\"sk-estimator-id-48\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-49\" type=\"checkbox\" ><label for=\"sk-estimator-id-49\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-50\" type=\"checkbox\" ><label for=\"sk-estimator-id-50\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-51\" type=\"checkbox\" ><label for=\"sk-estimator-id-51\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('ordinalencoder',\n",
       "                                 OrdinalEncoder(categories=[['Poor', 'Average',\n",
       "                                                             'Good',\n",
       "                                                             'Excellent']],\n",
       "                                                dtype=<class 'int'>),\n",
       "                                 ['class_attendance']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('onehotencoder', OneHotEncoder(), ['major']),\n",
       "                                ('drop', 'drop', ['enjoy_course'])])"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "X_transformed = ct.fit_transform(X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['university_years',\n",
       " 'lab1',\n",
       " 'lab2',\n",
       " 'lab3',\n",
       " 'lab4',\n",
       " 'quiz1',\n",
       " 'class_attendance',\n",
       " 'ml_experience',\n",
       " 'major_Biology',\n",
       " 'major_Computer Science',\n",
       " 'major_Economics',\n",
       " 'major_Linguistics',\n",
       " 'major_Mathematics',\n",
       " 'major_Mechanical Engineering',\n",
       " 'major_Physics',\n",
       " 'major_Psychology']"
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = (\n",
    "    numeric_feats\n",
    "    + ordinal_feats\n",
    "    + passthrough_feats    \n",
    "    + ct.named_transformers_[\"onehotencoder\"].get_feature_names_out().tolist()\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>university_years</th>\n",
       "      <th>lab1</th>\n",
       "      <th>lab2</th>\n",
       "      <th>lab3</th>\n",
       "      <th>lab4</th>\n",
       "      <th>quiz1</th>\n",
       "      <th>class_attendance</th>\n",
       "      <th>ml_experience</th>\n",
       "      <th>major_Biology</th>\n",
       "      <th>major_Computer Science</th>\n",
       "      <th>major_Economics</th>\n",
       "      <th>major_Linguistics</th>\n",
       "      <th>major_Mathematics</th>\n",
       "      <th>major_Mechanical Engineering</th>\n",
       "      <th>major_Physics</th>\n",
       "      <th>major_Psychology</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-0.217334</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.590827</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>-0.614206</td>\n",
       "      <td>-0.855972</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-1.264480</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-0.316552</td>\n",
       "      <td>-1.312974</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>0.362700</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>-1.103439</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-2.192133</td>\n",
       "      <td>-3.100139</td>\n",
       "      <td>-1.804821</td>\n",
       "      <td>-2.226978</td>\n",
       "      <td>-1.844409</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>-1.032566</td>\n",
       "      <td>-0.105089</td>\n",
       "      <td>0.278755</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>0.712198</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.893260</td>\n",
       "      <td>-1.705603</td>\n",
       "      <td>-1.465308</td>\n",
       "      <td>-1.333088</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.938697</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>-1.008306</td>\n",
       "      <td>-0.693936</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-1.303109</td>\n",
       "      <td>0.774844</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>-0.504429</td>\n",
       "      <td>0.874062</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-0.949597</td>\n",
       "      <td>2.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>-2.055985</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>-0.105089</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.515034</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>1.054653</td>\n",
       "      <td>1.492270</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>1.581372</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>1.092930</td>\n",
       "      <td>0.973280</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>1.862671</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.706783</td>\n",
       "      <td>0.294251</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>0.972036</td>\n",
       "      <td>-1.972240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>15</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>0.358913</td>\n",
       "      <td>-0.704099</td>\n",
       "      <td>-1.904039</td>\n",
       "      <td>0.819702</td>\n",
       "      <td>0.840028</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>16</th>\n",
       "      <td>1.869077</td>\n",
       "      <td>-1.612349</td>\n",
       "      <td>0.493921</td>\n",
       "      <td>0.675627</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>-0.054784</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>17</th>\n",
       "      <td>-0.093454</td>\n",
       "      <td>-0.336826</td>\n",
       "      <td>0.094581</td>\n",
       "      <td>-2.102474</td>\n",
       "      <td>-0.398970</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>0.242957</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>0.377973</td>\n",
       "      <td>-0.094302</td>\n",
       "      <td>-0.438275</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>19</th>\n",
       "      <td>-1.074719</td>\n",
       "      <td>-1.380436</td>\n",
       "      <td>1.092930</td>\n",
       "      <td>0.080319</td>\n",
       "      <td>-1.160640</td>\n",
       "      <td>0.456537</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20</th>\n",
       "      <td>0.887812</td>\n",
       "      <td>0.822740</td>\n",
       "      <td>0.693590</td>\n",
       "      <td>0.576409</td>\n",
       "      <td>1.124370</td>\n",
       "      <td>0.200876</td>\n",
       "      <td>3.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    university_years      lab1      lab2      lab3      lab4     quiz1  \\\n",
       "0          -0.093454  0.358913  0.893260 -0.217334  0.362700  0.840028   \n",
       "1          -1.074719  0.590827  0.294251 -0.614206 -0.855972  0.712198   \n",
       "2          -0.093454 -1.264480 -0.704099 -0.316552 -1.312974 -0.693936   \n",
       "3          -0.093454  0.242957  0.000000  0.576409  0.362700  0.456537   \n",
       "4           0.887812 -1.380436 -1.103439  0.377973  0.515034 -0.054784   \n",
       "5           1.869077 -2.192133 -3.100139 -1.804821 -2.226978 -1.844409   \n",
       "6           0.887812 -1.032566 -0.105089  0.278755 -0.094302  0.712198   \n",
       "7          -0.093454  0.706783  0.893260 -1.705603 -1.465308 -1.333088   \n",
       "8          -1.074719  0.938697  0.294251  0.774844 -1.008306 -0.693936   \n",
       "9           0.887812  0.706783 -1.303109  0.774844  0.819702 -0.054784   \n",
       "10         -0.093454  1.054653 -0.504429  0.874062  0.972036 -0.949597   \n",
       "11         -2.055985  0.706783 -0.105089  0.675627  0.515034 -0.054784   \n",
       "12         -1.074719  1.054653  1.492270  0.973280  1.581372  1.862671   \n",
       "13          0.887812  0.706783  1.092930  0.973280  0.972036  1.862671   \n",
       "14         -0.093454  0.706783  0.294251  0.675627  0.972036 -1.972240   \n",
       "15         -0.093454  0.358913 -0.704099 -1.904039  0.819702  0.840028   \n",
       "16          1.869077 -1.612349  0.493921  0.675627 -0.398970 -0.054784   \n",
       "17         -0.093454 -0.336826  0.094581 -2.102474 -0.398970  0.200876   \n",
       "18         -1.074719  0.242957  0.000000  0.377973 -0.094302 -0.438275   \n",
       "19         -1.074719 -1.380436  1.092930  0.080319 -1.160640  0.456537   \n",
       "20          0.887812  0.822740  0.693590  0.576409  1.124370  0.200876   \n",
       "\n",
       "    class_attendance  ml_experience  major_Biology  major_Computer Science  \\\n",
       "0                3.0            1.0            0.0                     1.0   \n",
       "1                1.0            1.0            0.0                     0.0   \n",
       "2                0.0            0.0            0.0                     0.0   \n",
       "3                3.0            0.0            0.0                     0.0   \n",
       "4                2.0            0.0            0.0                     0.0   \n",
       "5                2.0            1.0            0.0                     0.0   \n",
       "6                3.0            1.0            0.0                     1.0   \n",
       "7                0.0            0.0            0.0                     0.0   \n",
       "8                1.0            0.0            0.0                     0.0   \n",
       "9                1.0            1.0            0.0                     0.0   \n",
       "10               2.0            0.0            0.0                     0.0   \n",
       "11               1.0            1.0            0.0                     0.0   \n",
       "12               3.0            1.0            0.0                     0.0   \n",
       "13               3.0            0.0            0.0                     0.0   \n",
       "14               0.0            0.0            0.0                     0.0   \n",
       "15               2.0            1.0            0.0                     1.0   \n",
       "16               1.0            0.0            0.0                     1.0   \n",
       "17               1.0            1.0            0.0                     0.0   \n",
       "18               2.0            1.0            1.0                     0.0   \n",
       "19               0.0            0.0            0.0                     0.0   \n",
       "20               3.0            1.0            0.0                     0.0   \n",
       "\n",
       "    major_Economics  major_Linguistics  major_Mathematics  \\\n",
       "0               0.0                0.0                0.0   \n",
       "1               0.0                0.0                0.0   \n",
       "2               0.0                0.0                1.0   \n",
       "3               0.0                0.0                1.0   \n",
       "4               0.0                0.0                0.0   \n",
       "5               1.0                0.0                0.0   \n",
       "6               0.0                0.0                0.0   \n",
       "7               0.0                0.0                0.0   \n",
       "8               0.0                1.0                0.0   \n",
       "9               0.0                0.0                1.0   \n",
       "10              0.0                0.0                0.0   \n",
       "11              0.0                0.0                0.0   \n",
       "12              0.0                0.0                0.0   \n",
       "13              0.0                0.0                0.0   \n",
       "14              0.0                0.0                1.0   \n",
       "15              0.0                0.0                0.0   \n",
       "16              0.0                0.0                0.0   \n",
       "17              1.0                0.0                0.0   \n",
       "18              0.0                0.0                0.0   \n",
       "19              0.0                0.0                0.0   \n",
       "20              0.0                1.0                0.0   \n",
       "\n",
       "    major_Mechanical Engineering  major_Physics  major_Psychology  \n",
       "0                            0.0            0.0               0.0  \n",
       "1                            1.0            0.0               0.0  \n",
       "2                            0.0            0.0               0.0  \n",
       "3                            0.0            0.0               0.0  \n",
       "4                            0.0            0.0               1.0  \n",
       "5                            0.0            0.0               0.0  \n",
       "6                            0.0            0.0               0.0  \n",
       "7                            1.0            0.0               0.0  \n",
       "8                            0.0            0.0               0.0  \n",
       "9                            0.0            0.0               0.0  \n",
       "10                           0.0            0.0               1.0  \n",
       "11                           0.0            1.0               0.0  \n",
       "12                           0.0            1.0               0.0  \n",
       "13                           1.0            0.0               0.0  \n",
       "14                           0.0            0.0               0.0  \n",
       "15                           0.0            0.0               0.0  \n",
       "16                           0.0            0.0               0.0  \n",
       "17                           0.0            0.0               0.0  \n",
       "18                           0.0            0.0               0.0  \n",
       "19                           0.0            0.0               1.0  \n",
       "20                           0.0            0.0               0.0  "
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X_transformed, columns=column_names)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Dealing with unknown categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Let's create a pipeline with the column transformer and pass it to `cross_validate`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = make_pipeline(ct, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/model_selection/_validation.py:776: UserWarning: Scoring failed. The score on this train-test partition for these parameters will be set to nan. Details: \n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/model_selection/_validation.py\", line 767, in _score\n",
      "    scores = scorer(estimator, X_test, y_test)\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/metrics/_scorer.py\", line 429, in _passthrough_scorer\n",
      "    return estimator.score(*args, **kwargs)\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/pipeline.py\", line 695, in score\n",
      "    Xt = transform.transform(Xt)\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/compose/_column_transformer.py\", line 763, in transform\n",
      "    Xs = self._fit_transform(\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/compose/_column_transformer.py\", line 621, in _fit_transform\n",
      "    return Parallel(n_jobs=self.n_jobs)(\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/parallel.py\", line 1046, in __call__\n",
      "    while self.dispatch_one_batch(iterator):\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/parallel.py\", line 861, in dispatch_one_batch\n",
      "    self._dispatch(tasks)\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/parallel.py\", line 779, in _dispatch\n",
      "    job = self._backend.apply_async(batch, callback=cb)\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/_parallel_backends.py\", line 208, in apply_async\n",
      "    result = ImmediateResult(func)\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/_parallel_backends.py\", line 572, in __init__\n",
      "    self.results = batch()\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/parallel.py\", line 262, in __call__\n",
      "    return [func(*args, **kwargs)\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/joblib/parallel.py\", line 262, in <listcomp>\n",
      "    return [func(*args, **kwargs)\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/fixes.py\", line 117, in __call__\n",
      "    return self.function(*args, **kwargs)\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/pipeline.py\", line 853, in _transform_one\n",
      "    res = transformer.transform(X)\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/preprocessing/_encoders.py\", line 882, in transform\n",
      "    X_int, X_mask = self._transform(\n",
      "  File \"/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/preprocessing/_encoders.py\", line 160, in _transform\n",
      "    raise ValueError(msg)\n",
      "ValueError: Found unknown categories ['Biology'] in column 0 during transform\n",
      "\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.003883</td>\n",
       "      <td>0.001701</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.937500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.003105</td>\n",
       "      <td>0.001825</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.003069</td>\n",
       "      <td>0.001632</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.002812</td>\n",
       "      <td>0.001657</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.002722</td>\n",
       "      <td>0.005762</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.003883    0.001701        1.00     0.937500\n",
       "1  0.003105    0.001825        1.00     0.941176\n",
       "2  0.003069    0.001632        0.50     1.000000\n",
       "3  0.002812    0.001657        0.75     0.941176\n",
       "4  0.002722    0.005762         NaN     1.000000"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_validate(pipe, X, y, return_train_score=True)\n",
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- What's going on here??\n",
    "- Let's look at the error message:\n",
    "`ValueError: Found unknown categories ['Biology'] in column 0 during transform\n",
    "`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Computer Science          4\n",
       "Mathematics               4\n",
       "Mechanical Engineering    3\n",
       "Psychology                3\n",
       "Economics                 2\n",
       "Linguistics               2\n",
       "Physics                   2\n",
       "Biology                   1\n",
       "Name: major, dtype: int64"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[\"major\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- There is only one instance of Biology.\n",
    "- During cross-validation, this is getting put into the validation split.\n",
    "- By default, `OneHotEncoder` throws an error because you might want to know about this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Simplest fix:\n",
    "- Pass `handle_unknown=\"ignore\"` argument to `OneHotEncoder`\n",
    "- It creates a row with all zeros. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (\n",
    "        OrdinalEncoder(categories=[class_attendance_levels], dtype=int),\n",
    "        ordinal_feats,\n",
    "    ),  # Ordinal encoding on ordinal features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features\n",
    "    (\n",
    "        OneHotEncoder(handle_unknown=\"ignore\"),\n",
    "        categorical_feats,\n",
    "    ),  # OHE on categorical features    \n",
    "    (\"drop\", drop_feats),  # drop the drop features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-6 {color: black;background-color: white;}#sk-container-id-6 pre{padding: 0;}#sk-container-id-6 div.sk-toggleable {background-color: white;}#sk-container-id-6 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-6 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-6 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-6 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-6 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-6 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-6 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-6 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-6 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-6 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-6 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-6 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-6 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-6 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-6 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-6 div.sk-item {position: relative;z-index: 1;}#sk-container-id-6 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-6 div.sk-item::before, #sk-container-id-6 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-6 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-6 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-6 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-6 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-6 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-6 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-6 div.sk-label-container {text-align: center;}#sk-container-id-6 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-6 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-6\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-52\" type=\"checkbox\" ><label for=\"sk-estimator-id-52\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;]),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;, [&#x27;enjoy_course&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-53\" type=\"checkbox\" ><label for=\"sk-estimator-id-53\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-54\" type=\"checkbox\" ><label for=\"sk-estimator-id-54\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-55\" type=\"checkbox\" ><label for=\"sk-estimator-id-55\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-56\" type=\"checkbox\" ><label for=\"sk-estimator-id-56\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ordinalencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-57\" type=\"checkbox\" ><label for=\"sk-estimator-id-57\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;, &#x27;Good&#x27;, &#x27;Excellent&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-58\" type=\"checkbox\" ><label for=\"sk-estimator-id-58\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-59\" type=\"checkbox\" ><label for=\"sk-estimator-id-59\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-60\" type=\"checkbox\" ><label for=\"sk-estimator-id-60\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-61\" type=\"checkbox\" ><label for=\"sk-estimator-id-61\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-62\" type=\"checkbox\" ><label for=\"sk-estimator-id-62\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-63\" type=\"checkbox\" ><label for=\"sk-estimator-id-63\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('ordinalencoder',\n",
       "                                 OrdinalEncoder(categories=[['Poor', 'Average',\n",
       "                                                             'Good',\n",
       "                                                             'Excellent']],\n",
       "                                                dtype=<class 'int'>),\n",
       "                                 ['class_attendance']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('onehotencoder',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['major']),\n",
       "                                ('drop', 'drop', ['enjoy_course'])])"
      ]
     },
     "execution_count": 48,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe = make_pipeline(ct, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.003694</td>\n",
       "      <td>0.001822</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.937500</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.002776</td>\n",
       "      <td>0.001555</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.002656</td>\n",
       "      <td>0.001553</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.002709</td>\n",
       "      <td>0.001515</td>\n",
       "      <td>0.75</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.002813</td>\n",
       "      <td>0.001668</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.003694    0.001822        1.00     0.937500\n",
       "1  0.002776    0.001555        1.00     0.941176\n",
       "2  0.002656    0.001553        0.50     1.000000\n",
       "3  0.002709    0.001515        0.75     0.941176\n",
       "4  0.002813    0.001668        0.75     1.000000"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_validate(pipe, X, y, cv=5, return_train_score=True)\n",
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- With this approach, all unknown categories will be represented with all zeros and cross-validation is running OK now. \n",
    "\n",
    "Ask yourself the following questions when you work with categorical variables   \n",
    "- Do you want this behaviour? \n",
    "- Are you expecting to get many unknown categories? Do you want to be able to distinguish between them?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Cases where it's OK to break the golden rule \n",
    "\n",
    "- If it's some fix number of categories. For example, if it's something like provinces in Canada or majors taught at UBC. We know the categories in advance and this is one of the cases where it might be OK to violate the golden rule and get a list of all possible values for the categorical variable. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Categorical features with only two possible categories\n",
    "\n",
    "- Sometimes you have features with only two possible categories. \n",
    "- If we apply `OheHotEncoder` on such columns, it'll create two columns, which seems wasteful, as we could represent all information in the column in just one column with say 0's and 1's with presence of absence of one of the categories.\n",
    "- You can pass `drop=\"if_binary\"` argument to `OneHotEncoder` in order to create only one column in such scenario. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    yes\n",
       "1    yes\n",
       "2    yes\n",
       "3     no\n",
       "4    yes\n",
       "Name: enjoy_course, dtype: object"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X[\"enjoy_course\"].head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>enjoy_course</th>\n",
       "      <th>enjoy_course_enc</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>no</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>yes</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  enjoy_course  enjoy_course_enc\n",
       "0          yes                 1\n",
       "1          yes                 1\n",
       "2          yes                 1\n",
       "3           no                 0\n",
       "4          yes                 1\n",
       "5           no                 0\n",
       "6          yes                 1\n",
       "7           no                 0\n",
       "8           no                 0\n",
       "9          yes                 1"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ohe_enc = OneHotEncoder(drop=\"if_binary\", dtype=int, sparse=False)\n",
    "ohe_enc.fit(X[[\"enjoy_course\"]])\n",
    "transformed = ohe_enc.transform(X[[\"enjoy_course\"]])\n",
    "df = pd.DataFrame(data=transformed, columns=[\"enjoy_course_enc\"], index=X.index)\n",
    "pd.concat([X[[\"enjoy_course\"]], df], axis=1).head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "numeric_feats = [\n",
    "    \"university_years\",\n",
    "    \"lab1\",\n",
    "    \"lab2\",\n",
    "    \"lab3\",\n",
    "    \"lab4\",\n",
    "    \"quiz1\",\n",
    "]  # apply scaling\n",
    "categorical_feats = [\"major\"]  # apply one-hot encoding\n",
    "ordinal_feats = [\"class_attendance\"]  # apply ordinal encoding\n",
    "binary_feats = [\"enjoy_course\"]  # apply one-hot encoding with drop=\"if_binary\"\n",
    "passthrough_feats = [\"ml_experience\"]  # do not apply any transformation\n",
    "drop_feats = []"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "ct = make_column_transformer(\n",
    "    (\n",
    "        make_pipeline(SimpleImputer(), StandardScaler()),\n",
    "        numeric_feats,\n",
    "    ),  # scaling on numeric features\n",
    "    (\n",
    "        OrdinalEncoder(categories=[class_attendance_levels], dtype=int),\n",
    "        ordinal_feats,\n",
    "    ),  # Ordinal encoding on ordinal features\n",
    "    (\n",
    "        OneHotEncoder(drop=\"if_binary\", dtype=int),\n",
    "        binary_feats,\n",
    "    ),  # OHE on categorical features\n",
    "    (\"passthrough\", passthrough_feats),  # no transformations on the binary features    \n",
    "    (\n",
    "        OneHotEncoder(handle_unknown=\"ignore\"),\n",
    "        categorical_feats,\n",
    "    ),  # OHE on categorical features\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-7 {color: black;background-color: white;}#sk-container-id-7 pre{padding: 0;}#sk-container-id-7 div.sk-toggleable {background-color: white;}#sk-container-id-7 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-7 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-7 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-7 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-7 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-7 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-7 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-7 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-7 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-7 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-7 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-7 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-7 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-7 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-7 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-7 div.sk-item {position: relative;z-index: 1;}#sk-container-id-7 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-7 div.sk-item::before, #sk-container-id-7 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-7 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-7 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-7 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-7 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-7 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-7 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-7 div.sk-label-container {text-align: center;}#sk-container-id-7 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-7 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-7\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;onehotencoder-1&#x27;,\n",
       "                                 OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                               dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;enjoy_course&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder-2&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;])])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-64\" type=\"checkbox\" ><label for=\"sk-estimator-id-64\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer()),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;,\n",
       "                                  &#x27;lab4&#x27;, &#x27;quiz1&#x27;]),\n",
       "                                (&#x27;ordinalencoder&#x27;,\n",
       "                                 OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;,\n",
       "                                                             &#x27;Good&#x27;,\n",
       "                                                             &#x27;Excellent&#x27;]],\n",
       "                                                dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;class_attendance&#x27;]),\n",
       "                                (&#x27;onehotencoder-1&#x27;,\n",
       "                                 OneHotEncoder(drop=&#x27;if_binary&#x27;,\n",
       "                                               dtype=&lt;class &#x27;int&#x27;&gt;),\n",
       "                                 [&#x27;enjoy_course&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;ml_experience&#x27;]),\n",
       "                                (&#x27;onehotencoder-2&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;major&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-65\" type=\"checkbox\" ><label for=\"sk-estimator-id-65\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;university_years&#x27;, &#x27;lab1&#x27;, &#x27;lab2&#x27;, &#x27;lab3&#x27;, &#x27;lab4&#x27;, &#x27;quiz1&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-66\" type=\"checkbox\" ><label for=\"sk-estimator-id-66\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-67\" type=\"checkbox\" ><label for=\"sk-estimator-id-67\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-68\" type=\"checkbox\" ><label for=\"sk-estimator-id-68\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ordinalencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;class_attendance&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-69\" type=\"checkbox\" ><label for=\"sk-estimator-id-69\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OrdinalEncoder</label><div class=\"sk-toggleable__content\"><pre>OrdinalEncoder(categories=[[&#x27;Poor&#x27;, &#x27;Average&#x27;, &#x27;Good&#x27;, &#x27;Excellent&#x27;]],\n",
       "               dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-70\" type=\"checkbox\" ><label for=\"sk-estimator-id-70\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder-1</label><div class=\"sk-toggleable__content\"><pre>[&#x27;enjoy_course&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-71\" type=\"checkbox\" ><label for=\"sk-estimator-id-71\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(drop=&#x27;if_binary&#x27;, dtype=&lt;class &#x27;int&#x27;&gt;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-72\" type=\"checkbox\" ><label for=\"sk-estimator-id-72\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ml_experience&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-73\" type=\"checkbox\" ><label for=\"sk-estimator-id-73\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-74\" type=\"checkbox\" ><label for=\"sk-estimator-id-74\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder-2</label><div class=\"sk-toggleable__content\"><pre>[&#x27;major&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-75\" type=\"checkbox\" ><label for=\"sk-estimator-id-75\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer()),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['university_years', 'lab1', 'lab2', 'lab3',\n",
       "                                  'lab4', 'quiz1']),\n",
       "                                ('ordinalencoder',\n",
       "                                 OrdinalEncoder(categories=[['Poor', 'Average',\n",
       "                                                             'Good',\n",
       "                                                             'Excellent']],\n",
       "                                                dtype=<class 'int'>),\n",
       "                                 ['class_attendance']),\n",
       "                                ('onehotencoder-1',\n",
       "                                 OneHotEncoder(drop='if_binary',\n",
       "                                               dtype=<class 'int'>),\n",
       "                                 ['enjoy_course']),\n",
       "                                ('passthrough', 'passthrough',\n",
       "                                 ['ml_experience']),\n",
       "                                ('onehotencoder-2',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['major'])])"
      ]
     },
     "execution_count": 55,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ct"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe = make_pipeline(ct, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.004249</td>\n",
       "      <td>0.002322</td>\n",
       "      <td>1.00</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.003271</td>\n",
       "      <td>0.001830</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.003184</td>\n",
       "      <td>0.001976</td>\n",
       "      <td>0.50</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.003316</td>\n",
       "      <td>0.001788</td>\n",
       "      <td>1.00</td>\n",
       "      <td>0.941176</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.003072</td>\n",
       "      <td>0.001827</td>\n",
       "      <td>0.75</td>\n",
       "      <td>1.000000</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.004249    0.002322        1.00     1.000000\n",
       "1  0.003271    0.001830        1.00     0.941176\n",
       "2  0.003184    0.001976        0.50     1.000000\n",
       "3  0.003316    0.001788        1.00     0.941176\n",
       "4  0.003072    0.001827        0.75     1.000000"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "scores = cross_validate(pipe, X, y, cv=5, return_train_score=True)\n",
    "pd.DataFrame(scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{note}\n",
    "Do not read too much into the scores, as we are running cross-validation on a very small dataset with 21 examples. The main point here is to show you how can we use `ColumnTransformer` to apply different transformations on different columns.\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Break (5 min)\n",
    "\n",
    "![](img/eva-coffee.png)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## `ColumnTransformer` on the California housing dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6051</th>\n",
       "      <td>-117.75</td>\n",
       "      <td>34.04</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2948.0</td>\n",
       "      <td>636.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>602.0</td>\n",
       "      <td>3.1250</td>\n",
       "      <td>113600.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20113</th>\n",
       "      <td>-119.57</td>\n",
       "      <td>37.94</td>\n",
       "      <td>17.0</td>\n",
       "      <td>346.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.4861</td>\n",
       "      <td>137500.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14289</th>\n",
       "      <td>-117.13</td>\n",
       "      <td>32.74</td>\n",
       "      <td>46.0</td>\n",
       "      <td>3355.0</td>\n",
       "      <td>768.0</td>\n",
       "      <td>1457.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>2.6604</td>\n",
       "      <td>170100.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13665</th>\n",
       "      <td>-117.31</td>\n",
       "      <td>34.02</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1634.0</td>\n",
       "      <td>274.0</td>\n",
       "      <td>899.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>5.2139</td>\n",
       "      <td>129300.0</td>\n",
       "      <td>INLAND</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14471</th>\n",
       "      <td>-117.23</td>\n",
       "      <td>32.88</td>\n",
       "      <td>18.0</td>\n",
       "      <td>5566.0</td>\n",
       "      <td>1465.0</td>\n",
       "      <td>6303.0</td>\n",
       "      <td>1458.0</td>\n",
       "      <td>1.8580</td>\n",
       "      <td>205000.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "6051     -117.75     34.04                22.0       2948.0           636.0   \n",
       "20113    -119.57     37.94                17.0        346.0           130.0   \n",
       "14289    -117.13     32.74                46.0       3355.0           768.0   \n",
       "13665    -117.31     34.02                18.0       1634.0           274.0   \n",
       "14471    -117.23     32.88                18.0       5566.0          1465.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \\\n",
       "6051       2600.0       602.0         3.1250            113600.0   \n",
       "20113        51.0        20.0         3.4861            137500.0   \n",
       "14289      1457.0       708.0         2.6604            170100.0   \n",
       "13665       899.0       285.0         5.2139            129300.0   \n",
       "14471      6303.0      1458.0         1.8580            205000.0   \n",
       "\n",
       "      ocean_proximity  \n",
       "6051           INLAND  \n",
       "20113          INLAND  \n",
       "14289      NEAR OCEAN  \n",
       "13665          INLAND  \n",
       "14471      NEAR OCEAN  "
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "housing_df = pd.read_csv(\"data/housing.csv\")\n",
    "train_df, test_df = train_test_split(housing_df, test_size=0.1, random_state=123)\n",
    "\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Some column values are mean/median but some are not. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's add some new features to the dataset which could help predicting the target: `median_house_value`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "scrolled": true,
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "train_df = train_df.assign(\n",
    "    rooms_per_household=train_df[\"total_rooms\"] / train_df[\"households\"]\n",
    ")\n",
    "test_df = test_df.assign(\n",
    "    rooms_per_household=test_df[\"total_rooms\"] / test_df[\"households\"]\n",
    ")\n",
    "\n",
    "train_df = train_df.assign(\n",
    "    bedrooms_per_household=train_df[\"total_bedrooms\"] / train_df[\"households\"]\n",
    ")\n",
    "test_df = test_df.assign(\n",
    "    bedrooms_per_household=test_df[\"total_bedrooms\"] / test_df[\"households\"]\n",
    ")\n",
    "\n",
    "train_df = train_df.assign(\n",
    "    population_per_household=train_df[\"population\"] / train_df[\"households\"]\n",
    ")\n",
    "test_df = test_df.assign(\n",
    "    population_per_household=test_df[\"population\"] / test_df[\"households\"]\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>total_rooms</th>\n",
       "      <th>total_bedrooms</th>\n",
       "      <th>population</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>median_house_value</th>\n",
       "      <th>ocean_proximity</th>\n",
       "      <th>rooms_per_household</th>\n",
       "      <th>bedrooms_per_household</th>\n",
       "      <th>population_per_household</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6051</th>\n",
       "      <td>-117.75</td>\n",
       "      <td>34.04</td>\n",
       "      <td>22.0</td>\n",
       "      <td>2948.0</td>\n",
       "      <td>636.0</td>\n",
       "      <td>2600.0</td>\n",
       "      <td>602.0</td>\n",
       "      <td>3.1250</td>\n",
       "      <td>113600.0</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>4.897010</td>\n",
       "      <td>1.056478</td>\n",
       "      <td>4.318937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20113</th>\n",
       "      <td>-119.57</td>\n",
       "      <td>37.94</td>\n",
       "      <td>17.0</td>\n",
       "      <td>346.0</td>\n",
       "      <td>130.0</td>\n",
       "      <td>51.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.4861</td>\n",
       "      <td>137500.0</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>17.300000</td>\n",
       "      <td>6.500000</td>\n",
       "      <td>2.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14289</th>\n",
       "      <td>-117.13</td>\n",
       "      <td>32.74</td>\n",
       "      <td>46.0</td>\n",
       "      <td>3355.0</td>\n",
       "      <td>768.0</td>\n",
       "      <td>1457.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>2.6604</td>\n",
       "      <td>170100.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>4.738701</td>\n",
       "      <td>1.084746</td>\n",
       "      <td>2.057910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13665</th>\n",
       "      <td>-117.31</td>\n",
       "      <td>34.02</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1634.0</td>\n",
       "      <td>274.0</td>\n",
       "      <td>899.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>5.2139</td>\n",
       "      <td>129300.0</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>0.961404</td>\n",
       "      <td>3.154386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14471</th>\n",
       "      <td>-117.23</td>\n",
       "      <td>32.88</td>\n",
       "      <td>18.0</td>\n",
       "      <td>5566.0</td>\n",
       "      <td>1465.0</td>\n",
       "      <td>6303.0</td>\n",
       "      <td>1458.0</td>\n",
       "      <td>1.8580</td>\n",
       "      <td>205000.0</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>3.817558</td>\n",
       "      <td>1.004801</td>\n",
       "      <td>4.323045</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  total_rooms  total_bedrooms  \\\n",
       "6051     -117.75     34.04                22.0       2948.0           636.0   \n",
       "20113    -119.57     37.94                17.0        346.0           130.0   \n",
       "14289    -117.13     32.74                46.0       3355.0           768.0   \n",
       "13665    -117.31     34.02                18.0       1634.0           274.0   \n",
       "14471    -117.23     32.88                18.0       5566.0          1465.0   \n",
       "\n",
       "       population  households  median_income  median_house_value  \\\n",
       "6051       2600.0       602.0         3.1250            113600.0   \n",
       "20113        51.0        20.0         3.4861            137500.0   \n",
       "14289      1457.0       708.0         2.6604            170100.0   \n",
       "13665       899.0       285.0         5.2139            129300.0   \n",
       "14471      6303.0      1458.0         1.8580            205000.0   \n",
       "\n",
       "      ocean_proximity  rooms_per_household  bedrooms_per_household  \\\n",
       "6051           INLAND             4.897010                1.056478   \n",
       "20113          INLAND            17.300000                6.500000   \n",
       "14289      NEAR OCEAN             4.738701                1.084746   \n",
       "13665          INLAND             5.733333                0.961404   \n",
       "14471      NEAR OCEAN             3.817558                1.004801   \n",
       "\n",
       "       population_per_household  \n",
       "6051                   4.318937  \n",
       "20113                  2.550000  \n",
       "14289                  2.057910  \n",
       "13665                  3.154386  \n",
       "14471                  4.323045  "
      ]
     },
     "execution_count": 60,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's keep both numeric and categorical columns in the data.\n",
    "X_train = train_df.drop(columns=[\"median_house_value\", \"total_rooms\", \"total_bedrooms\", \"population\"])\n",
    "y_train = train_df[\"median_house_value\"]\n",
    "\n",
    "X_test = test_df.drop(columns=[\"median_house_value\", \"total_rooms\", \"total_bedrooms\", \"population\"])\n",
    "y_test = test_df[\"median_house_value\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.compose import ColumnTransformer, make_column_transformer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>ocean_proximity</th>\n",
       "      <th>rooms_per_household</th>\n",
       "      <th>bedrooms_per_household</th>\n",
       "      <th>population_per_household</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6051</th>\n",
       "      <td>-117.75</td>\n",
       "      <td>34.04</td>\n",
       "      <td>22.0</td>\n",
       "      <td>602.0</td>\n",
       "      <td>3.1250</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>4.897010</td>\n",
       "      <td>1.056478</td>\n",
       "      <td>4.318937</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>20113</th>\n",
       "      <td>-119.57</td>\n",
       "      <td>37.94</td>\n",
       "      <td>17.0</td>\n",
       "      <td>20.0</td>\n",
       "      <td>3.4861</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>17.300000</td>\n",
       "      <td>6.500000</td>\n",
       "      <td>2.550000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14289</th>\n",
       "      <td>-117.13</td>\n",
       "      <td>32.74</td>\n",
       "      <td>46.0</td>\n",
       "      <td>708.0</td>\n",
       "      <td>2.6604</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>4.738701</td>\n",
       "      <td>1.084746</td>\n",
       "      <td>2.057910</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13665</th>\n",
       "      <td>-117.31</td>\n",
       "      <td>34.02</td>\n",
       "      <td>18.0</td>\n",
       "      <td>285.0</td>\n",
       "      <td>5.2139</td>\n",
       "      <td>INLAND</td>\n",
       "      <td>5.733333</td>\n",
       "      <td>0.961404</td>\n",
       "      <td>3.154386</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14471</th>\n",
       "      <td>-117.23</td>\n",
       "      <td>32.88</td>\n",
       "      <td>18.0</td>\n",
       "      <td>1458.0</td>\n",
       "      <td>1.8580</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>3.817558</td>\n",
       "      <td>1.004801</td>\n",
       "      <td>4.323045</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9730</th>\n",
       "      <td>-121.74</td>\n",
       "      <td>36.79</td>\n",
       "      <td>16.0</td>\n",
       "      <td>611.0</td>\n",
       "      <td>4.3814</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>6.286416</td>\n",
       "      <td>1.014730</td>\n",
       "      <td>2.944354</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14690</th>\n",
       "      <td>-117.09</td>\n",
       "      <td>32.80</td>\n",
       "      <td>36.0</td>\n",
       "      <td>360.0</td>\n",
       "      <td>4.7188</td>\n",
       "      <td>NEAR OCEAN</td>\n",
       "      <td>6.008333</td>\n",
       "      <td>1.019444</td>\n",
       "      <td>2.541667</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7938</th>\n",
       "      <td>-118.11</td>\n",
       "      <td>33.86</td>\n",
       "      <td>33.0</td>\n",
       "      <td>393.0</td>\n",
       "      <td>5.3889</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>6.078880</td>\n",
       "      <td>1.043257</td>\n",
       "      <td>3.127226</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18365</th>\n",
       "      <td>-122.12</td>\n",
       "      <td>37.28</td>\n",
       "      <td>21.0</td>\n",
       "      <td>56.0</td>\n",
       "      <td>5.8691</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>6.232143</td>\n",
       "      <td>1.142857</td>\n",
       "      <td>2.660714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>10931</th>\n",
       "      <td>-117.91</td>\n",
       "      <td>33.74</td>\n",
       "      <td>25.0</td>\n",
       "      <td>922.0</td>\n",
       "      <td>2.9926</td>\n",
       "      <td>&lt;1H OCEAN</td>\n",
       "      <td>4.634490</td>\n",
       "      <td>1.046638</td>\n",
       "      <td>3.195228</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  households  median_income  \\\n",
       "6051     -117.75     34.04                22.0       602.0         3.1250   \n",
       "20113    -119.57     37.94                17.0        20.0         3.4861   \n",
       "14289    -117.13     32.74                46.0       708.0         2.6604   \n",
       "13665    -117.31     34.02                18.0       285.0         5.2139   \n",
       "14471    -117.23     32.88                18.0      1458.0         1.8580   \n",
       "9730     -121.74     36.79                16.0       611.0         4.3814   \n",
       "14690    -117.09     32.80                36.0       360.0         4.7188   \n",
       "7938     -118.11     33.86                33.0       393.0         5.3889   \n",
       "18365    -122.12     37.28                21.0        56.0         5.8691   \n",
       "10931    -117.91     33.74                25.0       922.0         2.9926   \n",
       "\n",
       "      ocean_proximity  rooms_per_household  bedrooms_per_household  \\\n",
       "6051           INLAND             4.897010                1.056478   \n",
       "20113          INLAND            17.300000                6.500000   \n",
       "14289      NEAR OCEAN             4.738701                1.084746   \n",
       "13665          INLAND             5.733333                0.961404   \n",
       "14471      NEAR OCEAN             3.817558                1.004801   \n",
       "9730        <1H OCEAN             6.286416                1.014730   \n",
       "14690      NEAR OCEAN             6.008333                1.019444   \n",
       "7938        <1H OCEAN             6.078880                1.043257   \n",
       "18365       <1H OCEAN             6.232143                1.142857   \n",
       "10931       <1H OCEAN             4.634490                1.046638   \n",
       "\n",
       "       population_per_household  \n",
       "6051                   4.318937  \n",
       "20113                  2.550000  \n",
       "14289                  2.057910  \n",
       "13665                  3.154386  \n",
       "14471                  4.323045  \n",
       "9730                   2.944354  \n",
       "14690                  2.541667  \n",
       "7938                   3.127226  \n",
       "18365                  2.660714  \n",
       "10931                  3.195228  "
      ]
     },
     "execution_count": 63,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['longitude', 'latitude', 'housing_median_age', 'households',\n",
       "       'median_income', 'ocean_proximity', 'rooms_per_household',\n",
       "       'bedrooms_per_household', 'population_per_household'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 64,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "# Identify the categorical and numeric columns\n",
    "numeric_features = [\n",
    "    \"longitude\",\n",
    "    \"latitude\",\n",
    "    \"housing_median_age\",\n",
    "    \"households\",\n",
    "    \"median_income\",\n",
    "    \"rooms_per_household\",\n",
    "    \"bedrooms_per_household\",\n",
    "    \"population_per_household\",\n",
    "]\n",
    "\n",
    "categorical_features = [\"ocean_proximity\"]\n",
    "target = \"median_income\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Let's create a `ColumnTransformer` for our dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 18576 entries, 6051 to 19966\n",
      "Data columns (total 9 columns):\n",
      " #   Column                    Non-Null Count  Dtype  \n",
      "---  ------                    --------------  -----  \n",
      " 0   longitude                 18576 non-null  float64\n",
      " 1   latitude                  18576 non-null  float64\n",
      " 2   housing_median_age        18576 non-null  float64\n",
      " 3   households                18576 non-null  float64\n",
      " 4   median_income             18576 non-null  float64\n",
      " 5   ocean_proximity           18576 non-null  object \n",
      " 6   rooms_per_household       18576 non-null  float64\n",
      " 7   bedrooms_per_household    18391 non-null  float64\n",
      " 8   population_per_household  18576 non-null  float64\n",
      "dtypes: float64(8), object(1)\n",
      "memory usage: 1.4+ MB\n"
     ]
    }
   ],
   "source": [
    "X_train.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<1H OCEAN     8221\n",
       "INLAND        5915\n",
       "NEAR OCEAN    2389\n",
       "NEAR BAY      2046\n",
       "ISLAND           5\n",
       "Name: ocean_proximity, dtype: int64"
      ]
     },
     "execution_count": 67,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[\"ocean_proximity\"].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "numeric_transformer = make_pipeline(SimpleImputer(strategy=\"median\"), StandardScaler())\n",
    "categorical_transformer = OneHotEncoder(handle_unknown=\"ignore\")\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (numeric_transformer, numeric_features),\n",
    "    (categorical_transformer, categorical_features),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-8 {color: black;background-color: white;}#sk-container-id-8 pre{padding: 0;}#sk-container-id-8 div.sk-toggleable {background-color: white;}#sk-container-id-8 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-8 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-8 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-8 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-8 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-8 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-8 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-8 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-8 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-8 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-8 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-8 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-8 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-8 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-8 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-8 div.sk-item {position: relative;z-index: 1;}#sk-container-id-8 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-8 div.sk-item::before, #sk-container-id-8 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-8 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-8 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-8 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-8 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-8 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-8 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-8 div.sk-label-container {text-align: center;}#sk-container-id-8 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-8 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-8\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-76\" type=\"checkbox\" ><label for=\"sk-estimator-id-76\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-77\" type=\"checkbox\" ><label for=\"sk-estimator-id-77\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;, &#x27;households&#x27;, &#x27;median_income&#x27;, &#x27;rooms_per_household&#x27;, &#x27;bedrooms_per_household&#x27;, &#x27;population_per_household&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-78\" type=\"checkbox\" ><label for=\"sk-estimator-id-78\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-79\" type=\"checkbox\" ><label for=\"sk-estimator-id-79\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-80\" type=\"checkbox\" ><label for=\"sk-estimator-id-80\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ocean_proximity&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-81\" type=\"checkbox\" ><label for=\"sk-estimator-id-81\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer(strategy='median')),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['longitude', 'latitude', 'housing_median_age',\n",
       "                                  'households', 'median_income',\n",
       "                                  'rooms_per_household',\n",
       "                                  'bedrooms_per_household',\n",
       "                                  'population_per_household']),\n",
       "                                ('onehotencoder',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['ocean_proximity'])])"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "X_train_pp = preprocessor.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- When we `fit` the preprocessor, it calls `fit` on _all_ the transformers\n",
    "- When we `transform` the preprocessor, it calls `transform` on _all_ the transformers. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can get the new names of the columns that were generated by the one-hot encoding:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-9 {color: black;background-color: white;}#sk-container-id-9 pre{padding: 0;}#sk-container-id-9 div.sk-toggleable {background-color: white;}#sk-container-id-9 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-9 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-9 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-9 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-9 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-9 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-9 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-9 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-9 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-9 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-9 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-9 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-9 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-9 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-9 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-9 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-9 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-9 div.sk-item {position: relative;z-index: 1;}#sk-container-id-9 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-9 div.sk-item::before, #sk-container-id-9 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-9 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-9 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-9 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-9 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-9 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-9 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-9 div.sk-label-container {text-align: center;}#sk-container-id-9 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-9 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-9\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-82\" type=\"checkbox\" ><label for=\"sk-estimator-id-82\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-83\" type=\"checkbox\" ><label for=\"sk-estimator-id-83\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;, &#x27;households&#x27;, &#x27;median_income&#x27;, &#x27;rooms_per_household&#x27;, &#x27;bedrooms_per_household&#x27;, &#x27;population_per_household&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-84\" type=\"checkbox\" ><label for=\"sk-estimator-id-84\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-85\" type=\"checkbox\" ><label for=\"sk-estimator-id-85\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-86\" type=\"checkbox\" ><label for=\"sk-estimator-id-86\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ocean_proximity&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-87\" type=\"checkbox\" ><label for=\"sk-estimator-id-87\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "ColumnTransformer(transformers=[('pipeline',\n",
       "                                 Pipeline(steps=[('simpleimputer',\n",
       "                                                  SimpleImputer(strategy='median')),\n",
       "                                                 ('standardscaler',\n",
       "                                                  StandardScaler())]),\n",
       "                                 ['longitude', 'latitude', 'housing_median_age',\n",
       "                                  'households', 'median_income',\n",
       "                                  'rooms_per_household',\n",
       "                                  'bedrooms_per_household',\n",
       "                                  'population_per_household']),\n",
       "                                ('onehotencoder',\n",
       "                                 OneHotEncoder(handle_unknown='ignore'),\n",
       "                                 ['ocean_proximity'])])"
      ]
     },
     "execution_count": 71,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['ocean_proximity_<1H OCEAN', 'ocean_proximity_INLAND',\n",
       "       'ocean_proximity_ISLAND', 'ocean_proximity_NEAR BAY',\n",
       "       'ocean_proximity_NEAR OCEAN'], dtype=object)"
      ]
     },
     "execution_count": 72,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "preprocessor.named_transformers_[\"onehotencoder\"].get_feature_names_out(\n",
    "    categorical_features\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Combining this with the numeric feature names gives us all the column names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['longitude',\n",
       " 'latitude',\n",
       " 'housing_median_age',\n",
       " 'households',\n",
       " 'median_income',\n",
       " 'rooms_per_household',\n",
       " 'bedrooms_per_household',\n",
       " 'population_per_household',\n",
       " 'ocean_proximity_<1H OCEAN',\n",
       " 'ocean_proximity_INLAND',\n",
       " 'ocean_proximity_ISLAND',\n",
       " 'ocean_proximity_NEAR BAY',\n",
       " 'ocean_proximity_NEAR OCEAN']"
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "column_names = numeric_features + list(\n",
    "    preprocessor.named_transformers_[\"onehotencoder\"].get_feature_names_out(\n",
    "        categorical_features\n",
    "    )\n",
    ")\n",
    "column_names"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's visualize the preprocessed training data as a dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>longitude</th>\n",
       "      <th>latitude</th>\n",
       "      <th>housing_median_age</th>\n",
       "      <th>households</th>\n",
       "      <th>median_income</th>\n",
       "      <th>rooms_per_household</th>\n",
       "      <th>bedrooms_per_household</th>\n",
       "      <th>population_per_household</th>\n",
       "      <th>ocean_proximity_&lt;1H OCEAN</th>\n",
       "      <th>ocean_proximity_INLAND</th>\n",
       "      <th>ocean_proximity_ISLAND</th>\n",
       "      <th>ocean_proximity_NEAR BAY</th>\n",
       "      <th>ocean_proximity_NEAR OCEAN</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.908140</td>\n",
       "      <td>-0.743917</td>\n",
       "      <td>-0.526078</td>\n",
       "      <td>0.266135</td>\n",
       "      <td>-0.389736</td>\n",
       "      <td>-0.210591</td>\n",
       "      <td>-0.083813</td>\n",
       "      <td>0.126398</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>-0.002057</td>\n",
       "      <td>1.083123</td>\n",
       "      <td>-0.923283</td>\n",
       "      <td>-1.253312</td>\n",
       "      <td>-0.198924</td>\n",
       "      <td>4.726412</td>\n",
       "      <td>11.166631</td>\n",
       "      <td>-0.050132</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1.218207</td>\n",
       "      <td>-1.352930</td>\n",
       "      <td>1.380504</td>\n",
       "      <td>0.542873</td>\n",
       "      <td>-0.635239</td>\n",
       "      <td>-0.273606</td>\n",
       "      <td>-0.025391</td>\n",
       "      <td>-0.099240</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1.128188</td>\n",
       "      <td>-0.753286</td>\n",
       "      <td>-0.843842</td>\n",
       "      <td>-0.561467</td>\n",
       "      <td>0.714077</td>\n",
       "      <td>0.122307</td>\n",
       "      <td>-0.280310</td>\n",
       "      <td>0.010183</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1.168196</td>\n",
       "      <td>-1.287344</td>\n",
       "      <td>-0.843842</td>\n",
       "      <td>2.500924</td>\n",
       "      <td>-1.059242</td>\n",
       "      <td>-0.640266</td>\n",
       "      <td>-0.190617</td>\n",
       "      <td>0.126808</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18571</th>\n",
       "      <td>0.733102</td>\n",
       "      <td>-0.804818</td>\n",
       "      <td>0.586095</td>\n",
       "      <td>-0.966131</td>\n",
       "      <td>-0.118182</td>\n",
       "      <td>0.063110</td>\n",
       "      <td>-0.099558</td>\n",
       "      <td>0.071541</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18572</th>\n",
       "      <td>1.163195</td>\n",
       "      <td>-1.057793</td>\n",
       "      <td>-1.161606</td>\n",
       "      <td>0.728235</td>\n",
       "      <td>0.357500</td>\n",
       "      <td>0.235096</td>\n",
       "      <td>-0.163397</td>\n",
       "      <td>0.007458</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18573</th>\n",
       "      <td>-1.097293</td>\n",
       "      <td>0.797355</td>\n",
       "      <td>-1.876574</td>\n",
       "      <td>0.514155</td>\n",
       "      <td>0.934269</td>\n",
       "      <td>0.211892</td>\n",
       "      <td>-0.135305</td>\n",
       "      <td>0.044029</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18574</th>\n",
       "      <td>-1.437367</td>\n",
       "      <td>1.008167</td>\n",
       "      <td>1.221622</td>\n",
       "      <td>-0.454427</td>\n",
       "      <td>0.006578</td>\n",
       "      <td>-0.273382</td>\n",
       "      <td>-0.149822</td>\n",
       "      <td>-0.132875</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>18575</th>\n",
       "      <td>0.242996</td>\n",
       "      <td>0.272667</td>\n",
       "      <td>-0.684960</td>\n",
       "      <td>-0.396991</td>\n",
       "      <td>-0.711754</td>\n",
       "      <td>0.025998</td>\n",
       "      <td>0.042957</td>\n",
       "      <td>0.051269</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>18576 rows × 13 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "       longitude  latitude  housing_median_age  households  median_income  \\\n",
       "0       0.908140 -0.743917           -0.526078    0.266135      -0.389736   \n",
       "1      -0.002057  1.083123           -0.923283   -1.253312      -0.198924   \n",
       "2       1.218207 -1.352930            1.380504    0.542873      -0.635239   \n",
       "3       1.128188 -0.753286           -0.843842   -0.561467       0.714077   \n",
       "4       1.168196 -1.287344           -0.843842    2.500924      -1.059242   \n",
       "...          ...       ...                 ...         ...            ...   \n",
       "18571   0.733102 -0.804818            0.586095   -0.966131      -0.118182   \n",
       "18572   1.163195 -1.057793           -1.161606    0.728235       0.357500   \n",
       "18573  -1.097293  0.797355           -1.876574    0.514155       0.934269   \n",
       "18574  -1.437367  1.008167            1.221622   -0.454427       0.006578   \n",
       "18575   0.242996  0.272667           -0.684960   -0.396991      -0.711754   \n",
       "\n",
       "       rooms_per_household  bedrooms_per_household  population_per_household  \\\n",
       "0                -0.210591               -0.083813                  0.126398   \n",
       "1                 4.726412               11.166631                 -0.050132   \n",
       "2                -0.273606               -0.025391                 -0.099240   \n",
       "3                 0.122307               -0.280310                  0.010183   \n",
       "4                -0.640266               -0.190617                  0.126808   \n",
       "...                    ...                     ...                       ...   \n",
       "18571             0.063110               -0.099558                  0.071541   \n",
       "18572             0.235096               -0.163397                  0.007458   \n",
       "18573             0.211892               -0.135305                  0.044029   \n",
       "18574            -0.273382               -0.149822                 -0.132875   \n",
       "18575             0.025998                0.042957                  0.051269   \n",
       "\n",
       "       ocean_proximity_<1H OCEAN  ocean_proximity_INLAND  \\\n",
       "0                            0.0                     1.0   \n",
       "1                            0.0                     1.0   \n",
       "2                            0.0                     0.0   \n",
       "3                            0.0                     1.0   \n",
       "4                            0.0                     0.0   \n",
       "...                          ...                     ...   \n",
       "18571                        1.0                     0.0   \n",
       "18572                        1.0                     0.0   \n",
       "18573                        1.0                     0.0   \n",
       "18574                        0.0                     0.0   \n",
       "18575                        0.0                     1.0   \n",
       "\n",
       "       ocean_proximity_ISLAND  ocean_proximity_NEAR BAY  \\\n",
       "0                         0.0                       0.0   \n",
       "1                         0.0                       0.0   \n",
       "2                         0.0                       0.0   \n",
       "3                         0.0                       0.0   \n",
       "4                         0.0                       0.0   \n",
       "...                       ...                       ...   \n",
       "18571                     0.0                       0.0   \n",
       "18572                     0.0                       0.0   \n",
       "18573                     0.0                       0.0   \n",
       "18574                     0.0                       1.0   \n",
       "18575                     0.0                       0.0   \n",
       "\n",
       "       ocean_proximity_NEAR OCEAN  \n",
       "0                             0.0  \n",
       "1                             0.0  \n",
       "2                             1.0  \n",
       "3                             0.0  \n",
       "4                             1.0  \n",
       "...                           ...  \n",
       "18571                         0.0  \n",
       "18572                         0.0  \n",
       "18573                         0.0  \n",
       "18574                         0.0  \n",
       "18575                         0.0  \n",
       "\n",
       "[18576 rows x 13 columns]"
      ]
     },
     "execution_count": 74,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(X_train_pp, columns=column_names)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                fit_time         score_time          test_score  \\\n",
       "dummy  0.001 (+/- 0.001)  0.000 (+/- 0.000)  -0.001 (+/- 0.001)   \n",
       "\n",
       "             train_score  \n",
       "dummy  0.000 (+/- 0.000)  "
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results_dict = {}\n",
    "dummy = DummyRegressor()\n",
    "results_dict[\"dummy\"] = mean_std_cross_val_scores(\n",
    "    dummy, X_train, y_train, return_train_score=True\n",
    ")\n",
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.svm import SVR\n",
    "\n",
    "knn_pipe = make_pipeline(preprocessor, KNeighborsRegressor())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-10 {color: black;background-color: white;}#sk-container-id-10 pre{padding: 0;}#sk-container-id-10 div.sk-toggleable {background-color: white;}#sk-container-id-10 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-10 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-10 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-10 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-10 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-10 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-10 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-10 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-10 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-10 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-10 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-10 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-10 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-10 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-10 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-10 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-10 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-10 div.sk-item {position: relative;z-index: 1;}#sk-container-id-10 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-10 div.sk-item::before, #sk-container-id-10 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-10 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-10 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-10 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-10 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-10 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-10 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-10 div.sk-label-container {text-align: center;}#sk-container-id-10 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-10 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-10\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                                  (&#x27;standardscaler&#x27;,\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [&#x27;longitude&#x27;, &#x27;latitude&#x27;,\n",
       "                                                   &#x27;housing_median_age&#x27;,\n",
       "                                                   &#x27;households&#x27;,\n",
       "                                                   &#x27;median_income&#x27;,\n",
       "                                                   &#x27;rooms_per_household&#x27;,\n",
       "                                                   &#x27;bedrooms_per_household&#x27;,\n",
       "                                                   &#x27;population_per_household&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                                  [&#x27;ocean_proximity&#x27;])])),\n",
       "                (&#x27;kneighborsregressor&#x27;, KNeighborsRegressor())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-88\" type=\"checkbox\" ><label for=\"sk-estimator-id-88\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                                  Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                                   SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                                  (&#x27;standardscaler&#x27;,\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  [&#x27;longitude&#x27;, &#x27;latitude&#x27;,\n",
       "                                                   &#x27;housing_median_age&#x27;,\n",
       "                                                   &#x27;households&#x27;,\n",
       "                                                   &#x27;median_income&#x27;,\n",
       "                                                   &#x27;rooms_per_household&#x27;,\n",
       "                                                   &#x27;bedrooms_per_household&#x27;,\n",
       "                                                   &#x27;population_per_household&#x27;]),\n",
       "                                                 (&#x27;onehotencoder&#x27;,\n",
       "                                                  OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                                  [&#x27;ocean_proximity&#x27;])])),\n",
       "                (&#x27;kneighborsregressor&#x27;, KNeighborsRegressor())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-89\" type=\"checkbox\" ><label for=\"sk-estimator-id-89\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;pipeline&#x27;,\n",
       "                                 Pipeline(steps=[(&#x27;simpleimputer&#x27;,\n",
       "                                                  SimpleImputer(strategy=&#x27;median&#x27;)),\n",
       "                                                 (&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler())]),\n",
       "                                 [&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;,\n",
       "                                  &#x27;households&#x27;, &#x27;median_income&#x27;,\n",
       "                                  &#x27;rooms_per_household&#x27;,\n",
       "                                  &#x27;bedrooms_per_household&#x27;,\n",
       "                                  &#x27;population_per_household&#x27;]),\n",
       "                                (&#x27;onehotencoder&#x27;,\n",
       "                                 OneHotEncoder(handle_unknown=&#x27;ignore&#x27;),\n",
       "                                 [&#x27;ocean_proximity&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-90\" type=\"checkbox\" ><label for=\"sk-estimator-id-90\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">pipeline</label><div class=\"sk-toggleable__content\"><pre>[&#x27;longitude&#x27;, &#x27;latitude&#x27;, &#x27;housing_median_age&#x27;, &#x27;households&#x27;, &#x27;median_income&#x27;, &#x27;rooms_per_household&#x27;, &#x27;bedrooms_per_household&#x27;, &#x27;population_per_household&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-91\" type=\"checkbox\" ><label for=\"sk-estimator-id-91\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SimpleImputer</label><div class=\"sk-toggleable__content\"><pre>SimpleImputer(strategy=&#x27;median&#x27;)</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-92\" type=\"checkbox\" ><label for=\"sk-estimator-id-92\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-93\" type=\"checkbox\" ><label for=\"sk-estimator-id-93\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">onehotencoder</label><div class=\"sk-toggleable__content\"><pre>[&#x27;ocean_proximity&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-94\" type=\"checkbox\" ><label for=\"sk-estimator-id-94\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">OneHotEncoder</label><div class=\"sk-toggleable__content\"><pre>OneHotEncoder(handle_unknown=&#x27;ignore&#x27;)</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-95\" type=\"checkbox\" ><label for=\"sk-estimator-id-95\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">KNeighborsRegressor</label><div class=\"sk-toggleable__content\"><pre>KNeighborsRegressor()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('columntransformer',\n",
       "                 ColumnTransformer(transformers=[('pipeline',\n",
       "                                                  Pipeline(steps=[('simpleimputer',\n",
       "                                                                   SimpleImputer(strategy='median')),\n",
       "                                                                  ('standardscaler',\n",
       "                                                                   StandardScaler())]),\n",
       "                                                  ['longitude', 'latitude',\n",
       "                                                   'housing_median_age',\n",
       "                                                   'households',\n",
       "                                                   'median_income',\n",
       "                                                   'rooms_per_household',\n",
       "                                                   'bedrooms_per_household',\n",
       "                                                   'population_per_household']),\n",
       "                                                 ('onehotencoder',\n",
       "                                                  OneHotEncoder(handle_unknown='ignore'),\n",
       "                                                  ['ocean_proximity'])])),\n",
       "                ('kneighborsregressor', KNeighborsRegressor())])"
      ]
     },
     "execution_count": 77,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "knn_pipe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "metadata": {},
   "outputs": [],
   "source": [
    "results_dict[\"imp + scaling + ohe + KNN\"] = mean_std_cross_val_scores(\n",
    "    knn_pipe, X_train, y_train, return_train_score=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + KNN</th>\n",
       "      <td>0.016 (+/- 0.001)</td>\n",
       "      <td>0.127 (+/- 0.011)</td>\n",
       "      <td>0.706 (+/- 0.015)</td>\n",
       "      <td>0.806 (+/- 0.015)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                    fit_time         score_time  \\\n",
       "dummy                      0.001 (+/- 0.001)  0.000 (+/- 0.000)   \n",
       "imp + scaling + ohe + KNN  0.016 (+/- 0.001)  0.127 (+/- 0.011)   \n",
       "\n",
       "                                   test_score        train_score  \n",
       "dummy                      -0.001 (+/- 0.001)  0.000 (+/- 0.000)  \n",
       "imp + scaling + ohe + KNN   0.706 (+/- 0.015)  0.806 (+/- 0.015)  "
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "metadata": {},
   "outputs": [],
   "source": [
    "svr_pipe = make_pipeline(preprocessor, SVR())\n",
    "results_dict[\"imp + scaling + ohe + SVR (default)\"] = mean_std_cross_val_scores(\n",
    "    svr_pipe, X_train, y_train, return_train_score=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + KNN</th>\n",
       "      <td>0.016 (+/- 0.001)</td>\n",
       "      <td>0.127 (+/- 0.011)</td>\n",
       "      <td>0.706 (+/- 0.015)</td>\n",
       "      <td>0.806 (+/- 0.015)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + SVR (default)</th>\n",
       "      <td>5.999 (+/- 0.131)</td>\n",
       "      <td>1.705 (+/- 0.011)</td>\n",
       "      <td>-0.049 (+/- 0.012)</td>\n",
       "      <td>-0.048 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              fit_time         score_time  \\\n",
       "dummy                                0.001 (+/- 0.001)  0.000 (+/- 0.000)   \n",
       "imp + scaling + ohe + KNN            0.016 (+/- 0.001)  0.127 (+/- 0.011)   \n",
       "imp + scaling + ohe + SVR (default)  5.999 (+/- 0.131)  1.705 (+/- 0.011)   \n",
       "\n",
       "                                             test_score         train_score  \n",
       "dummy                                -0.001 (+/- 0.001)   0.000 (+/- 0.000)  \n",
       "imp + scaling + ohe + KNN             0.706 (+/- 0.015)   0.806 (+/- 0.015)  \n",
       "imp + scaling + ohe + SVR (default)  -0.049 (+/- 0.012)  -0.048 (+/- 0.001)  "
      ]
     },
     "execution_count": 81,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The results with `scikit-learn`'s default SVR hyperparameters are pretty bad. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "metadata": {},
   "outputs": [],
   "source": [
    "svr_C_pipe = make_pipeline(preprocessor, SVR(C=10000))\n",
    "results_dict[\"imp + scaling + ohe + SVR (C=10000)\"] = mean_std_cross_val_scores(\n",
    "    svr_C_pipe, X_train, y_train, return_train_score=True\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>-0.001 (+/- 0.001)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + KNN</th>\n",
       "      <td>0.016 (+/- 0.001)</td>\n",
       "      <td>0.127 (+/- 0.011)</td>\n",
       "      <td>0.706 (+/- 0.015)</td>\n",
       "      <td>0.806 (+/- 0.015)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + SVR (default)</th>\n",
       "      <td>5.999 (+/- 0.131)</td>\n",
       "      <td>1.705 (+/- 0.011)</td>\n",
       "      <td>-0.049 (+/- 0.012)</td>\n",
       "      <td>-0.048 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>imp + scaling + ohe + SVR (C=10000)</th>\n",
       "      <td>5.511 (+/- 0.088)</td>\n",
       "      <td>1.730 (+/- 0.027)</td>\n",
       "      <td>0.708 (+/- 0.011)</td>\n",
       "      <td>0.713 (+/- 0.017)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              fit_time         score_time  \\\n",
       "dummy                                0.001 (+/- 0.001)  0.000 (+/- 0.000)   \n",
       "imp + scaling + ohe + KNN            0.016 (+/- 0.001)  0.127 (+/- 0.011)   \n",
       "imp + scaling + ohe + SVR (default)  5.999 (+/- 0.131)  1.705 (+/- 0.011)   \n",
       "imp + scaling + ohe + SVR (C=10000)  5.511 (+/- 0.088)  1.730 (+/- 0.027)   \n",
       "\n",
       "                                             test_score         train_score  \n",
       "dummy                                -0.001 (+/- 0.001)   0.000 (+/- 0.000)  \n",
       "imp + scaling + ohe + KNN             0.706 (+/- 0.015)   0.806 (+/- 0.015)  \n",
       "imp + scaling + ohe + SVR (default)  -0.049 (+/- 0.012)  -0.048 (+/- 0.001)  \n",
       "imp + scaling + ohe + SVR (C=10000)   0.708 (+/- 0.011)   0.713 (+/- 0.017)  "
      ]
     },
     "execution_count": 83,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pd.DataFrame(results_dict).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a bigger value for `C` the results are much better. We need to carry out systematic hyperparameter optimization to get better results. (Coming up next week.)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "- Note that categorical features are different than free text features. Sometimes there are columns containing free text information and we we'll look at ways to deal with them in the later part of this lecture. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### OHE with many categories"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- Do we have enough data for rare categories to learn anything meaningful? \n",
    "- How about grouping them into bigger categories?\n",
    "    - Example: country names into continents such as \"South America\" or \"Asia\"\n",
    "- Or having \"other\" category for rare cases? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Do we actually want to use certain features for prediction?\n",
    "\n",
    "- Do you want to use certain features such as **gender** or **race** in prediction?\n",
    "- Remember that the systems you build are going to be used in some applications. \n",
    "- It's extremely important to be mindful of the consequences of including certain features in your predictive model. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Preprocessing the targets?\n",
    "\n",
    "- Generally no need for this when doing classification. \n",
    "- In regression it makes sense in some cases. More on this later. \n",
    "- `sklearn` is fine with categorical labels ($y$-values) for classification problems. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Encoding text data  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "toy_spam = [\n",
    "    [\n",
    "        \"URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!\",\n",
    "        \"spam\",\n",
    "    ],\n",
    "    [\"Lol you are always so convincing.\", \"non spam\"],\n",
    "    [\"Nah I don't think he goes to usf, he lives around here though\", \"non spam\"],\n",
    "    [\n",
    "        \"URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!\",\n",
    "        \"spam\",\n",
    "    ],\n",
    "    [\n",
    "        \"Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030\",\n",
    "        \"spam\",\n",
    "    ],\n",
    "    [\"Congrats! I can't wait to see you!!\", \"non spam\"],\n",
    "]\n",
    "toy_df = pd.DataFrame(toy_spam, columns=[\"sms\", \"target\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Spam/non spam toy example "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- What if the feature is in the form of raw text?\n",
    "- The feature `sms` below is neither categorical nor ordinal. \n",
    "- How can we encode it so that we can pass it to the machine learning algorithms we have seen so far? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sms</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Lol you are always so convincing.</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Nah I don't think he goes to usf, he lives around here though</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</td>\n",
       "      <td>spam</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Congrats! I can't wait to see you!!</td>\n",
       "      <td>non spam</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                          sms  \\\n",
       "0                                                                URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!   \n",
       "1                                                                                                                           Lol you are always so convincing.   \n",
       "2                                                                                               Nah I don't think he goes to usf, he lives around here though   \n",
       "3                                                                                 URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!   \n",
       "4  Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   \n",
       "5                                                                                                                         Congrats! I can't wait to see you!!   \n",
       "\n",
       "     target  \n",
       "0      spam  \n",
       "1  non spam  \n",
       "2  non spam  \n",
       "3      spam  \n",
       "4      spam  \n",
       "5  non spam  "
      ]
     },
     "execution_count": 85,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "toy_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What if we apply OHE? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 86,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead tr th {\n",
       "        text-align: left;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr>\n",
       "      <th></th>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Congrats! I can't wait to see you!!  \\\n",
       "0                                 0.0   \n",
       "1                                 0.0   \n",
       "2                                 0.0   \n",
       "3                                 0.0   \n",
       "4                                 0.0   \n",
       "5                                 1.0   \n",
       "\n",
       "  Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  \\\n",
       "0                                                                                                                                                        0.0   \n",
       "1                                                                                                                                                        0.0   \n",
       "2                                                                                                                                                        0.0   \n",
       "3                                                                                                                                                        0.0   \n",
       "4                                                                                                                                                        1.0   \n",
       "5                                                                                                                                                        0.0   \n",
       "\n",
       "  Lol you are always so convincing.  \\\n",
       "0                               0.0   \n",
       "1                               1.0   \n",
       "2                               0.0   \n",
       "3                               0.0   \n",
       "4                               0.0   \n",
       "5                               0.0   \n",
       "\n",
       "  Nah I don't think he goes to usf, he lives around here though  \\\n",
       "0                                                           0.0   \n",
       "1                                                           0.0   \n",
       "2                                                           1.0   \n",
       "3                                                           0.0   \n",
       "4                                                           0.0   \n",
       "5                                                           0.0   \n",
       "\n",
       "  URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!  \\\n",
       "0                                                                         0.0   \n",
       "1                                                                         0.0   \n",
       "2                                                                         0.0   \n",
       "3                                                                         1.0   \n",
       "4                                                                         0.0   \n",
       "5                                                                         0.0   \n",
       "\n",
       "  URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!  \n",
       "0                                                                                          1.0  \n",
       "1                                                                                          0.0  \n",
       "2                                                                                          0.0  \n",
       "3                                                                                          0.0  \n",
       "4                                                                                          0.0  \n",
       "5                                                                                          0.0  "
      ]
     },
     "execution_count": 86,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "### DO NOT DO THIS.\n",
    "enc = OneHotEncoder(sparse=False)\n",
    "transformed = enc.fit_transform(toy_df[[\"sms\"]])\n",
    "pd.DataFrame(transformed, columns=enc.categories_)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "- We do not have a fixed number of categories here. \n",
    "- Each \"category\" (feature value) is likely to occur only once in the training data and we won't learn anything meaningful if we apply one-hot encoding or ordinal encoding on this feature. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- How can we encode or represent raw text data into fixed number of features so that we can learn some useful patterns from it?  \n",
    "- This is a well studied problem in the field of Natural Language Processing (NLP), which is concerned with giving computers the ability to understand written and spoken language. \n",
    "- Some popular representations of raw text include: \n",
    "    - **Bag of words** \n",
    "    - TF-IDF\n",
    "    - Embedding representations "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bag of words (BOW) representation\n",
    "\n",
    "- One of the most popular representation of raw text \n",
    "- Ignores the syntax and word order\n",
    "- It has two components: \n",
    "    - The vocabulary (all unique words in all documents) \n",
    "    - A value indicating either the presence or absence or the count of each word in the document. \n",
    "\n",
    "![](img/bag-of-words.png)\n",
    "\n",
    "<!-- <center>\n",
    "<img src='./img/bag-of-words.png' width=\"600\">\n",
    "</center>\n",
    " -->\n",
    "[Source](https://web.stanford.edu/~jurafsky/slp3/4.pdf)       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Extracting BOW features using `scikit-learn`\n",
    "- `CountVectorizer`\n",
    "    - Converts a collection of text documents to a matrix of word counts.  \n",
    "    - Each row represents a \"document\" (e.g., a text message in our example). \n",
    "    - Each column represents a word in the vocabulary (the set of unique words) in the training data. \n",
    "    - Each cell represents how often the word occurs in the document.       "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{note}\n",
    "In the Natural Language Processing (NLP) community text data  is referred to as a **corpus** (plural: corpora). ```   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 87,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>08002986030</th>\n",
       "      <th>100000</th>\n",
       "      <th>11</th>\n",
       "      <th>900</th>\n",
       "      <th>always</th>\n",
       "      <th>are</th>\n",
       "      <th>around</th>\n",
       "      <th>as</th>\n",
       "      <th>been</th>\n",
       "      <th>call</th>\n",
       "      <th>...</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>usf</th>\n",
       "      <th>valued</th>\n",
       "      <th>wait</th>\n",
       "      <th>week</th>\n",
       "      <th>with</th>\n",
       "      <th>won</th>\n",
       "      <th>you</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            08002986030  \\\n",
       "sms                                                                                                                                                                       \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                          0   \n",
       "Lol you are always so convincing.                                                                                                                                     0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                         0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                           0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030            1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                                   0   \n",
       "\n",
       "                                                                                                                                                            100000  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            11  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 0   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            900  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            always  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            are  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            around  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            as  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            been  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            call  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            ...  \\\n",
       "sms                                                                                                                                                         ...   \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                ...   \n",
       "Lol you are always so convincing.                                                                                                                           ...   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                               ...   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                 ...   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  ...   \n",
       "Congrats! I can't wait to see you!!                                                                                                                         ...   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            usf  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            valued  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            wait  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            1   \n",
       "\n",
       "                                                                                                                                                            week  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            with  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            won  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            you  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1   \n",
       "\n",
       "                                                                                                                                                            your  \n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0  \n",
       "Lol you are always so convincing.                                                                                                                              0  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1  \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0  \n",
       "\n",
       "[6 rows x 61 columns]"
      ]
     },
     "execution_count": 87,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "vec = CountVectorizer()\n",
    "X_counts = vec.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec.get_feature_names(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "pandas.core.series.Series"
      ]
     },
     "execution_count": 88,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "type(toy_df[\"sms\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{important}\n",
    "Note that unlike other transformers we are passing a `Series` object to `fit_transform`. For other transformers, you can define one transformer for more than one columns. But with `CountVectorizer` you need to define separate `CountVectorizer` transformers for each text column, if you have more than one text columns.    \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<6x61 sparse matrix of type '<class 'numpy.int64'>'\n",
       "\twith 71 stored elements in Compressed Sparse Row format>"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_counts"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### Why sparse matrices? \n",
    "\n",
    "- Most words do not appear in a given document.\n",
    "- We get massive computational savings if we only store the nonzero elements.\n",
    "- There is a bit of overhead, because we also need to store the locations:\n",
    "    - e.g. \"location (3,27): 1\".\n",
    "    \n",
    "- However, if the fraction of nonzero is small, this is a huge win."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 90,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The total number of elements:  366\n",
      "The number of non-zero elements:  71\n",
      "Proportion of non-zero elements: 0.1940\n",
      "The value at cell 3,27 is: 1\n"
     ]
    }
   ],
   "source": [
    "print(\"The total number of elements: \", np.prod(X_counts.shape))\n",
    "print(\"The number of non-zero elements: \", X_counts.nnz)\n",
    "print(\n",
    "    \"Proportion of non-zero elements: %0.4f\" % (X_counts.nnz / np.prod(X_counts.shape))\n",
    ")\n",
    "print(\n",
    "    \"The value at cell 3,%d is: %d\"\n",
    "    % (vec.vocabulary_[\"jackpot\"], X_counts[3, vec.vocabulary_[\"jackpot\"]])\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "\n",
    "Question for you\n",
    "- What would happen if you apply `StandardScaler` on sparse data? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### `OneHotEncoder` and sparse features \n",
    "- By default, `OneHotEncoder` also creates sparse features. \n",
    "- You could set `sparse=False` to get a regular `numpy` array. \n",
    "- If there are a huge number of categories, it may be beneficial to keep them sparse.\n",
    "- For smaller number of categories, it doesn't matter much."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Important hyperparameters of `CountVectorizer` \n",
    "\n",
    "- `binary`\n",
    "    - whether to use absence/presence feature values or counts\n",
    "- `max_features`\n",
    "    - only consider top `max_features` ordered by frequency in the corpus\n",
    "- `max_df`\n",
    "    - ignore features which occur in more than `max_df` documents \n",
    "- `min_df` \n",
    "    - ignore features which occur in less than `min_df` documents \n",
    "- `ngram_range`\n",
    "    - consider word sequences in the given range "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's look at all features, i.e., words (along with their frequencies)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>08002986030</th>\n",
       "      <th>100000</th>\n",
       "      <th>11</th>\n",
       "      <th>900</th>\n",
       "      <th>always</th>\n",
       "      <th>are</th>\n",
       "      <th>around</th>\n",
       "      <th>as</th>\n",
       "      <th>been</th>\n",
       "      <th>call</th>\n",
       "      <th>...</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>usf</th>\n",
       "      <th>valued</th>\n",
       "      <th>wait</th>\n",
       "      <th>week</th>\n",
       "      <th>with</th>\n",
       "      <th>won</th>\n",
       "      <th>you</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            08002986030  \\\n",
       "sms                                                                                                                                                                       \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                          0   \n",
       "Lol you are always so convincing.                                                                                                                                     0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                         0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                           0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030            1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                                   0   \n",
       "\n",
       "                                                                                                                                                            100000  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            11  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 0   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            900  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            always  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            are  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            around  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            as  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            been  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            call  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            ...  \\\n",
       "sms                                                                                                                                                         ...   \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                ...   \n",
       "Lol you are always so convincing.                                                                                                                           ...   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                               ...   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                 ...   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  ...   \n",
       "Congrats! I can't wait to see you!!                                                                                                                         ...   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            usf  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            valued  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            wait  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            1   \n",
       "\n",
       "                                                                                                                                                            week  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            with  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            won  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            you  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1   \n",
       "\n",
       "                                                                                                                                                            your  \n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0  \n",
       "Lol you are always so convincing.                                                                                                                              0  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1  \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0  \n",
       "\n",
       "[6 rows x 61 columns]"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec = CountVectorizer()\n",
    "X_counts = vec.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec.get_feature_names(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "When we use `binary=True`, the representation uses presence/absence of words instead of word counts.   "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 92,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>08002986030</th>\n",
       "      <th>100000</th>\n",
       "      <th>11</th>\n",
       "      <th>900</th>\n",
       "      <th>always</th>\n",
       "      <th>are</th>\n",
       "      <th>around</th>\n",
       "      <th>as</th>\n",
       "      <th>been</th>\n",
       "      <th>call</th>\n",
       "      <th>...</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>usf</th>\n",
       "      <th>valued</th>\n",
       "      <th>wait</th>\n",
       "      <th>week</th>\n",
       "      <th>with</th>\n",
       "      <th>won</th>\n",
       "      <th>you</th>\n",
       "      <th>your</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>6 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            08002986030  \\\n",
       "sms                                                                                                                                                                       \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                          0   \n",
       "Lol you are always so convincing.                                                                                                                                     0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                         0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                           0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030            1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                                   0   \n",
       "\n",
       "                                                                                                                                                            100000  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            11  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 0   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            900  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            always  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            are  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            around  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            as  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          0   \n",
       "\n",
       "                                                                                                                                                            been  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            call  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            ...  \\\n",
       "sms                                                                                                                                                         ...   \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                ...   \n",
       "Lol you are always so convincing.                                                                                                                           ...   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                               ...   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                 ...   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030  ...   \n",
       "Congrats! I can't wait to see you!!                                                                                                                         ...   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            usf  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            valued  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            wait  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            1   \n",
       "\n",
       "                                                                                                                                                            week  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            with  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            won  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            you  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1   \n",
       "Lol you are always so convincing.                                                                                                                             1   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1   \n",
       "\n",
       "                                                                                                                                                            your  \n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0  \n",
       "Lol you are always so convincing.                                                                                                                              0  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    0  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     1  \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0  \n",
       "\n",
       "[6 rows x 61 columns]"
      ]
     },
     "execution_count": 92,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec_binary = CountVectorizer(binary=True)\n",
    "X_counts = vec_binary.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec_binary.get_feature_names(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can control the size of X (the number of features) using `max_features`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>free</th>\n",
       "      <th>have</th>\n",
       "      <th>mobile</th>\n",
       "      <th>the</th>\n",
       "      <th>to</th>\n",
       "      <th>update</th>\n",
       "      <th>urgent</th>\n",
       "      <th>you</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>sms</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Nah I don't think he goes to usf, he lives around here though</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030</th>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Congrats! I can't wait to see you!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                                                                                                                            free  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   0   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            have  \\\n",
       "sms                                                                                                                                                                \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                   1   \n",
       "Lol you are always so convincing.                                                                                                                              0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                  0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                    1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030     0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                            0   \n",
       "\n",
       "                                                                                                                                                            mobile  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            the  \\\n",
       "sms                                                                                                                                                               \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  0   \n",
       "Lol you are always so convincing.                                                                                                                             0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                           0   \n",
       "\n",
       "                                                                                                                                                            to  \\\n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                 1   \n",
       "Lol you are always so convincing.                                                                                                                            0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                1   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                  0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030   2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                          1   \n",
       "\n",
       "                                                                                                                                                            update  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     0   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      0   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       2   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            urgent  \\\n",
       "sms                                                                                                                                                                  \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                     1   \n",
       "Lol you are always so convincing.                                                                                                                                0   \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                    0   \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                      1   \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030       0   \n",
       "Congrats! I can't wait to see you!!                                                                                                                              0   \n",
       "\n",
       "                                                                                                                                                            you  \n",
       "sms                                                                                                                                                              \n",
       "URGENT!! As a valued network customer you have been selected to receive a £900 prize reward!                                                                  1  \n",
       "Lol you are always so convincing.                                                                                                                             1  \n",
       "Nah I don't think he goes to usf, he lives around here though                                                                                                 0  \n",
       "URGENT! You have won a 1 week FREE membership in our £100000 prize Jackpot!                                                                                   1  \n",
       "Had your mobile 11 months or more? U R entitled to Update to the latest colour mobiles with camera for Free! Call The Mobile Update Co FREE on 08002986030    0  \n",
       "Congrats! I can't wait to see you!!                                                                                                                           1  "
      ]
     },
     "execution_count": 93,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec8 = CountVectorizer(max_features=8)\n",
    "X_counts = vec8.fit_transform(toy_df[\"sms\"])\n",
    "bow_df = pd.DataFrame(\n",
    "    X_counts.toarray(), columns=vec8.get_feature_names(), index=toy_df[\"sms\"]\n",
    ")\n",
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{note}\n",
    "Notice that `vec8` and `vec8_binary` have different vocabularies, which is kind of unexpected behaviour and doesn't match the documentation of `scikit-learn`. \n",
    "\n",
    "[Here](https://github.com/scikit-learn/scikit-learn/blob/master/sklearn/feature_extraction/text.py#L1206-L1225) is the code for `binary=True` condition in `scikit-learn`. As we can see, the binarization is done before limiting the features to `max_features`, and so now we are actually looking at the document counts (in how many documents it occurs) rather than term count. This is not explained anywhere in the documentation. \n",
    "\n",
    "The ties in counts between different words makes it even more confusing. I don't think it'll have a big impact on the results but this is good to know! Remember that `scikit-learn` developers are also humans who are prone to make mistakes. So it's always a good habit to question whatever tools we use every now and then. \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 94,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>you</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>free</th>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>have</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mobile</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>the</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>update</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>urgent</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        counts\n",
       "to           5\n",
       "you          4\n",
       "free         3\n",
       "have         2\n",
       "mobile       2\n",
       "the          2\n",
       "update       2\n",
       "urgent       2"
      ]
     },
     "execution_count": 94,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec8 = CountVectorizer(max_features=8)\n",
    "X_counts = vec8.fit_transform(toy_df[\"sms\"])\n",
    "pd.DataFrame(\n",
    "    data=X_counts.sum(axis=0).tolist()[0],\n",
    "    index=vec8.get_feature_names(),\n",
    "    columns=[\"counts\"],\n",
    ").sort_values(\"counts\", ascending=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>counts</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>to</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>you</th>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>free</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>have</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>prize</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>urgent</th>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mobiles</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>months</th>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         counts\n",
       "to            4\n",
       "you           4\n",
       "free          2\n",
       "have          2\n",
       "prize         2\n",
       "urgent        2\n",
       "mobiles       1\n",
       "months        1"
      ]
     },
     "execution_count": 95,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vec8_binary = CountVectorizer(binary=True, max_features=8)\n",
    "X_counts = vec8_binary.fit_transform(toy_df[\"sms\"])\n",
    "pd.DataFrame(\n",
    "    data=X_counts.sum(axis=0).tolist()[0],\n",
    "    index=vec8_binary.get_feature_names(),\n",
    "    columns=[\"counts\"],\n",
    ").sort_values(\"counts\", ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Preprocessing\n",
    "\n",
    "- Note that `CountVectorizer` is carrying out some preprocessing such as because of the default argument values \n",
    "    - Converting words to lowercase (`lowercase=True`)\n",
    "    - getting rid of punctuation and special characters (`token_pattern ='(?u)\\\\b\\\\w\\\\w+\\\\b'`)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 96,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "pipe = make_pipeline(CountVectorizer(), SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 97,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-11 {color: black;background-color: white;}#sk-container-id-11 pre{padding: 0;}#sk-container-id-11 div.sk-toggleable {background-color: white;}#sk-container-id-11 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-11 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-11 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-11 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-11 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-11 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-11 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-11 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-11 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-11 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-11 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-11 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-11 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-11 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-11 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-11 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-11 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-11 div.sk-item {position: relative;z-index: 1;}#sk-container-id-11 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-11 div.sk-item::before, #sk-container-id-11 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-11 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-11 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-11 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-11 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-11 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-11 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-11 div.sk-label-container {text-align: center;}#sk-container-id-11 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-11 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-11\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;countvectorizer&#x27;, CountVectorizer()), (&#x27;svc&#x27;, SVC())])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-96\" type=\"checkbox\" ><label for=\"sk-estimator-id-96\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;countvectorizer&#x27;, CountVectorizer()), (&#x27;svc&#x27;, SVC())])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-97\" type=\"checkbox\" ><label for=\"sk-estimator-id-97\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CountVectorizer</label><div class=\"sk-toggleable__content\"><pre>CountVectorizer()</pre></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-98\" type=\"checkbox\" ><label for=\"sk-estimator-id-98\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">SVC</label><div class=\"sk-toggleable__content\"><pre>SVC()</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('countvectorizer', CountVectorizer()), ('svc', SVC())])"
      ]
     },
     "execution_count": 97,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.fit(toy_df[\"sms\"], toy_df[\"target\"])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['spam', 'non spam', 'non spam', 'spam', 'spam', 'non spam'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 98,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.predict(toy_df[\"sms\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Is this a realistic representation of text data? \n",
    "\n",
    "- Of course this is not a great representation of language\n",
    "    - We are throwing out everything we know about language and losing a lot of information. \n",
    "    - It assumes that there is no syntax and compositional meaning in language.  \n",
    "- But it works surprisingly well for many tasks. \n",
    "- We will learn more expressive representations in the coming weeks. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Demo of incorporating text features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Recall that we had dropped `song_title` feature when we worked with the Spotify dataset. \n",
    "\n",
    "Let's try to include it in our pipeline and examine whether we get better results. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 99,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "spotify_df = pd.read_csv(\"data/spotify.csv\", index_col=0)\n",
    "X_spotify = spotify_df.drop(columns=[\"target\"])\n",
    "y_spotify = spotify_df[\"target\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 100,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    X_spotify, y_spotify, test_size=0.2, random_state=123\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 101,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1613, 15)"
      ]
     },
     "execution_count": 101,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 102,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>acousticness</th>\n",
       "      <th>danceability</th>\n",
       "      <th>duration_ms</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>key</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>mode</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>tempo</th>\n",
       "      <th>time_signature</th>\n",
       "      <th>valence</th>\n",
       "      <th>song_title</th>\n",
       "      <th>artist</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1505</th>\n",
       "      <td>0.004770</td>\n",
       "      <td>0.585</td>\n",
       "      <td>214740</td>\n",
       "      <td>0.614</td>\n",
       "      <td>0.000155</td>\n",
       "      <td>10</td>\n",
       "      <td>0.0762</td>\n",
       "      <td>-5.594</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0370</td>\n",
       "      <td>114.059</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.2730</td>\n",
       "      <td>Cool for the Summer</td>\n",
       "      <td>Demi Lovato</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>813</th>\n",
       "      <td>0.114000</td>\n",
       "      <td>0.665</td>\n",
       "      <td>216728</td>\n",
       "      <td>0.513</td>\n",
       "      <td>0.303000</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1220</td>\n",
       "      <td>-7.314</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3310</td>\n",
       "      <td>100.344</td>\n",
       "      <td>3.0</td>\n",
       "      <td>0.0373</td>\n",
       "      <td>Damn Son Where'd You Find This? (feat. Kelly Holiday) - Markus Maximus Remix</td>\n",
       "      <td>Markus Maximus</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>615</th>\n",
       "      <td>0.030200</td>\n",
       "      <td>0.798</td>\n",
       "      <td>216585</td>\n",
       "      <td>0.481</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7</td>\n",
       "      <td>0.1280</td>\n",
       "      <td>-10.488</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3140</td>\n",
       "      <td>127.136</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.6400</td>\n",
       "      <td>Trill Hoe</td>\n",
       "      <td>Western Tink</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>0.106000</td>\n",
       "      <td>0.912</td>\n",
       "      <td>194040</td>\n",
       "      <td>0.317</td>\n",
       "      <td>0.000208</td>\n",
       "      <td>6</td>\n",
       "      <td>0.0723</td>\n",
       "      <td>-12.719</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0378</td>\n",
       "      <td>99.346</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.9490</td>\n",
       "      <td>Who Is He (And What Is He to You?)</td>\n",
       "      <td>Bill Withers</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>0.021100</td>\n",
       "      <td>0.697</td>\n",
       "      <td>236456</td>\n",
       "      <td>0.905</td>\n",
       "      <td>0.893000</td>\n",
       "      <td>6</td>\n",
       "      <td>0.1190</td>\n",
       "      <td>-7.787</td>\n",
       "      <td>0</td>\n",
       "      <td>0.0339</td>\n",
       "      <td>119.977</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.3110</td>\n",
       "      <td>Acamar</td>\n",
       "      <td>Frankey</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012</th>\n",
       "      <td>0.001060</td>\n",
       "      <td>0.584</td>\n",
       "      <td>274404</td>\n",
       "      <td>0.932</td>\n",
       "      <td>0.002690</td>\n",
       "      <td>1</td>\n",
       "      <td>0.1290</td>\n",
       "      <td>-3.501</td>\n",
       "      <td>1</td>\n",
       "      <td>0.3330</td>\n",
       "      <td>74.976</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.2110</td>\n",
       "      <td>Like A Bitch - Kill The Noise Remix</td>\n",
       "      <td>Kill The Noise</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1346</th>\n",
       "      <td>0.000021</td>\n",
       "      <td>0.535</td>\n",
       "      <td>203500</td>\n",
       "      <td>0.974</td>\n",
       "      <td>0.000149</td>\n",
       "      <td>10</td>\n",
       "      <td>0.2630</td>\n",
       "      <td>-3.566</td>\n",
       "      <td>0</td>\n",
       "      <td>0.1720</td>\n",
       "      <td>116.956</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.4310</td>\n",
       "      <td>Flag of the Beast</td>\n",
       "      <td>Emmure</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1406</th>\n",
       "      <td>0.503000</td>\n",
       "      <td>0.410</td>\n",
       "      <td>256333</td>\n",
       "      <td>0.648</td>\n",
       "      <td>0.000000</td>\n",
       "      <td>7</td>\n",
       "      <td>0.2190</td>\n",
       "      <td>-4.469</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0362</td>\n",
       "      <td>60.391</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.3420</td>\n",
       "      <td>Don't You Cry For Me</td>\n",
       "      <td>Cobi</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1389</th>\n",
       "      <td>0.705000</td>\n",
       "      <td>0.894</td>\n",
       "      <td>222307</td>\n",
       "      <td>0.161</td>\n",
       "      <td>0.003300</td>\n",
       "      <td>4</td>\n",
       "      <td>0.3120</td>\n",
       "      <td>-14.311</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0880</td>\n",
       "      <td>104.968</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.8180</td>\n",
       "      <td>장가갈 수 있을까 Can I Get Married?</td>\n",
       "      <td>Coffeeboy</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1534</th>\n",
       "      <td>0.623000</td>\n",
       "      <td>0.470</td>\n",
       "      <td>394920</td>\n",
       "      <td>0.156</td>\n",
       "      <td>0.187000</td>\n",
       "      <td>2</td>\n",
       "      <td>0.1040</td>\n",
       "      <td>-17.036</td>\n",
       "      <td>1</td>\n",
       "      <td>0.0399</td>\n",
       "      <td>118.176</td>\n",
       "      <td>4.0</td>\n",
       "      <td>0.0591</td>\n",
       "      <td>Blue Ballad</td>\n",
       "      <td>Phil Woods</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1613 rows × 15 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      acousticness  danceability  duration_ms  energy  instrumentalness  key  \\\n",
       "1505      0.004770         0.585       214740   0.614          0.000155   10   \n",
       "813       0.114000         0.665       216728   0.513          0.303000    0   \n",
       "615       0.030200         0.798       216585   0.481          0.000000    7   \n",
       "319       0.106000         0.912       194040   0.317          0.000208    6   \n",
       "320       0.021100         0.697       236456   0.905          0.893000    6   \n",
       "...            ...           ...          ...     ...               ...  ...   \n",
       "2012      0.001060         0.584       274404   0.932          0.002690    1   \n",
       "1346      0.000021         0.535       203500   0.974          0.000149   10   \n",
       "1406      0.503000         0.410       256333   0.648          0.000000    7   \n",
       "1389      0.705000         0.894       222307   0.161          0.003300    4   \n",
       "1534      0.623000         0.470       394920   0.156          0.187000    2   \n",
       "\n",
       "      liveness  loudness  mode  speechiness    tempo  time_signature  valence  \\\n",
       "1505    0.0762    -5.594     0       0.0370  114.059             4.0   0.2730   \n",
       "813     0.1220    -7.314     1       0.3310  100.344             3.0   0.0373   \n",
       "615     0.1280   -10.488     1       0.3140  127.136             4.0   0.6400   \n",
       "319     0.0723   -12.719     0       0.0378   99.346             4.0   0.9490   \n",
       "320     0.1190    -7.787     0       0.0339  119.977             4.0   0.3110   \n",
       "...        ...       ...   ...          ...      ...             ...      ...   \n",
       "2012    0.1290    -3.501     1       0.3330   74.976             4.0   0.2110   \n",
       "1346    0.2630    -3.566     0       0.1720  116.956             4.0   0.4310   \n",
       "1406    0.2190    -4.469     1       0.0362   60.391             4.0   0.3420   \n",
       "1389    0.3120   -14.311     1       0.0880  104.968             4.0   0.8180   \n",
       "1534    0.1040   -17.036     1       0.0399  118.176             4.0   0.0591   \n",
       "\n",
       "                                                                        song_title  \\\n",
       "1505                                                           Cool for the Summer   \n",
       "813   Damn Son Where'd You Find This? (feat. Kelly Holiday) - Markus Maximus Remix   \n",
       "615                                                                      Trill Hoe   \n",
       "319                                             Who Is He (And What Is He to You?)   \n",
       "320                                                                         Acamar   \n",
       "...                                                                            ...   \n",
       "2012                                           Like A Bitch - Kill The Noise Remix   \n",
       "1346                                                             Flag of the Beast   \n",
       "1406                                                          Don't You Cry For Me   \n",
       "1389                                                  장가갈 수 있을까 Can I Get Married?   \n",
       "1534                                                                   Blue Ballad   \n",
       "\n",
       "              artist  \n",
       "1505     Demi Lovato  \n",
       "813   Markus Maximus  \n",
       "615     Western Tink  \n",
       "319     Bill Withers  \n",
       "320          Frankey  \n",
       "...              ...  \n",
       "2012  Kill The Noise  \n",
       "1346          Emmure  \n",
       "1406            Cobi  \n",
       "1389       Coffeeboy  \n",
       "1534      Phil Woods  \n",
       "\n",
       "[1613 rows x 15 columns]"
      ]
     },
     "execution_count": 102,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "Let's look at the distribution of values in the `song_title` column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 103,
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pyramids                                     2\n",
       "Look At Wrist                                2\n",
       "Baby                                         2\n",
       "The One                                      2\n",
       "Best Friend                                  2\n",
       "                                            ..\n",
       "City Of Dreams - Radio Edit                  1\n",
       "Face It                                      1\n",
       "The Winner Is - from Little Miss Sunshine    1\n",
       "History                                      1\n",
       "Blue Ballad                                  1\n",
       "Name: song_title, Length: 1579, dtype: int64"
      ]
     },
     "execution_count": 103,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train[\"song_title\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "tags": []
   },
   "source": [
    "- Most of the song titles are unique, which makes sense. \n",
    "- What would happen if we apply one-hot encoding to this feature? \n",
    "- Can we encode this as a text feature? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 104,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['acousticness', 'danceability', 'duration_ms', 'energy',\n",
       "       'instrumentalness', 'key', 'liveness', 'loudness', 'mode',\n",
       "       'speechiness', 'tempo', 'time_signature', 'valence', 'song_title',\n",
       "       'artist'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 104,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 105,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "numeric_features = [\n",
    "    \"acousticness\",\n",
    "    \"danceability\",\n",
    "    \"duration_ms\",\n",
    "    \"energy\",\n",
    "    \"instrumentalness\",\n",
    "    \"key\",\n",
    "    \"liveness\",\n",
    "    \"loudness\",\n",
    "    \"mode\",\n",
    "    \"speechiness\",\n",
    "    \"tempo\",\n",
    "    \"time_signature\",\n",
    "    \"valence\",\n",
    "]\n",
    "drop_features = ['artist']\n",
    "text_feature = \"song_title\"  # note that we are not creating a list here.\n",
    "\n",
    "preprocessor = make_column_transformer(\n",
    "    (StandardScaler(), numeric_features),\n",
    "    (CountVectorizer(max_features=2000, stop_words=\"english\"), text_feature),\n",
    "    (\"drop\", drop_features)\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "```{important}\n",
    "Note that unlike other feature types we are defining `text_feature` as a string and not as a list. \n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Visualizing the transformed data "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 106,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(1613, 1897)"
      ]
     },
     "execution_count": 106,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "transformed = preprocessor.fit_transform(X_train, y_train)\n",
    "transformed.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 107,
   "metadata": {},
   "outputs": [],
   "source": [
    "vocab = preprocessor.named_transformers_[\"countvectorizer\"].get_feature_names_out()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 108,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['000', '10', '100', ..., '있을까', '장가갈', '지금'], dtype=object)"
      ]
     },
     "execution_count": 108,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 109,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "column_names = numeric_features + vocab.tolist()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 110,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>acousticness</th>\n",
       "      <th>danceability</th>\n",
       "      <th>duration_ms</th>\n",
       "      <th>energy</th>\n",
       "      <th>instrumentalness</th>\n",
       "      <th>key</th>\n",
       "      <th>liveness</th>\n",
       "      <th>loudness</th>\n",
       "      <th>mode</th>\n",
       "      <th>speechiness</th>\n",
       "      <th>...</th>\n",
       "      <th>너와의</th>\n",
       "      <th>루시아</th>\n",
       "      <th>변명</th>\n",
       "      <th>여기</th>\n",
       "      <th>이곳에서</th>\n",
       "      <th>이대로</th>\n",
       "      <th>있어줘요</th>\n",
       "      <th>있을까</th>\n",
       "      <th>장가갈</th>\n",
       "      <th>지금</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1505</th>\n",
       "      <td>-0.697633</td>\n",
       "      <td>-0.194548</td>\n",
       "      <td>-0.398940</td>\n",
       "      <td>-0.318116</td>\n",
       "      <td>-0.492359</td>\n",
       "      <td>1.275623</td>\n",
       "      <td>-0.737898</td>\n",
       "      <td>0.395794</td>\n",
       "      <td>-1.280599</td>\n",
       "      <td>-0.617752</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>813</th>\n",
       "      <td>-0.276291</td>\n",
       "      <td>0.295726</td>\n",
       "      <td>-0.374443</td>\n",
       "      <td>-0.795552</td>\n",
       "      <td>0.598355</td>\n",
       "      <td>-1.487342</td>\n",
       "      <td>-0.438792</td>\n",
       "      <td>-0.052394</td>\n",
       "      <td>0.780884</td>\n",
       "      <td>2.728394</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>615</th>\n",
       "      <td>-0.599540</td>\n",
       "      <td>1.110806</td>\n",
       "      <td>-0.376205</td>\n",
       "      <td>-0.946819</td>\n",
       "      <td>-0.492917</td>\n",
       "      <td>0.446734</td>\n",
       "      <td>-0.399607</td>\n",
       "      <td>-0.879457</td>\n",
       "      <td>0.780884</td>\n",
       "      <td>2.534909</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>319</th>\n",
       "      <td>-0.307150</td>\n",
       "      <td>1.809445</td>\n",
       "      <td>-0.654016</td>\n",
       "      <td>-1.722063</td>\n",
       "      <td>-0.492168</td>\n",
       "      <td>0.170437</td>\n",
       "      <td>-0.763368</td>\n",
       "      <td>-1.460798</td>\n",
       "      <td>-1.280599</td>\n",
       "      <td>-0.608647</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>320</th>\n",
       "      <td>-0.634642</td>\n",
       "      <td>0.491835</td>\n",
       "      <td>-0.131344</td>\n",
       "      <td>1.057468</td>\n",
       "      <td>2.723273</td>\n",
       "      <td>0.170437</td>\n",
       "      <td>-0.458384</td>\n",
       "      <td>-0.175645</td>\n",
       "      <td>-1.280599</td>\n",
       "      <td>-0.653035</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2012</th>\n",
       "      <td>-0.711944</td>\n",
       "      <td>-0.200676</td>\n",
       "      <td>0.336272</td>\n",
       "      <td>1.185100</td>\n",
       "      <td>-0.483229</td>\n",
       "      <td>-1.211046</td>\n",
       "      <td>-0.393077</td>\n",
       "      <td>0.941176</td>\n",
       "      <td>0.780884</td>\n",
       "      <td>2.751157</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1346</th>\n",
       "      <td>-0.715953</td>\n",
       "      <td>-0.500969</td>\n",
       "      <td>-0.537445</td>\n",
       "      <td>1.383637</td>\n",
       "      <td>-0.492380</td>\n",
       "      <td>1.275623</td>\n",
       "      <td>0.482038</td>\n",
       "      <td>0.924239</td>\n",
       "      <td>-1.280599</td>\n",
       "      <td>0.918743</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1406</th>\n",
       "      <td>1.224228</td>\n",
       "      <td>-1.267021</td>\n",
       "      <td>0.113591</td>\n",
       "      <td>-0.157395</td>\n",
       "      <td>-0.492917</td>\n",
       "      <td>0.446734</td>\n",
       "      <td>0.194687</td>\n",
       "      <td>0.688940</td>\n",
       "      <td>0.780884</td>\n",
       "      <td>-0.626857</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1389</th>\n",
       "      <td>2.003419</td>\n",
       "      <td>1.699134</td>\n",
       "      <td>-0.305695</td>\n",
       "      <td>-2.459489</td>\n",
       "      <td>-0.481032</td>\n",
       "      <td>-0.382156</td>\n",
       "      <td>0.802042</td>\n",
       "      <td>-1.875632</td>\n",
       "      <td>0.780884</td>\n",
       "      <td>-0.037298</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1534</th>\n",
       "      <td>1.687114</td>\n",
       "      <td>-0.899316</td>\n",
       "      <td>1.821334</td>\n",
       "      <td>-2.483125</td>\n",
       "      <td>0.180574</td>\n",
       "      <td>-0.934749</td>\n",
       "      <td>-0.556344</td>\n",
       "      <td>-2.585697</td>\n",
       "      <td>0.780884</td>\n",
       "      <td>-0.584746</td>\n",
       "      <td>...</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>1613 rows × 1897 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      acousticness  danceability  duration_ms    energy  instrumentalness  \\\n",
       "1505     -0.697633     -0.194548    -0.398940 -0.318116         -0.492359   \n",
       "813      -0.276291      0.295726    -0.374443 -0.795552          0.598355   \n",
       "615      -0.599540      1.110806    -0.376205 -0.946819         -0.492917   \n",
       "319      -0.307150      1.809445    -0.654016 -1.722063         -0.492168   \n",
       "320      -0.634642      0.491835    -0.131344  1.057468          2.723273   \n",
       "...            ...           ...          ...       ...               ...   \n",
       "2012     -0.711944     -0.200676     0.336272  1.185100         -0.483229   \n",
       "1346     -0.715953     -0.500969    -0.537445  1.383637         -0.492380   \n",
       "1406      1.224228     -1.267021     0.113591 -0.157395         -0.492917   \n",
       "1389      2.003419      1.699134    -0.305695 -2.459489         -0.481032   \n",
       "1534      1.687114     -0.899316     1.821334 -2.483125          0.180574   \n",
       "\n",
       "           key  liveness  loudness      mode  speechiness  ...  너와의  루시아   변명  \\\n",
       "1505  1.275623 -0.737898  0.395794 -1.280599    -0.617752  ...  0.0  0.0  0.0   \n",
       "813  -1.487342 -0.438792 -0.052394  0.780884     2.728394  ...  0.0  0.0  0.0   \n",
       "615   0.446734 -0.399607 -0.879457  0.780884     2.534909  ...  0.0  0.0  0.0   \n",
       "319   0.170437 -0.763368 -1.460798 -1.280599    -0.608647  ...  0.0  0.0  0.0   \n",
       "320   0.170437 -0.458384 -0.175645 -1.280599    -0.653035  ...  0.0  0.0  0.0   \n",
       "...        ...       ...       ...       ...          ...  ...  ...  ...  ...   \n",
       "2012 -1.211046 -0.393077  0.941176  0.780884     2.751157  ...  0.0  0.0  0.0   \n",
       "1346  1.275623  0.482038  0.924239 -1.280599     0.918743  ...  0.0  0.0  0.0   \n",
       "1406  0.446734  0.194687  0.688940  0.780884    -0.626857  ...  0.0  0.0  0.0   \n",
       "1389 -0.382156  0.802042 -1.875632  0.780884    -0.037298  ...  0.0  0.0  0.0   \n",
       "1534 -0.934749 -0.556344 -2.585697  0.780884    -0.584746  ...  0.0  0.0  0.0   \n",
       "\n",
       "       여기  이곳에서  이대로  있어줘요  있을까  장가갈   지금  \n",
       "1505  0.0   0.0  0.0   0.0  0.0  0.0  0.0  \n",
       "813   0.0   0.0  0.0   0.0  0.0  0.0  0.0  \n",
       "615   0.0   0.0  0.0   0.0  0.0  0.0  0.0  \n",
       "319   0.0   0.0  0.0   0.0  0.0  0.0  0.0  \n",
       "320   0.0   0.0  0.0   0.0  0.0  0.0  0.0  \n",
       "...   ...   ...  ...   ...  ...  ...  ...  \n",
       "2012  0.0   0.0  0.0   0.0  0.0  0.0  0.0  \n",
       "1346  0.0   0.0  0.0   0.0  0.0  0.0  0.0  \n",
       "1406  0.0   0.0  0.0   0.0  0.0  0.0  0.0  \n",
       "1389  0.0   0.0  0.0   0.0  1.0  1.0  0.0  \n",
       "1534  0.0   0.0  0.0   0.0  0.0  0.0  0.0  \n",
       "\n",
       "[1613 rows x 1897 columns]"
      ]
     },
     "execution_count": 110,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(transformed.toarray(), columns=column_names, index=X_train.index)\n",
    "df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Visualizing the vocabulary "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['000', '10', '100', '10cm', '11', '112', '12', '1208', '144', '18'],\n",
       "      dtype=object)"
      ]
     },
     "execution_count": 111,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[0:10]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 112,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['duele', 'duet', 'duke', 'dustland', 'dutchie', 'dynamite',\n",
       "       'earth', 'easy', 'eazy', 'echelon'], dtype=object)"
      ]
     },
     "execution_count": 112,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[500:510]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 113,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['wide', 'wifey', 'wild', 'wildcard', 'wildfire', 'wiley',\n",
       "       'willing', 'win', 'wind', 'window'], dtype=object)"
      ]
     },
     "execution_count": 113,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[1800:1810]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 114,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array(['000', 'ap', 'blind', 'cha', 'dallask', 'duele', 'flashlight',\n",
       "       'grace', 'icarus', 'lafa', 'making', 'neck', 'pharaohs', 'redeem',\n",
       "       'seeb', 'soundtrack', 'talons', 'unanswered', 'wide'], dtype=object)"
      ]
     },
     "execution_count": 114,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "vocab[0::100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's find songs containing the word _earth_ in them. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "506"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "earth_index_vocab = np.where(vocab == \"earth\")[0][0]\n",
    "earth_index_vocab"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "519"
      ]
     },
     "execution_count": 116,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "earth_index_in_df = len(numeric_features) + earth_index_vocab\n",
    "earth_index_in_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>dutchie</th>\n",
       "      <th>dynamite</th>\n",
       "      <th>earth</th>\n",
       "      <th>easy</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1851</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1948</th>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      dutchie  dynamite  earth  easy\n",
       "1851      0.0       0.0    1.0   0.0\n",
       "1948      0.0       0.0    1.0   0.0"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "earth_songs = df[df.iloc[:, earth_index_in_df] == 1]\n",
    "earth_songs.iloc[:, earth_index_in_df - 2 : earth_index_in_df + 2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 118,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Int64Index([1851, 1948], dtype='int64')"
      ]
     },
     "execution_count": 118,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "earth_songs.index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 119,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1851             Softest Place On Earth\n",
       "1948    Earth Song - Remastered Version\n",
       "Name: song_title, dtype: object"
      ]
     },
     "execution_count": 119,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.loc[earth_songs.index][\"song_title\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Model building "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a pipeline using SVC. \n",
    "- SVC works well with sparse features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = make_pipeline(preprocessor, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.048014</td>\n",
       "      <td>0.010126</td>\n",
       "      <td>0.749226</td>\n",
       "      <td>0.867442</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.046795</td>\n",
       "      <td>0.009712</td>\n",
       "      <td>0.758514</td>\n",
       "      <td>0.862016</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.046193</td>\n",
       "      <td>0.009659</td>\n",
       "      <td>0.712074</td>\n",
       "      <td>0.865116</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.046260</td>\n",
       "      <td>0.009624</td>\n",
       "      <td>0.739130</td>\n",
       "      <td>0.856700</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.046955</td>\n",
       "      <td>0.010129</td>\n",
       "      <td>0.729814</td>\n",
       "      <td>0.855151</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.048014    0.010126    0.749226     0.867442\n",
       "1  0.046795    0.009712    0.758514     0.862016\n",
       "2  0.046193    0.009659    0.712074     0.865116\n",
       "3  0.046260    0.009624    0.739130     0.856700\n",
       "4  0.046955    0.010129    0.729814     0.855151"
      ]
     },
     "execution_count": 121,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = pd.DataFrame(cross_validate(pipe, X_train, y_train, return_train_score=True))\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.046844\n",
       "score_time     0.009850\n",
       "test_score     0.737752\n",
       "train_score    0.861285\n",
       "dtype: float64"
      ]
     },
     "execution_count": 122,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Is our CV improving after incorporating this feature?\n",
    "Let's examine what numbers we get when we don't include it. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe_num = make_pipeline(StandardScaler(), SVC())\n",
    "\n",
    "X_train_num = X_train.drop(columns=[\"song_title\", 'artist'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.030469</td>\n",
       "      <td>0.010427</td>\n",
       "      <td>0.758514</td>\n",
       "      <td>0.810078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.030952</td>\n",
       "      <td>0.010600</td>\n",
       "      <td>0.749226</td>\n",
       "      <td>0.810078</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.029415</td>\n",
       "      <td>0.009717</td>\n",
       "      <td>0.705882</td>\n",
       "      <td>0.821705</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.028925</td>\n",
       "      <td>0.009640</td>\n",
       "      <td>0.748447</td>\n",
       "      <td>0.815647</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.029485</td>\n",
       "      <td>0.009691</td>\n",
       "      <td>0.732919</td>\n",
       "      <td>0.812548</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.030469    0.010427    0.758514     0.810078\n",
       "1  0.030952    0.010600    0.749226     0.810078\n",
       "2  0.029415    0.009717    0.705882     0.821705\n",
       "3  0.028925    0.009640    0.748447     0.815647\n",
       "4  0.029485    0.009691    0.732919     0.812548"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = pd.DataFrame(\n",
    "    cross_validate(pipe_num, X_train_num, y_train, return_train_score=True)\n",
    ")\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.029849\n",
       "score_time     0.010015\n",
       "test_score     0.738998\n",
       "train_score    0.814011\n",
       "dtype: float64"
      ]
     },
     "execution_count": 125,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Not a big difference in the results. \n",
    "- Seems like there is more overfitting when we included the `song_title` feature. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- What about the `artist` column?\n",
    "- Does it make sense to apply BOW encoding to it? \n",
    "- Let's look at the distribution of values in the `artist` column. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Drake              14\n",
       "Disclosure         12\n",
       "Rick Ross          11\n",
       "WALK THE MOON      10\n",
       "Crystal Castles     8\n",
       "                   ..\n",
       "Classixx            1\n",
       "Jordan Feliz        1\n",
       "Travis Hayes        1\n",
       "The Silvertones     1\n",
       "Phil Woods          1\n",
       "Name: artist, Length: 1131, dtype: int64"
      ]
     },
     "execution_count": 126,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train['artist'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Drake              14\n",
       "Disclosure         12\n",
       "Rick Ross          11\n",
       "WALK THE MOON      10\n",
       "Crystal Castles     8\n",
       "Big Time Rush       8\n",
       "FIDLAR              8\n",
       "Fall Out Boy        8\n",
       "Demi Lovato         7\n",
       "Kanye West          7\n",
       "Kina Grannis        7\n",
       "Backstreet Boys     7\n",
       "Beach House         6\n",
       "Young Thug          6\n",
       "*NSYNC              6\n",
       "Name: artist, dtype: int64"
      ]
     },
     "execution_count": 127,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "most_frequent = X_train[\"artist\"].value_counts().iloc[:15]\n",
    "most_frequent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    },
    "tags": []
   },
   "source": [
    "- We have many unique artists. Probably it's not worth to create a \"other\" category here. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "numeric_features = [\n",
    "    \"acousticness\",\n",
    "    \"danceability\",\n",
    "    \"duration_ms\",\n",
    "    \"energy\",\n",
    "    \"instrumentalness\",\n",
    "    \"key\",\n",
    "    \"liveness\",\n",
    "    \"loudness\",\n",
    "    \"mode\",\n",
    "    \"speechiness\",\n",
    "    \"tempo\",\n",
    "    \"time_signature\",\n",
    "    \"valence\",\n",
    "]\n",
    "categorical_features = ['artist']\n",
    "text_feature = \"song_title\"  # note that we are not creating a list here.\n",
    "\n",
    "preprocessor_artist = make_column_transformer(\n",
    "    (StandardScaler(), numeric_features),\n",
    "    (OneHotEncoder(sparse=False, dtype=int, handle_unknown=\"ignore\", categories=[most_frequent.index.values]), categorical_features),\n",
    "    (CountVectorizer(max_features=2000, stop_words=\"english\"), text_feature),\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = make_pipeline(preprocessor_artist, SVC())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.046828</td>\n",
       "      <td>0.010173</td>\n",
       "      <td>0.755418</td>\n",
       "      <td>0.870543</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.045806</td>\n",
       "      <td>0.010004</td>\n",
       "      <td>0.761610</td>\n",
       "      <td>0.864341</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.046960</td>\n",
       "      <td>0.010174</td>\n",
       "      <td>0.712074</td>\n",
       "      <td>0.868217</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.047701</td>\n",
       "      <td>0.010743</td>\n",
       "      <td>0.742236</td>\n",
       "      <td>0.865995</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.050811</td>\n",
       "      <td>0.010520</td>\n",
       "      <td>0.732919</td>\n",
       "      <td>0.858249</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   fit_time  score_time  test_score  train_score\n",
       "0  0.046828    0.010173    0.755418     0.870543\n",
       "1  0.045806    0.010004    0.761610     0.864341\n",
       "2  0.046960    0.010174    0.712074     0.868217\n",
       "3  0.047701    0.010743    0.742236     0.865995\n",
       "4  0.050811    0.010520    0.732919     0.858249"
      ]
     },
     "execution_count": 130,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results = pd.DataFrame(cross_validate(pipe, X_train, y_train, return_train_score=True))\n",
    "results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.047621\n",
       "score_time     0.010323\n",
       "test_score     0.740851\n",
       "train_score    0.865469\n",
       "dtype: float64"
      ]
     },
     "execution_count": 131,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "results.mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Tiny bit improvement in the mean CV scores but we are still overfitting. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## ❓❓ Questions for you "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    },
    "tags": []
   },
   "source": [
    "### (iClicker) Exercise 6.2 \n",
    "\n",
    "**iClicker cloud join link: https://join.iclicker.com/3DP5H**\n",
    "\n",
    "**Select all of the following statements which are TRUE.**\n",
    "\n",
    "- (A) `handle_unknown=\"ignore\"` would treat all unknown categories equally. \n",
    "- (B) As you increase the value for `max_features` hyperparameter of `CountVectorizer` the training score is likely to go up. \n",
    "- (C) Suppose you are encoding text data using `CountVectorizer`. If you encounter a word in the validation or the test split that's not available in the training data, we'll get an error. \n",
    "- (D) In the code below, inside `cross_validate`, each fold might have slightly different number of features (columns) in the fold.\n",
    "\n",
    "```\n",
    "pipe = (CountVectorizer(), SVC())\n",
    "cross_validate(pipe, X_train, y_train)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## What did we learn today?\n",
    "\n",
    "- Motivation to use `ColumnTransformer`\n",
    "- `ColumnTransformer` syntax\n",
    "- Defining transformers with multiple transformations\n",
    "- How to visualize transformed features in a dataframe \n",
    "- More on ordinal features \n",
    "- Different arguments `OneHotEncoder`\n",
    "    - `handle_unknow=\"ignore\"`\n",
    "    - `if_binary`\n",
    "- Dealing with text features\n",
    "    - Bag of words representation: `CountVectorizer`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/eva-talksoon.png)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}