lectures/13_feature-engineering-selection.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "![](img/330-banner.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "# Lecture 13: Feature engineering and feature selection \n",
    "\n",
    "UBC 2022-23\n",
    "\n",
    "Instructor: Varada Kolhatkar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Imports"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import sys\n",
    "\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import numpy.random as npr\n",
    "import pandas as pd\n",
    "from sklearn.compose import (\n",
    "    ColumnTransformer,\n",
    "    TransformedTargetRegressor,\n",
    "    make_column_transformer,\n",
    ")\n",
    "from sklearn.dummy import DummyClassifier, DummyRegressor\n",
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.impute import SimpleImputer\n",
    "from sklearn.linear_model import LinearRegression, LogisticRegression, Ridge, RidgeCV\n",
    "from sklearn.metrics import make_scorer, mean_squared_error, r2_score\n",
    "from sklearn.model_selection import cross_val_score, cross_validate, train_test_split\n",
    "from sklearn.pipeline import Pipeline, make_pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder, StandardScaler\n",
    "from sklearn.svm import SVC"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "## Learning outcomes \n",
    "\n",
    "From this lecture, students are expected to be able to:\n",
    "\n",
    "- Explain what feature engineering is and the importance of feature engineering in building machine learning models.  \n",
    "- Carry out preliminary feature engineering on text data. \n",
    "- Explain the general concept of feature selection. \n",
    "- Discuss and compare different feature selection methods at a high level. \n",
    "- Use `sklearn`'s implementation of recursive feature elimination (`RFE`) and forward and backward selection (`SequentialFeatureSelector`). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Feature engineering: Motivation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is feature engineering?  \n",
    "\n",
    "- Better features: more flexibility, higher score, we can get by with simple and more interpretable models. \n",
    "- If your features, i.e., representation is bad, whatever fancier model you build is not going to help.\n",
    "\n",
    "<blockquote>\n",
    "<b>Feature engineering</b> is the process of transforming raw data into features that better represent the underlying problem to the predictive models, resulting in improved model accuracy on unseen data.<br> \n",
    "- Jason Brownlee    \n",
    "</blockquote>    \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Some quotes on feature engineering \n",
    "\n",
    "A quote by Pedro Domingos [A Few Useful Things to Know About Machine Learning](https://homes.cs.washington.edu/~pedrod/papers/cacm12.pdf)\n",
    "\n",
    "<blockquote>\n",
    "... At the end of the day, some machine learning projects succeed and some fail. What makes the difference? Easily the most important factor is the features used. \n",
    "</blockquote>\n",
    "\n",
    "\n",
    "A quote by Andrew Ng, [Machine Learning and AI via Brain simulations](https://ai.stanford.edu/~ang/slides/DeepLearning-Mar2013.pptx)\n",
    "\n",
    "<blockquote>\n",
    "Coming up with features is difficult, time-consuming, requires expert knowledge. \"Applied machine learning\" is basically feature engineering.\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Better features usually help more than a better model.\n",
    "- Good features would ideally:\n",
    "    - capture most important aspects of the problem\n",
    "    - allow learning with few examples \n",
    "    - generalize to new scenarios.\n",
    "\n",
    "- There is a trade-off between simple and expressive features:\n",
    "    - With simple features overfitting risk is low, but scores might be low.\n",
    "    - With complicated features scores can be high, but so is overfitting risk."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### The best features may be dependent on the model you use.\n",
    "\n",
    "- Examples:\n",
    "    - For counting-based methods like decision trees separate relevant groups of variable values\n",
    "        - Discretization makes sense \n",
    "    - For distance-based methods like KNN, we want different class labels to be \"far\".\n",
    "        - Standardization \n",
    "    - For regression-based methods like linear regression, we want targets to have a linear dependency on features."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Domain-specific transformations\n",
    "\n",
    "In some domains there are natural transformations to do:\n",
    "- Spectrograms (sound data)\n",
    "- Wavelets (image data)\n",
    "- Convolutions \n",
    "\n",
    "![](img/spectogram.png)\n",
    "\n",
    "<!-- <img src=\"img/spectogram.png\" width=\"800\" height=\"800\"> -->\n",
    "\n",
    "[Source](https://en.wikipedia.org/wiki/Spectrogram)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "In this lecture, I'll show you an example of feature engineering on text data. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Feature engineering for text data "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We will be using [Covid tweets](https://www.kaggle.com/code/kerneler/starter-covid-19-nlp-text-d3a3baa6-e/data) dataset for this. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Negative              1041\n",
       "Positive               947\n",
       "Neutral                619\n",
       "Extremely Positive     599\n",
       "Extremely Negative     592\n",
       "Name: Sentiment, dtype: int64"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv('data/Corona_NLP_test.csv')\n",
    "df['Sentiment'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_df, test_df = train_test_split(df, test_size=0.2, random_state=123)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserName</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>Location</th>\n",
       "      <th>TweetAt</th>\n",
       "      <th>OriginalTweet</th>\n",
       "      <th>Sentiment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1927</th>\n",
       "      <td>1928</td>\n",
       "      <td>46880</td>\n",
       "      <td>Seattle, WA</td>\n",
       "      <td>13-03-2020</td>\n",
       "      <td>While I don't like all of Amazon's choices, to...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1068</th>\n",
       "      <td>1069</td>\n",
       "      <td>46021</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13-03-2020</td>\n",
       "      <td>Me: shit buckets, its time to do the weekly s...</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>803</th>\n",
       "      <td>804</td>\n",
       "      <td>45756</td>\n",
       "      <td>The Outer Limits</td>\n",
       "      <td>12-03-2020</td>\n",
       "      <td>@SecPompeo @realDonaldTrump You mean the plan ...</td>\n",
       "      <td>Neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2846</th>\n",
       "      <td>2847</td>\n",
       "      <td>47799</td>\n",
       "      <td>Flagstaff, AZ</td>\n",
       "      <td>15-03-2020</td>\n",
       "      <td>@lauvagrande People who are sick arent panic ...</td>\n",
       "      <td>Extremely Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3768</th>\n",
       "      <td>3769</td>\n",
       "      <td>48721</td>\n",
       "      <td>Montreal, Canada</td>\n",
       "      <td>16-03-2020</td>\n",
       "      <td>Coronavirus Panic: Toilet Paper Is the People...</td>\n",
       "      <td>Negative</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1122</th>\n",
       "      <td>1123</td>\n",
       "      <td>46075</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13-03-2020</td>\n",
       "      <td>Photos of our local grocery store shelveswher...</td>\n",
       "      <td>Extremely Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1346</th>\n",
       "      <td>1347</td>\n",
       "      <td>46299</td>\n",
       "      <td>Toronto</td>\n",
       "      <td>13-03-2020</td>\n",
       "      <td>Just went to the the grocery store (Highland F...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3454</th>\n",
       "      <td>3455</td>\n",
       "      <td>48407</td>\n",
       "      <td>Houston, TX</td>\n",
       "      <td>16-03-2020</td>\n",
       "      <td>Real talk though. Am I the only one spending h...</td>\n",
       "      <td>Neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3437</th>\n",
       "      <td>3438</td>\n",
       "      <td>48390</td>\n",
       "      <td>Washington, DC</td>\n",
       "      <td>16-03-2020</td>\n",
       "      <td>The supermarket business is booming! #COVID2019</td>\n",
       "      <td>Neutral</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3582</th>\n",
       "      <td>3583</td>\n",
       "      <td>48535</td>\n",
       "      <td>St James' Park, Newcastle</td>\n",
       "      <td>16-03-2020</td>\n",
       "      <td>Evening All Here s the story on the and the im...</td>\n",
       "      <td>Positive</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3038 rows × 6 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "      UserName  ScreenName                    Location     TweetAt  \\\n",
       "1927      1928       46880                 Seattle, WA  13-03-2020   \n",
       "1068      1069       46021                         NaN  13-03-2020   \n",
       "803        804       45756            The Outer Limits  12-03-2020   \n",
       "2846      2847       47799               Flagstaff, AZ  15-03-2020   \n",
       "3768      3769       48721            Montreal, Canada  16-03-2020   \n",
       "...        ...         ...                         ...         ...   \n",
       "1122      1123       46075                         NaN  13-03-2020   \n",
       "1346      1347       46299                     Toronto  13-03-2020   \n",
       "3454      3455       48407                 Houston, TX  16-03-2020   \n",
       "3437      3438       48390              Washington, DC  16-03-2020   \n",
       "3582      3583       48535  St James' Park, Newcastle   16-03-2020   \n",
       "\n",
       "                                          OriginalTweet           Sentiment  \n",
       "1927  While I don't like all of Amazon's choices, to...            Positive  \n",
       "1068  Me: shit buckets, its time to do the weekly s...            Negative  \n",
       "803   @SecPompeo @realDonaldTrump You mean the plan ...             Neutral  \n",
       "2846  @lauvagrande People who are sick arent panic ...  Extremely Negative  \n",
       "3768  Coronavirus Panic: Toilet Paper Is the People...            Negative  \n",
       "...                                                 ...                 ...  \n",
       "1122  Photos of our local grocery store shelveswher...  Extremely Positive  \n",
       "1346  Just went to the the grocery store (Highland F...            Positive  \n",
       "3454  Real talk though. Am I the only one spending h...             Neutral  \n",
       "3437    The supermarket business is booming! #COVID2019             Neutral  \n",
       "3582  Evening All Here s the story on the and the im...            Positive  \n",
       "\n",
       "[3038 rows x 6 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Index(['UserName', 'ScreenName', 'Location', 'TweetAt', 'OriginalTweet',\n",
       "       'Sentiment'],\n",
       "      dtype='object')"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "United States                     63\n",
       "London, England                   37\n",
       "Los Angeles, CA                   30\n",
       "New York, NY                      29\n",
       "Washington, DC                    29\n",
       "                                  ..\n",
       "Suburb of Chicago                  1\n",
       "philippines                        1\n",
       "Dont ask for freedom, take it.     1\n",
       "Windsor Heights, IA                1\n",
       "St James' Park, Newcastle          1\n",
       "Name: Location, Length: 1441, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df['Location'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, y_train = train_df[['OriginalTweet', 'Location']], train_df['Sentiment']\n",
    "X_test, y_test = test_df[['OriginalTweet', 'Location']], test_df['Sentiment']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Negative              852\n",
       "Positive              743\n",
       "Neutral               501\n",
       "Extremely Negative    472\n",
       "Extremely Positive    470\n",
       "Name: Sentiment, dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "scoring_metrics = 'accuracy'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "results = {}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def mean_std_cross_val_scores(model, X_train, y_train, **kwargs):\n",
    "    \"\"\"\n",
    "    Returns mean and std of cross validation\n",
    "\n",
    "    Parameters\n",
    "    ----------\n",
    "    model :\n",
    "        scikit-learn model\n",
    "    X_train : numpy array or pandas DataFrame\n",
    "        X in the training data\n",
    "    y_train :\n",
    "        y in the training data\n",
    "\n",
    "    Returns\n",
    "    ----------\n",
    "        pandas Series with mean scores from cross_validation\n",
    "    \"\"\"\n",
    "\n",
    "    scores = cross_validate(model, X_train, y_train, **kwargs)\n",
    "\n",
    "    mean_scores = pd.DataFrame(scores).mean()\n",
    "    std_scores = pd.DataFrame(scores).std()\n",
    "    out_col = []\n",
    "\n",
    "    for i in range(len(mean_scores)):\n",
    "        out_col.append((f\"%0.3f (+/- %0.3f)\" % (mean_scores[i], std_scores[i])))\n",
    "\n",
    "    return pd.Series(data=out_col, index=mean_scores.index)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Dummy classifier"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>0.280 (+/- 0.001)</td>\n",
       "      <td>0.280 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                fit_time         score_time         test_score  \\\n",
       "dummy  0.001 (+/- 0.000)  0.000 (+/- 0.000)  0.280 (+/- 0.001)   \n",
       "\n",
       "             train_score  \n",
       "dummy  0.280 (+/- 0.000)  "
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dummy = DummyClassifier()\n",
    "results[\"dummy\"] = mean_std_cross_val_scores(\n",
    "    dummy, X_train, y_train, return_train_score=True, scoring=scoring_metrics\n",
    ")\n",
    "pd.DataFrame(results).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Bag-of-words model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>0.280 (+/- 0.001)</td>\n",
       "      <td>0.280 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.442 (+/- 0.027)</td>\n",
       "      <td>0.007 (+/- 0.000)</td>\n",
       "      <td>0.413 (+/- 0.011)</td>\n",
       "      <td>0.999 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              fit_time         score_time         test_score  \\\n",
       "dummy                0.001 (+/- 0.000)  0.000 (+/- 0.000)  0.280 (+/- 0.001)   \n",
       "logistic regression  0.442 (+/- 0.027)  0.007 (+/- 0.000)  0.413 (+/- 0.011)   \n",
       "\n",
       "                           train_score  \n",
       "dummy                0.280 (+/- 0.000)  \n",
       "logistic regression  0.999 (+/- 0.000)  "
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "pipe = make_pipeline(CountVectorizer(stop_words='english'), \n",
    "                     LogisticRegression(max_iter=1000))\n",
    "results[\"logistic regression\"] = mean_std_cross_val_scores(\n",
    "    pipe, X_train['OriginalTweet'], y_train, return_train_score=True, scoring=scoring_metrics\n",
    ")\n",
    "pd.DataFrame(results).T"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Is it possible to further improve the scores?\n",
    "\n",
    "- How about adding new features based on our intuitions? Let's extract our own features that might be useful for this prediction task. In other words, let's carry out **feature engineering**. \n",
    "\n",
    "- The code below adds some very basic length-related and sentiment features. We will be using a popular library called `nltk` for this exercise. If you have successfully created the course `conda` environment on your machine, you should already have this package in the environment.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- How do we extract interesting information from text?\n",
    "- We use **pre-trained models**! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- A couple of popular libraries which include such pre-trained models. \n",
    "- `nltk`\n",
    "```\n",
    "conda install -c anaconda nltk \n",
    "```        \n",
    "- spaCy\n",
    "```\n",
    "conda install -c conda-forge spacy\n",
    "```\n",
    "\n",
    "For emoji support: \n",
    "```\n",
    "pip install spacymoji\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- You also need to download the language model which contains all the pre-trained models. For that run the following in your course `conda` environment or here.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "import spacy\n",
    "\n",
    "# !python -m spacy download en_core_web_md"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import nltk\n",
    "\n",
    "#nltk.download(\"punkt\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package vader_lexicon to\n",
      "[nltk_data]     /Users/kvarada/nltk_data...\n",
      "[nltk_data]   Package vader_lexicon is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to /Users/kvarada/nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    }
   ],
   "source": [
    "nltk.download(\"vader_lexicon\")\n",
    "nltk.download(\"punkt\")\n",
    "from nltk.sentiment.vader import SentimentIntensityAnalyzer\n",
    "\n",
    "sid = SentimentIntensityAnalyzer()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'neg': 0.0, 'neu': 0.368, 'pos': 0.632, 'compound': 0.8225}\n"
     ]
    }
   ],
   "source": [
    "s = \"CPSC 330 students are smart, sweet, and funny.\"\n",
    "print(sid.polarity_scores(s))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'neg': 0.249, 'neu': 0.751, 'pos': 0.0, 'compound': -0.5106}\n"
     ]
    }
   ],
   "source": [
    "s = \"CPSC 330 students are tired because of all the hard work they have been doing.\"\n",
    "print(sid.polarity_scores(s))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### [spaCy](https://spacy.io/) \n",
    "\n",
    "A useful package for text processing and feature extraction\n",
    "- Active development: https://github.com/explosion/spaCy\n",
    "- Interactive lessons by Ines Montani: https://course.spacy.io/en/\n",
    "- Good documentation, easy to use, and customizable. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "import en_core_web_md  # pre-trained model\n",
    "import spacy\n",
    "\n",
    "nlp = en_core_web_md.load()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "sample_text = \"\"\"Dolly Parton is a gift to us all. \n",
    "From writing all-time great songs like “Jolene” and “I Will Always Love You”, \n",
    "to great performances in films like 9 to 5, to helping fund a COVID-19 vaccine, \n",
    "she’s given us so much. Now, Netflix bring us Dolly Parton’s Christmas on the Square, \n",
    "an original musical that stars Christine Baranski as a Scrooge-like landowner \n",
    "who threatens to evict an entire town on Christmas Eve to make room for a new mall. \n",
    "Directed and choreographed by the legendary Debbie Allen and counting Jennifer Lewis \n",
    "and Parton herself amongst its cast, Christmas on the Square seems like the perfect movie\n",
    "to save Christmas 2020. 😻 👍🏿\"\"\"\n",
    "\n",
    "# [Adapted from here.](https://thepopbreak.com/2020/11/22/dolly-partons-christmas-on-the-square-review-not-quite-a-christmas-miracle/)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "Spacy extracts all interesting information from text with this call."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "doc = nlp(sample_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's look at part-of-speech tags. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[(Dolly, 'PROPN'), (Parton, 'PROPN'), (is, 'AUX'), (a, 'DET'), (gift, 'NOUN'), (to, 'ADP'), (us, 'PRON'), (all, 'PRON'), (., 'PUNCT'), (\n",
      ", 'SPACE'), (From, 'ADP'), (writing, 'VERB'), (all, 'DET'), (-, 'PUNCT'), (time, 'NOUN'), (great, 'ADJ'), (songs, 'NOUN'), (like, 'ADP'), (“, 'PUNCT'), (Jolene, 'PROPN')]\n"
     ]
    }
   ],
   "source": [
    "print([(token, token.pos_) for token in doc][:20])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Often we want to know who did what to whom. \n",
    "- **Named entities** give you this information.  \n",
    "- What are named entities in the text? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Named entities:\n",
      " [('Dolly Parton', 'PERSON'), ('Jolene', 'WORK_OF_ART'), ('I Will Always Love You', 'WORK_OF_ART'), ('9', 'CARDINAL'), ('Netflix', 'ORG'), ('Dolly Parton’s', 'PERSON'), ('Christmas', 'DATE'), ('Square', 'FAC'), ('Christine Baranski', 'PERSON'), ('Christmas Eve', 'DATE'), ('Debbie Allen', 'PERSON'), ('Jennifer Lewis', 'PERSON'), ('Parton', 'PERSON'), ('Christmas', 'DATE'), ('Square', 'FAC'), ('Christmas 2020', 'DATE'), ('😻', 'GPE')]\n",
      "\n",
      "ORG means:  Companies, agencies, institutions, etc.\n",
      "\n",
      "PERSON means:  People, including fictional\n",
      "\n",
      "DATE means:  Absolute or relative dates or periods\n"
     ]
    }
   ],
   "source": [
    "print(\"Named entities:\\n\", [(ent.text, ent.label_) for ent in doc.ents])\n",
    "print(\"\\nORG means: \", spacy.explain(\"ORG\"))\n",
    "print(\"\\nPERSON means: \", spacy.explain(\"PERSON\"))\n",
    "print(\"\\nDATE means: \", spacy.explain(\"DATE\"))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">\n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Dolly Parton\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " is a gift to us all. </br>From writing all-time great songs like “\n",
       "<mark class=\"entity\" style=\"background: #f0d0ff; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Jolene\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">WORK_OF_ART</span>\n",
       "</mark>\n",
       "” and “\n",
       "<mark class=\"entity\" style=\"background: #f0d0ff; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    I Will Always Love You\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">WORK_OF_ART</span>\n",
       "</mark>\n",
       "”, </br>to great performances in films like \n",
       "<mark class=\"entity\" style=\"background: #e4e7d2; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    9\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">CARDINAL</span>\n",
       "</mark>\n",
       " to 5, to helping fund a COVID-19 vaccine, </br>she’s given us so much. Now, \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Netflix\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " bring us \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Dolly Parton’s\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Christmas\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " on the \n",
       "<mark class=\"entity\" style=\"background: #9cc9cc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Square\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">FAC</span>\n",
       "</mark>\n",
       ", </br>an original musical that stars \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Christine Baranski\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " as a Scrooge-like landowner </br>who threatens to evict an entire town on \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Christmas Eve\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " to make room for a new mall. </br>Directed and choreographed by the legendary \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Debbie Allen\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " and counting \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Jennifer Lewis\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " </br>and \n",
       "<mark class=\"entity\" style=\"background: #aa9cfc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Parton\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PERSON</span>\n",
       "</mark>\n",
       " herself amongst its cast, \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Christmas\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       " on the \n",
       "<mark class=\"entity\" style=\"background: #9cc9cc; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Square\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">FAC</span>\n",
       "</mark>\n",
       " seems like the perfect movie</br>to save \n",
       "<mark class=\"entity\" style=\"background: #bfe1d9; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Christmas 2020\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">DATE</span>\n",
       "</mark>\n",
       ". \n",
       "<mark class=\"entity\" style=\"background: #feca74; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    😻\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">GPE</span>\n",
       "</mark>\n",
       " 👍🏿</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from spacy import displacy\n",
    "\n",
    "displacy.render(doc, style=\"ent\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "###  An example from a project \n",
    "\n",
    "Goal: Extract and visualize inter-corporate relationships from disclosed annual 10-K reports of public companies. \n",
    "\n",
    "[Source for the text below.](https://www.bbc.com/news/business-39875417)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "text = (\n",
    "    \"Heavy hitters, including Microsoft and Google, \"\n",
    "    \"are competing for customers in cloud services with the likes of IBM and Salesforce.\"\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<span class=\"tex2jax_ignore\"><div class=\"entities\" style=\"line-height: 2.5; direction: ltr\">Heavy hitters, including \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Microsoft\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Google\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       ", are competing for customers in cloud services with the likes of \n",
       "<mark class=\"entity\" style=\"background: #7aecec; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    IBM\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">ORG</span>\n",
       "</mark>\n",
       " and \n",
       "<mark class=\"entity\" style=\"background: #bfeeb7; padding: 0.45em 0.6em; margin: 0 0.25em; line-height: 1; border-radius: 0.35em;\">\n",
       "    Salesforce\n",
       "    <span style=\"font-size: 0.8em; font-weight: bold; line-height: 1; border-radius: 0.35em; vertical-align: middle; margin-left: 0.5rem\">PRODUCT</span>\n",
       "</mark>\n",
       ".</div></span>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Named entities:\n",
      " [('Microsoft', 'ORG'), ('Google', 'ORG'), ('IBM', 'ORG'), ('Salesforce', 'PRODUCT')]\n"
     ]
    }
   ],
   "source": [
    "doc = nlp(text)\n",
    "displacy.render(doc, style=\"ent\")\n",
    "print(\"Named entities:\\n\", [(ent.text, ent.label_) for ent in doc.ents])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "If you want emoji identification support install [`spacymoji`](https://pypi.org/project/spacymoji/) in the course environment. \n",
    "\n",
    "```\n",
    "pip install spacymoji\n",
    "```\n",
    "\n",
    "After installing `spacymoji`, if it's still complaining about module not found, my guess is that you do not have `pip` installed in your `conda` environment. Go to your course `conda` environment install `pip` and install the `spacymoji` package in the environment using the `pip` you just installed in the current environment. \n",
    "\n",
    "```\n",
    "conda install pip\n",
    "YOUR_MINICONDA_PATH/miniconda3/envs/cpsc330/bin/pip install spacymoji\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from spacymoji import Emoji\n",
    "\n",
    "nlp.add_pipe(\"emoji\", first=True);"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Does the text have any emojis? If yes, extract the description. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[('😻', 138, 'smiling cat face with heart-eyes'),\n",
       " ('👍🏿', 139, 'thumbs up dark skin tone')]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "doc = nlp(sample_text)\n",
    "doc._.emoji"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Simple feature engineering for our problem. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "import en_core_web_md\n",
    "import spacy\n",
    "\n",
    "nlp = en_core_web_md.load()\n",
    "from spacymoji import Emoji\n",
    "\n",
    "nlp.add_pipe(\"emoji\", first=True)\n",
    "\n",
    "def get_relative_length(text, TWITTER_ALLOWED_CHARS=280.0):\n",
    "    \"\"\"\n",
    "    Returns the relative length of text.\n",
    "\n",
    "    Parameters:\n",
    "    ------\n",
    "    text: (str)\n",
    "    the input text\n",
    "\n",
    "    Keyword arguments:\n",
    "    ------\n",
    "    TWITTER_ALLOWED_CHARS: (float)\n",
    "    the denominator for finding relative length\n",
    "\n",
    "    Returns:\n",
    "    -------\n",
    "    relative length of text: (float)\n",
    "\n",
    "    \"\"\"\n",
    "    return len(text) / TWITTER_ALLOWED_CHARS\n",
    "\n",
    "\n",
    "def get_length_in_words(text):\n",
    "    \"\"\"\n",
    "    Returns the length of the text in words.\n",
    "\n",
    "    Parameters:\n",
    "    ------\n",
    "    text: (str)\n",
    "    the input text\n",
    "\n",
    "    Returns:\n",
    "    -------\n",
    "    length of tokenized text: (int)\n",
    "\n",
    "    \"\"\"\n",
    "    return len(nltk.word_tokenize(text))\n",
    "\n",
    "\n",
    "def get_sentiment(text):\n",
    "    \"\"\"\n",
    "    Returns the compound score representing the sentiment: -1 (most extreme negative) and +1 (most extreme positive)\n",
    "    The compound score is a normalized score calculated by summing the valence scores of each word in the lexicon.\n",
    "\n",
    "    Parameters:\n",
    "    ------\n",
    "    text: (str)\n",
    "    the input text\n",
    "\n",
    "    Returns:\n",
    "    -------\n",
    "    sentiment of the text: (str)\n",
    "    \"\"\"\n",
    "    scores = sid.polarity_scores(text)\n",
    "    return scores[\"compound\"]\n",
    "\n",
    "def get_avg_word_length(text):\n",
    "    \"\"\"\n",
    "    Returns the average word length of the given text.\n",
    "\n",
    "    Parameters:\n",
    "    text -- (str)\n",
    "    \"\"\"\n",
    "    words = text.split()\n",
    "    return sum(len(word) for word in words) / len(words)\n",
    "\n",
    "\n",
    "def has_emoji(text):\n",
    "    \"\"\"\n",
    "    Returns the average word length of the given text.\n",
    "\n",
    "    Parameters:\n",
    "    text -- (str)\n",
    "    \"\"\"\n",
    "    doc = nlp(text)\n",
    "    return 1 if doc._.has_emoji else 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>UserName</th>\n",
       "      <th>ScreenName</th>\n",
       "      <th>Location</th>\n",
       "      <th>TweetAt</th>\n",
       "      <th>OriginalTweet</th>\n",
       "      <th>Sentiment</th>\n",
       "      <th>n_words</th>\n",
       "      <th>vader_sentiment</th>\n",
       "      <th>rel_char_len</th>\n",
       "      <th>average_word_length</th>\n",
       "      <th>all_caps</th>\n",
       "      <th>has_emoji</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1927</th>\n",
       "      <td>1928</td>\n",
       "      <td>46880</td>\n",
       "      <td>Seattle, WA</td>\n",
       "      <td>13-03-2020</td>\n",
       "      <td>While I don't like all of Amazon's choices, to...</td>\n",
       "      <td>Positive</td>\n",
       "      <td>31</td>\n",
       "      <td>-0.1053</td>\n",
       "      <td>0.589286</td>\n",
       "      <td>5.640000</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1068</th>\n",
       "      <td>1069</td>\n",
       "      <td>46021</td>\n",
       "      <td>NaN</td>\n",
       "      <td>13-03-2020</td>\n",
       "      <td>Me: shit buckets, its time to do the weekly s...</td>\n",
       "      <td>Negative</td>\n",
       "      <td>52</td>\n",
       "      <td>-0.2500</td>\n",
       "      <td>0.932143</td>\n",
       "      <td>4.636364</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>803</th>\n",
       "      <td>804</td>\n",
       "      <td>45756</td>\n",
       "      <td>The Outer Limits</td>\n",
       "      <td>12-03-2020</td>\n",
       "      <td>@SecPompeo @realDonaldTrump You mean the plan ...</td>\n",
       "      <td>Neutral</td>\n",
       "      <td>44</td>\n",
       "      <td>0.0000</td>\n",
       "      <td>0.910714</td>\n",
       "      <td>6.741935</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2846</th>\n",
       "      <td>2847</td>\n",
       "      <td>47799</td>\n",
       "      <td>Flagstaff, AZ</td>\n",
       "      <td>15-03-2020</td>\n",
       "      <td>@lauvagrande People who are sick arent panic ...</td>\n",
       "      <td>Extremely Negative</td>\n",
       "      <td>46</td>\n",
       "      <td>-0.8481</td>\n",
       "      <td>0.907143</td>\n",
       "      <td>5.023810</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3768</th>\n",
       "      <td>3769</td>\n",
       "      <td>48721</td>\n",
       "      <td>Montreal, Canada</td>\n",
       "      <td>16-03-2020</td>\n",
       "      <td>Coronavirus Panic: Toilet Paper Is the People...</td>\n",
       "      <td>Negative</td>\n",
       "      <td>21</td>\n",
       "      <td>-0.5106</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>9.846154</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      UserName  ScreenName          Location     TweetAt  \\\n",
       "1927      1928       46880       Seattle, WA  13-03-2020   \n",
       "1068      1069       46021               NaN  13-03-2020   \n",
       "803        804       45756  The Outer Limits  12-03-2020   \n",
       "2846      2847       47799     Flagstaff, AZ  15-03-2020   \n",
       "3768      3769       48721  Montreal, Canada  16-03-2020   \n",
       "\n",
       "                                          OriginalTweet           Sentiment  \\\n",
       "1927  While I don't like all of Amazon's choices, to...            Positive   \n",
       "1068  Me: shit buckets, its time to do the weekly s...            Negative   \n",
       "803   @SecPompeo @realDonaldTrump You mean the plan ...             Neutral   \n",
       "2846  @lauvagrande People who are sick arent panic ...  Extremely Negative   \n",
       "3768  Coronavirus Panic: Toilet Paper Is the People...            Negative   \n",
       "\n",
       "      n_words  vader_sentiment  rel_char_len  average_word_length  all_caps  \\\n",
       "1927       31          -0.1053      0.589286             5.640000         0   \n",
       "1068       52          -0.2500      0.932143             4.636364         0   \n",
       "803        44           0.0000      0.910714             6.741935         0   \n",
       "2846       46          -0.8481      0.907143             5.023810         0   \n",
       "3768       21          -0.5106      0.500000             9.846154         0   \n",
       "\n",
       "      has_emoji  \n",
       "1927          0  \n",
       "1068          0  \n",
       "803           0  \n",
       "2846          0  \n",
       "3768          0  "
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df = train_df.assign(n_words=train_df[\"OriginalTweet\"].apply(get_length_in_words))\n",
    "train_df = train_df.assign(vader_sentiment=train_df[\"OriginalTweet\"].apply(get_sentiment))\n",
    "train_df = train_df.assign(rel_char_len=train_df[\"OriginalTweet\"].apply(get_relative_length))\n",
    "\n",
    "test_df = test_df.assign(n_words=test_df[\"OriginalTweet\"].apply(get_length_in_words))\n",
    "test_df = test_df.assign(vader_sentiment=test_df[\"OriginalTweet\"].apply(get_sentiment))\n",
    "test_df = test_df.assign(rel_char_len=test_df[\"OriginalTweet\"].apply(get_relative_length))\n",
    "\n",
    "\n",
    "train_df = train_df.assign(\n",
    "    average_word_length=train_df[\"OriginalTweet\"].apply(get_avg_word_length)\n",
    ")\n",
    "test_df = test_df.assign(average_word_length=test_df[\"OriginalTweet\"].apply(get_avg_word_length))\n",
    "\n",
    "# whether all letters are uppercase or not (all_caps)\n",
    "train_df = train_df.assign(\n",
    "    all_caps=train_df[\"OriginalTweet\"].apply(lambda x: 1 if x.isupper() else 0)\n",
    ")\n",
    "test_df = test_df.assign(\n",
    "    all_caps=test_df[\"OriginalTweet\"].apply(lambda x: 1 if x.isupper() else 0)\n",
    ")\n",
    "\n",
    "train_df = train_df.assign(has_emoji=train_df[\"OriginalTweet\"].apply(has_emoji))\n",
    "test_df = test_df.assign(has_emoji=test_df[\"OriginalTweet\"].apply(has_emoji))\n",
    "\n",
    "train_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3038, 12)"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "(train_df['all_caps'] == 1).sum()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = train_df.drop(columns=['Sentiment'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "numeric_features = ['vader_sentiment', \n",
    "                    'rel_char_len', \n",
    "                    'average_word_length']\n",
    "passthrough_features = ['all_caps', 'has_emoji'] \n",
    "text_feature = 'OriginalTweet'\n",
    "drop_features = ['UserName', 'ScreenName', 'Location', 'TweetAt']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "preprocessor = make_column_transformer(\n",
    "    (StandardScaler(), numeric_features),\n",
    "    (\"passthrough\", passthrough_features), \n",
    "    (CountVectorizer(stop_words='english'), text_feature),\n",
    "    (\"drop\", drop_features)\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>fit_time</th>\n",
       "      <th>score_time</th>\n",
       "      <th>test_score</th>\n",
       "      <th>train_score</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>dummy</th>\n",
       "      <td>0.001 (+/- 0.000)</td>\n",
       "      <td>0.000 (+/- 0.000)</td>\n",
       "      <td>0.280 (+/- 0.001)</td>\n",
       "      <td>0.280 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>logistic regression</th>\n",
       "      <td>0.442 (+/- 0.027)</td>\n",
       "      <td>0.007 (+/- 0.000)</td>\n",
       "      <td>0.413 (+/- 0.011)</td>\n",
       "      <td>0.999 (+/- 0.000)</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LR (more feats)</th>\n",
       "      <td>0.551 (+/- 0.054)</td>\n",
       "      <td>0.010 (+/- 0.001)</td>\n",
       "      <td>0.689 (+/- 0.007)</td>\n",
       "      <td>0.998 (+/- 0.001)</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                              fit_time         score_time         test_score  \\\n",
       "dummy                0.001 (+/- 0.000)  0.000 (+/- 0.000)  0.280 (+/- 0.001)   \n",
       "logistic regression  0.442 (+/- 0.027)  0.007 (+/- 0.000)  0.413 (+/- 0.011)   \n",
       "LR (more feats)      0.551 (+/- 0.054)  0.010 (+/- 0.001)  0.689 (+/- 0.007)   \n",
       "\n",
       "                           train_score  \n",
       "dummy                0.280 (+/- 0.000)  \n",
       "logistic regression  0.999 (+/- 0.000)  \n",
       "LR (more feats)      0.998 (+/- 0.001)  "
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe = make_pipeline(preprocessor, LogisticRegression(max_iter=1000))\n",
    "results[\"LR (more feats)\"] = mean_std_cross_val_scores(\n",
    "    pipe, X_train, y_train, return_train_score=True, scoring=scoring_metrics\n",
    ")\n",
    "pd.DataFrame(results).T"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>#sk-container-id-1 {color: black;background-color: white;}#sk-container-id-1 pre{padding: 0;}#sk-container-id-1 div.sk-toggleable {background-color: white;}#sk-container-id-1 label.sk-toggleable__label {cursor: pointer;display: block;width: 100%;margin-bottom: 0;padding: 0.3em;box-sizing: border-box;text-align: center;}#sk-container-id-1 label.sk-toggleable__label-arrow:before {content: \"▸\";float: left;margin-right: 0.25em;color: #696969;}#sk-container-id-1 label.sk-toggleable__label-arrow:hover:before {color: black;}#sk-container-id-1 div.sk-estimator:hover label.sk-toggleable__label-arrow:before {color: black;}#sk-container-id-1 div.sk-toggleable__content {max-height: 0;max-width: 0;overflow: hidden;text-align: left;background-color: #f0f8ff;}#sk-container-id-1 div.sk-toggleable__content pre {margin: 0.2em;color: black;border-radius: 0.25em;background-color: #f0f8ff;}#sk-container-id-1 input.sk-toggleable__control:checked~div.sk-toggleable__content {max-height: 200px;max-width: 100%;overflow: auto;}#sk-container-id-1 input.sk-toggleable__control:checked~label.sk-toggleable__label-arrow:before {content: \"▾\";}#sk-container-id-1 div.sk-estimator input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-label input.sk-toggleable__control:checked~label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 input.sk-hidden--visually {border: 0;clip: rect(1px 1px 1px 1px);clip: rect(1px, 1px, 1px, 1px);height: 1px;margin: -1px;overflow: hidden;padding: 0;position: absolute;width: 1px;}#sk-container-id-1 div.sk-estimator {font-family: monospace;background-color: #f0f8ff;border: 1px dotted black;border-radius: 0.25em;box-sizing: border-box;margin-bottom: 0.5em;}#sk-container-id-1 div.sk-estimator:hover {background-color: #d4ebff;}#sk-container-id-1 div.sk-parallel-item::after {content: \"\";width: 100%;border-bottom: 1px solid gray;flex-grow: 1;}#sk-container-id-1 div.sk-label:hover label.sk-toggleable__label {background-color: #d4ebff;}#sk-container-id-1 div.sk-serial::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: 0;}#sk-container-id-1 div.sk-serial {display: flex;flex-direction: column;align-items: center;background-color: white;padding-right: 0.2em;padding-left: 0.2em;position: relative;}#sk-container-id-1 div.sk-item {position: relative;z-index: 1;}#sk-container-id-1 div.sk-parallel {display: flex;align-items: stretch;justify-content: center;background-color: white;position: relative;}#sk-container-id-1 div.sk-item::before, #sk-container-id-1 div.sk-parallel-item::before {content: \"\";position: absolute;border-left: 1px solid gray;box-sizing: border-box;top: 0;bottom: 0;left: 50%;z-index: -1;}#sk-container-id-1 div.sk-parallel-item {display: flex;flex-direction: column;z-index: 1;position: relative;background-color: white;}#sk-container-id-1 div.sk-parallel-item:first-child::after {align-self: flex-end;width: 50%;}#sk-container-id-1 div.sk-parallel-item:last-child::after {align-self: flex-start;width: 50%;}#sk-container-id-1 div.sk-parallel-item:only-child::after {width: 0;}#sk-container-id-1 div.sk-dashed-wrapped {border: 1px dashed gray;margin: 0 0.4em 0.5em 0.4em;box-sizing: border-box;padding-bottom: 0.4em;background-color: white;}#sk-container-id-1 div.sk-label label {font-family: monospace;font-weight: bold;display: inline-block;line-height: 1.2em;}#sk-container-id-1 div.sk-label-container {text-align: center;}#sk-container-id-1 div.sk-container {/* jupyter's `normalize.less` sets `[hidden] { display: none; }` but bootstrap.min.css set `[hidden] { display: none !important; }` so we also need the `!important` here to be able to override the default hidden behavior on the sphinx rendered scikit-learn.org. See: https://github.com/scikit-learn/scikit-learn/issues/21755 */display: inline-block !important;position: relative;}#sk-container-id-1 div.sk-text-repr-fallback {display: none;}</style><div id=\"sk-container-id-1\" class=\"sk-top-container\"><div class=\"sk-text-repr-fallback\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler(),\n",
       "                                                  [&#x27;vader_sentiment&#x27;,\n",
       "                                                   &#x27;rel_char_len&#x27;,\n",
       "                                                   &#x27;average_word_length&#x27;]),\n",
       "                                                 (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                                  [&#x27;all_caps&#x27;, &#x27;has_emoji&#x27;]),\n",
       "                                                 (&#x27;countvectorizer&#x27;,\n",
       "                                                  CountVectorizer(stop_words=&#x27;english&#x27;),\n",
       "                                                  &#x27;OriginalTweet&#x27;),\n",
       "                                                 (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                                  [&#x27;UserName&#x27;, &#x27;ScreenName&#x27;,\n",
       "                                                   &#x27;Location&#x27;, &#x27;TweetAt&#x27;])])),\n",
       "                (&#x27;logisticregression&#x27;, LogisticRegression(max_iter=1000))])</pre><b>In a Jupyter environment, please rerun this cell to show the HTML representation or trust the notebook. <br />On GitHub, the HTML representation is unable to render, please try loading this page with nbviewer.org.</b></div><div class=\"sk-container\" hidden><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-1\" type=\"checkbox\" ><label for=\"sk-estimator-id-1\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">Pipeline</label><div class=\"sk-toggleable__content\"><pre>Pipeline(steps=[(&#x27;columntransformer&#x27;,\n",
       "                 ColumnTransformer(transformers=[(&#x27;standardscaler&#x27;,\n",
       "                                                  StandardScaler(),\n",
       "                                                  [&#x27;vader_sentiment&#x27;,\n",
       "                                                   &#x27;rel_char_len&#x27;,\n",
       "                                                   &#x27;average_word_length&#x27;]),\n",
       "                                                 (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                                  [&#x27;all_caps&#x27;, &#x27;has_emoji&#x27;]),\n",
       "                                                 (&#x27;countvectorizer&#x27;,\n",
       "                                                  CountVectorizer(stop_words=&#x27;english&#x27;),\n",
       "                                                  &#x27;OriginalTweet&#x27;),\n",
       "                                                 (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                                  [&#x27;UserName&#x27;, &#x27;ScreenName&#x27;,\n",
       "                                                   &#x27;Location&#x27;, &#x27;TweetAt&#x27;])])),\n",
       "                (&#x27;logisticregression&#x27;, LogisticRegression(max_iter=1000))])</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item sk-dashed-wrapped\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-2\" type=\"checkbox\" ><label for=\"sk-estimator-id-2\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">columntransformer: ColumnTransformer</label><div class=\"sk-toggleable__content\"><pre>ColumnTransformer(transformers=[(&#x27;standardscaler&#x27;, StandardScaler(),\n",
       "                                 [&#x27;vader_sentiment&#x27;, &#x27;rel_char_len&#x27;,\n",
       "                                  &#x27;average_word_length&#x27;]),\n",
       "                                (&#x27;passthrough&#x27;, &#x27;passthrough&#x27;,\n",
       "                                 [&#x27;all_caps&#x27;, &#x27;has_emoji&#x27;]),\n",
       "                                (&#x27;countvectorizer&#x27;,\n",
       "                                 CountVectorizer(stop_words=&#x27;english&#x27;),\n",
       "                                 &#x27;OriginalTweet&#x27;),\n",
       "                                (&#x27;drop&#x27;, &#x27;drop&#x27;,\n",
       "                                 [&#x27;UserName&#x27;, &#x27;ScreenName&#x27;, &#x27;Location&#x27;,\n",
       "                                  &#x27;TweetAt&#x27;])])</pre></div></div></div><div class=\"sk-parallel\"><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-3\" type=\"checkbox\" ><label for=\"sk-estimator-id-3\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">standardscaler</label><div class=\"sk-toggleable__content\"><pre>[&#x27;vader_sentiment&#x27;, &#x27;rel_char_len&#x27;, &#x27;average_word_length&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-4\" type=\"checkbox\" ><label for=\"sk-estimator-id-4\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">StandardScaler</label><div class=\"sk-toggleable__content\"><pre>StandardScaler()</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-5\" type=\"checkbox\" ><label for=\"sk-estimator-id-5\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>[&#x27;all_caps&#x27;, &#x27;has_emoji&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-6\" type=\"checkbox\" ><label for=\"sk-estimator-id-6\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">passthrough</label><div class=\"sk-toggleable__content\"><pre>passthrough</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-7\" type=\"checkbox\" ><label for=\"sk-estimator-id-7\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">countvectorizer</label><div class=\"sk-toggleable__content\"><pre>OriginalTweet</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-8\" type=\"checkbox\" ><label for=\"sk-estimator-id-8\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">CountVectorizer</label><div class=\"sk-toggleable__content\"><pre>CountVectorizer(stop_words=&#x27;english&#x27;)</pre></div></div></div></div></div></div><div class=\"sk-parallel-item\"><div class=\"sk-item\"><div class=\"sk-label-container\"><div class=\"sk-label sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-9\" type=\"checkbox\" ><label for=\"sk-estimator-id-9\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>[&#x27;UserName&#x27;, &#x27;ScreenName&#x27;, &#x27;Location&#x27;, &#x27;TweetAt&#x27;]</pre></div></div></div><div class=\"sk-serial\"><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-10\" type=\"checkbox\" ><label for=\"sk-estimator-id-10\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">drop</label><div class=\"sk-toggleable__content\"><pre>drop</pre></div></div></div></div></div></div></div></div><div class=\"sk-item\"><div class=\"sk-estimator sk-toggleable\"><input class=\"sk-toggleable__control sk-hidden--visually\" id=\"sk-estimator-id-11\" type=\"checkbox\" ><label for=\"sk-estimator-id-11\" class=\"sk-toggleable__label sk-toggleable__label-arrow\">LogisticRegression</label><div class=\"sk-toggleable__content\"><pre>LogisticRegression(max_iter=1000)</pre></div></div></div></div></div></div></div>"
      ],
      "text/plain": [
       "Pipeline(steps=[('columntransformer',\n",
       "                 ColumnTransformer(transformers=[('standardscaler',\n",
       "                                                  StandardScaler(),\n",
       "                                                  ['vader_sentiment',\n",
       "                                                   'rel_char_len',\n",
       "                                                   'average_word_length']),\n",
       "                                                 ('passthrough', 'passthrough',\n",
       "                                                  ['all_caps', 'has_emoji']),\n",
       "                                                 ('countvectorizer',\n",
       "                                                  CountVectorizer(stop_words='english'),\n",
       "                                                  'OriginalTweet'),\n",
       "                                                 ('drop', 'drop',\n",
       "                                                  ['UserName', 'ScreenName',\n",
       "                                                   'Location', 'TweetAt'])])),\n",
       "                ('logisticregression', LogisticRegression(max_iter=1000))])"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "cv_feats = pipe.named_steps['columntransformer'].named_transformers_['countvectorizer'].get_feature_names()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "metadata": {},
   "outputs": [],
   "source": [
    "feat_names = numeric_features + passthrough_features + cv_feats"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "coefs = pipe.named_steps['logisticregression'].coef_[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>features</th>\n",
       "      <th>coefficients</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>vader_sentiment</td>\n",
       "      <td>-6.141919</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11331</th>\n",
       "      <td>won</td>\n",
       "      <td>-1.369740</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2551</th>\n",
       "      <td>coronapocalypse</td>\n",
       "      <td>-0.809931</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2214</th>\n",
       "      <td>closed</td>\n",
       "      <td>-0.744717</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8661</th>\n",
       "      <td>retail</td>\n",
       "      <td>-0.723808</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9862</th>\n",
       "      <td>stupid</td>\n",
       "      <td>1.157669</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3299</th>\n",
       "      <td>don</td>\n",
       "      <td>1.159067</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4879</th>\n",
       "      <td>hell</td>\n",
       "      <td>1.311957</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3129</th>\n",
       "      <td>die</td>\n",
       "      <td>1.366538</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7504</th>\n",
       "      <td>panic</td>\n",
       "      <td>1.527156</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>11664 rows × 2 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              features  coefficients\n",
       "0      vader_sentiment     -6.141919\n",
       "11331              won     -1.369740\n",
       "2551   coronapocalypse     -0.809931\n",
       "2214            closed     -0.744717\n",
       "8661            retail     -0.723808\n",
       "...                ...           ...\n",
       "9862            stupid      1.157669\n",
       "3299               don      1.159067\n",
       "4879              hell      1.311957\n",
       "3129               die      1.366538\n",
       "7504             panic      1.527156\n",
       "\n",
       "[11664 rows x 2 columns]"
      ]
     },
     "execution_count": 41,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.DataFrame(\n",
    "    data={\n",
    "        \"features\": feat_names,\n",
    "        \"coefficients\": coefs,\n",
    "    }\n",
    ")\n",
    "df.sort_values('coefficients')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We get some improvements with our engineered features! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Common features used in text classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Bag of words \n",
    "\n",
    "- So far for text data we have been using bag of word features. \n",
    "- They are good enough for many tasks. But ... \n",
    "- This encoding throws out a lot of things we know about language\n",
    "- It assumes that word order is not that important.   \n",
    "- So if you want to improve the scores further on text classification tasks you carry out **feature engineering**. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "Let's look at some examples from research papers.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Example: Label \"Personalized\" Important E-mails: \n",
    "- [The Learning Behind Gmail Priority Inbox](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36955.pdf)\n",
    "- Features: bag of words, trigrams, regular expressions, and so on.\n",
    "- There might be some \"globally\" important messages:\n",
    "    - \"This is your mother, something terrible happened, give me a call ASAP.\"\n",
    "- But your \"important\" message may be unimportant to others.\n",
    "     - Similar for spam: \"spam\" for one user could be \"not spam\" for another."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Social features (e.g., percentage of sender emails that is read by the recipient)\n",
    "- Content features (e.g., recent terms the user has been using in emails)\n",
    "- Thread features (e.g., whether the user has started the thread)\n",
    "- ..."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "source": [
    "### [The Learning Behind Gmail Priority Inbox](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/36955.pdf)\n",
    "\n",
    "![](img/gmail_priority_inbox.png)\n",
    "\n",
    "<!-- <img src=\"img/gmail_priority_inbox.png\" width=\"1000\" height=\"1000\"> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Feature engineering examples: [Automatically Identifying Good Conversations Online](http://www.courtneynapoles.com/res/icwsm17-automatically.pdf)\n",
    "\n",
    "![](img/classifying_good_conversations_online.png)\n",
    "\n",
    "<!-- <img src=\"img/classifying_good_conversations_online.png\" width=\"800\" height=\"800\"> -->\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### (Optional) Term weighing (TF-IDF) \n",
    "\n",
    "- A measure of relatedness between words and documents\n",
    "- Intuition: Meaningful words may occur repeatedly in related documents, but functional words (e.g., _make_, _the_) may be distributed evenly over all documents"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "\n",
    "$$tf.idf(w_i,d_j) = (1+log(tf_{ij})) log\\frac{D}{df_i}$$\n",
    "\n",
    "\n",
    "where, \n",
    "- $tf_{ij}$ &rarr; number of occurrences of the term $w_i$ in document $d_j$\n",
    "- $D$ &rarr; number of documents\n",
    "- $df_i$ &rarr; number of documents in which $w_i$ occurs\n",
    "\n",
    "Check `TfidfVectorizer` from `sklearn`. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### N-grams \n",
    "\n",
    "- Incorporating more context \n",
    "- A contiguous sequence of _n_ items (characters, tokens) in text.\n",
    "    <blockquote>\n",
    "        CPSC330 students are hard-working .\n",
    "    </blockquote>    \n",
    "\n",
    "- 2-grams (bigrams): a contiguous sequence of two words\n",
    "    * _CPSC330 students, students are, are hard-working, hard-working ._\n",
    "- 3-grams (trigrams): a contiguous sequence of three words\n",
    "    * _CPSC330 students are, students are hard-working, are hard-working ._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "You can extract ngram features using `CountVectorizer` by passing `ngram_range`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/sklearn/utils/deprecation.py:87: FutureWarning: Function get_feature_names is deprecated; get_feature_names is deprecated in 1.0 and will be removed in 1.2. Please use get_feature_names_out instead.\n",
      "  warnings.warn(msg, category=FutureWarning)\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_extraction.text import CountVectorizer\n",
    "\n",
    "X = [\n",
    "    \"URGENT!! As a valued network customer you have been selected to receive a $900 prize reward!\",\n",
    "    \"Lol you are always so convincing.\",\n",
    "    \"URGENT!! Call right away!!\",\n",
    "]\n",
    "vec = CountVectorizer(ngram_range=(1, 3))\n",
    "X_counts = vec.fit_transform(X)\n",
    "bow_df = pd.DataFrame(X_counts.toarray(), columns=vec.get_feature_names(), index=X)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>900</th>\n",
       "      <th>900 prize</th>\n",
       "      <th>900 prize reward</th>\n",
       "      <th>always</th>\n",
       "      <th>always so</th>\n",
       "      <th>always so convincing</th>\n",
       "      <th>are</th>\n",
       "      <th>are always</th>\n",
       "      <th>are always so</th>\n",
       "      <th>as</th>\n",
       "      <th>...</th>\n",
       "      <th>urgent call</th>\n",
       "      <th>urgent call right</th>\n",
       "      <th>valued</th>\n",
       "      <th>valued network</th>\n",
       "      <th>valued network customer</th>\n",
       "      <th>you</th>\n",
       "      <th>you are</th>\n",
       "      <th>you are always</th>\n",
       "      <th>you have</th>\n",
       "      <th>you have been</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>URGENT!! As a valued network customer you have been selected to receive a $900 prize reward!</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Lol you are always so convincing.</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>URGENT!! Call right away!!</th>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>3 rows × 61 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                                    900  900 prize  \\\n",
       "URGENT!! As a valued network customer you have ...    1          1   \n",
       "Lol you are always so convincing.                     0          0   \n",
       "URGENT!! Call right away!!                            0          0   \n",
       "\n",
       "                                                    900 prize reward  always  \\\n",
       "URGENT!! As a valued network customer you have ...                 1       0   \n",
       "Lol you are always so convincing.                                  0       1   \n",
       "URGENT!! Call right away!!                                         0       0   \n",
       "\n",
       "                                                    always so  \\\n",
       "URGENT!! As a valued network customer you have ...          0   \n",
       "Lol you are always so convincing.                           1   \n",
       "URGENT!! Call right away!!                                  0   \n",
       "\n",
       "                                                    always so convincing  are  \\\n",
       "URGENT!! As a valued network customer you have ...                     0    0   \n",
       "Lol you are always so convincing.                                      1    1   \n",
       "URGENT!! Call right away!!                                             0    0   \n",
       "\n",
       "                                                    are always  are always so  \\\n",
       "URGENT!! As a valued network customer you have ...           0              0   \n",
       "Lol you are always so convincing.                            1              1   \n",
       "URGENT!! Call right away!!                                   0              0   \n",
       "\n",
       "                                                    as  ...  urgent call  \\\n",
       "URGENT!! As a valued network customer you have ...   1  ...            0   \n",
       "Lol you are always so convincing.                    0  ...            0   \n",
       "URGENT!! Call right away!!                           0  ...            1   \n",
       "\n",
       "                                                    urgent call right  valued  \\\n",
       "URGENT!! As a valued network customer you have ...                  0       1   \n",
       "Lol you are always so convincing.                                   0       0   \n",
       "URGENT!! Call right away!!                                          1       0   \n",
       "\n",
       "                                                    valued network  \\\n",
       "URGENT!! As a valued network customer you have ...               1   \n",
       "Lol you are always so convincing.                                0   \n",
       "URGENT!! Call right away!!                                       0   \n",
       "\n",
       "                                                    valued network customer  \\\n",
       "URGENT!! As a valued network customer you have ...                        1   \n",
       "Lol you are always so convincing.                                         0   \n",
       "URGENT!! Call right away!!                                                0   \n",
       "\n",
       "                                                    you  you are  \\\n",
       "URGENT!! As a valued network customer you have ...    1        0   \n",
       "Lol you are always so convincing.                     1        1   \n",
       "URGENT!! Call right away!!                            0        0   \n",
       "\n",
       "                                                    you are always  you have  \\\n",
       "URGENT!! As a valued network customer you have ...               0         1   \n",
       "Lol you are always so convincing.                                1         0   \n",
       "URGENT!! Call right away!!                                       0         0   \n",
       "\n",
       "                                                    you have been  \n",
       "URGENT!! As a valued network customer you have ...              1  \n",
       "Lol you are always so convincing.                               0  \n",
       "URGENT!! Call right away!!                                      0  \n",
       "\n",
       "[3 rows x 61 columns]"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "bow_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### ASIDE: [Google n-gram viewer](https://books.google.com/ngrams)\n",
    " \n",
    "- All Our N-gram are Belong to You\n",
    "    - https://ai.googleblog.com/2006/08/all-our-n-gram-are-belong-toyou.html\n",
    "\n",
    "<blockquote>\n",
    "Here at Google Research we have been using word n-gram models for a variety\n",
    "of R&D projects, such as statistical machine translation, speech recognition,\n",
    "spelling correction, entity detection, information extraction, and others.\n",
    "That's why we decided to share this enormous dataset with everyone. We\n",
    "processed 1,024,908,267,229 words of running text and are publishing the\n",
    "counts for all 1,176,470,663 five-word sequences that appear at least 40\n",
    "times. There are 13,588,391 unique words, after discarding words that appear\n",
    "less than 200 times.”\n",
    "</blockquote>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/Users/kvarada/opt/miniconda3/envs/cpsc330/lib/python3.10/site-packages/IPython/core/display.py:419: UserWarning: Consider using IPython.display.IFrame instead\n",
      "  warnings.warn(\"Consider using IPython.display.IFrame instead\")\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<iframe src=https://books.google.com/ngrams/ width=1000 height=800></iframe>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from IPython.display import HTML\n",
    "\n",
    "url = \"https://books.google.com/ngrams/\"\n",
    "HTML(\"<iframe src=%s width=1000 height=800></iframe>\" % url)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Aside: [Google n-gram viewer](https://books.google.com/ngrams)\n",
    " \n",
    "- Count the occurrences of the bigram _smart women_ in the corpus from 1800 to 2000 \n",
    "\n",
    "![](img/ngram_viewer_smart_woman.png)\n",
    "\n",
    "<!-- <img src=\"img/ngram_viewer_smart_woman.png\" width=\"800\" height=\"800\"> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Aside: [Google n-gram viewer](https://books.google.com/ngrams)\n",
    " \n",
    "- Trends in the word _challenge_ used as a noun vs. verb\n",
    "\n",
    "![](img/ngram_viewer_challenge_NN_VB.png)\n",
    "\n",
    "<!-- <img src=\"img/ngram_viewer_challenge_NN_VB.png\" width=\"800\" height=\"800\"> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Part-of-speech features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Part-of-speech (POS) in English\n",
    "\n",
    "- Part-of-speech: A kind of syntactic category that tells you some of the grammatical properties of a word.\n",
    "    * Noun &rarr; water, sun, cat  \n",
    "    * Verb &rarr; run, eat, teach\n",
    "\n",
    "    \n",
    "<blockquote>\n",
    "The ____ was running. \n",
    "</blockquote>    \n",
    "\n",
    "- Only a noun fits here. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Part-of-speech (POS) features\n",
    "\n",
    "- POS features use POS information for the words in text.  \n",
    "\n",
    "<blockquote>\n",
    "    CPSC330/<span style=\"color:green\">PROPER_NOUN</span> students/<span style=\"color:green\">NOUN</span> are/<span style=\"color:green\">VERB</span> hard-working/<span style=\"color:green\">ADJECTIVE</span>\n",
    "</blockquote>    \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### An example from a project \n",
    "\n",
    "- Data: a bunch of documents \n",
    "- Task: identify texts with *permissions* and identify who is giving permission to whom. \n",
    "\n",
    "<blockquote>\n",
    "<b>You</b> may <b>disclose</b> Google confidential information when compelled to do so by law if <b>you</b> provide <b>us</b> reasonable prior notice, unless a court orders that <b>we</b> not receive notice.\n",
    "</blockquote>\n",
    "\n",
    "- A very simple solution\n",
    "    * Look for pronouns and verbs. \n",
    "    * Add POS tags as features in your model. \n",
    "    * Maybe look up words similar to **disclose**.\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "### Penn Treebank part-of-speech tags (bonus)\n",
    "\n",
    "![](img/PTB_POS.png)\n",
    "\n",
    "<!-- <img src=\"img/PTB_POS.png\" width=\"900\" height=\"900\"/> -->"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- You also need to download the language model which contains all the pre-trained models. For that run the following in your course `conda` environment. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Interim summary \n",
    "\n",
    "- Feature engineering is finding the useful representation of the data that can help us effectively solve our problem. \n",
    "- In the context of text data, if we want to go beyond bag-of-words and incorporate human knowledge in models, we carry out feature engineering. \n",
    "- Some common features include:\n",
    "    - ngram features\n",
    "    - part-of-speech features\n",
    "    - named entity features\n",
    "    - emoticons in text\n",
    "- These are usually extracted from pre-trained models using libraries such as `spaCy`.  \n",
    "- Now a lot of this has moved to deep learning.\n",
    "- But many industries still rely on manual feature engineering. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Feature engineering \n",
    "\n",
    "- The best features are application-dependent.\n",
    "- It's hard to give general advice. But here are some guidelines. \n",
    "    - Ask the domain experts.\n",
    "    - Go through academic papers in the discipline. \n",
    "    - Often have idea of right discretization/standardization/transformation.\n",
    "    - If no domain expert, cross-validation will help.\n",
    "- If you have lots of data, use deep learning methods. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "<blockquote>\n",
    "    The algorithms we used are very standard for Kagglers ... We spent most of our efforts in feature engineering... <br>\n",
    "- Xavier Conort, on winning the Flight Quest challenge on Kaggle    \n",
    "</blockquote>    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Break (5 min)\n",
    "\n",
    "![](img/eva-coffee.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<br><br><br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "## Feature selection: Introduction and motivation "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "- With so many ways to add new features, we can increase dimensionality of the data. \n",
    "- More features means more complex models, which means increasing the chance of overfitting. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### What is feature selection?\n",
    "\n",
    "- Find the features\t(columns) $X$ that are important for predicting\t$y$, and remove the features that aren't. \n",
    "\n",
    "- Given $X = \\begin{bmatrix}x_1 & x_2 & \\dots & x_n\\\\  \\\\  \\\\  \\end{bmatrix}$ and $y = \\begin{bmatrix}\\\\  \\\\  \\\\  \\end{bmatrix}$, find the columns $1 \\leq j \\leq n$ in $X$ that are important for predicting $y$.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Why feature selection? \n",
    "\n",
    "- Interpretability: Models are more interpretable with fewer features. If you get the same performance with 10 features instead of 500 features, why not use the model with smaller number of features?     \n",
    "- Computation: Models fit/predict faster with fewer columns.\n",
    "- Data collection: What type of new data should I collect? It may be cheaper to collect fewer columns.\n",
    "- Fundamental tradeoff: Can I reduce overfitting by removing useless features?\n",
    "\n",
    "Feature selection can often result in better performing (less overfit), easier to understand, and faster model. \n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### How do we carry out feature selection? \n",
    "\n",
    "- There are a number of ways. \n",
    "- You could use domain knowledge to discard features. \n",
    "- We are briefly going to look at two automatic feature selection methods from `sklearn`: \n",
    "    - Model-based selection \n",
    "    - Recursive feature elimination\n",
    "    - Forward/backward selection \n",
    "- Very related to looking at feature importances."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.datasets import load_breast_cancer\n",
    "\n",
    "cancer = load_breast_cancer()\n",
    "X_train, X_test, y_train, y_test = train_test_split(\n",
    "    cancer.data, cancer.target, random_state=0, test_size=0.5\n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(284, 30)"
      ]
     },
     "execution_count": 46,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.001488\n",
       "score_time     0.000123\n",
       "test_score     0.968233\n",
       "train_score    0.987681\n",
       "dtype: float64"
      ]
     },
     "execution_count": 47,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr_all_feats = make_pipeline(StandardScaler(), LogisticRegression(max_iter=1000))\n",
    "pipe_lr_all_feats.fit(X_train, y_train)\n",
    "pd.DataFrame(\n",
    "    cross_validate(pipe_lr_all_feats, X_train, y_train, return_train_score=True)\n",
    ").mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Model-based selection\n",
    "\n",
    "- Use a supervised machine learning model to judge the importance of each feature.\n",
    "- Keep only the most important once. \n",
    "- Supervised machine learning model used for feature selection can be different that the one used as the final estimator. \n",
    "- Use a model which has some way to calculate feature importances."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- To use model-based selection, we use `SelectFromModel` transformer.\n",
    "- It selects features which have the feature importances greater than the provided threshold.\n",
    "- Below I'm using `RandomForestClassifier` for feature selection with threahold \"median\" of feature importances. \n",
    "- Approximately how many features will be selected? "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.feature_selection import SelectFromModel\n",
    "\n",
    "select_rf = SelectFromModel(\n",
    "    RandomForestClassifier(n_estimators=100, random_state=42), \n",
    "    threshold=\"median\"\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "We can put the feature selection transformer in a pipeline. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.056484\n",
       "score_time     0.004522\n",
       "test_score     0.950564\n",
       "train_score    0.974480\n",
       "dtype: float64"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr_model_based = make_pipeline(\n",
    "    StandardScaler(), select_rf, LogisticRegression(max_iter=1000)\n",
    ")\n",
    "\n",
    "pd.DataFrame(\n",
    "    cross_validate(pipe_lr_model_based, X_train, y_train, return_train_score=True)\n",
    ").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.054929\n",
       "score_time     0.004565\n",
       "test_score     0.950564\n",
       "train_score    0.974480\n",
       "dtype: float64"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr_model_based = make_pipeline(\n",
    "    StandardScaler(), select_rf, LogisticRegression(max_iter=1000)\n",
    ")\n",
    "\n",
    "pd.DataFrame(\n",
    "    cross_validate(pipe_lr_model_based, X_train, y_train, return_train_score=True)\n",
    ").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(284, 15)"
      ]
     },
     "execution_count": 51,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_lr_model_based.fit(X_train, y_train)\n",
    "pipe_lr_model_based.named_steps[\"selectfrommodel\"].transform(X_train).shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Similar results with only 15 features instead of 30 features. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Recursive feature elimination (RFE)\n",
    "\n",
    "- Build a series of models\n",
    "- At each iteration, discard the least important feature according to the model. \n",
    "- Computationally expensive\n",
    "- Basic idea\n",
    "    - fit model\n",
    "    - find least important feature\n",
    "    - remove\n",
    "    - iterate."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### RFE algorithm \n",
    "\n",
    "1. Decide $k$, the number of features to select. \n",
    "2. Assign importances to features, e.g. by fitting a model and looking at `coef_` or `feature_importances_`.\n",
    "3. Remove the least important feature.\n",
    "4. Repeat steps 2-3 until only $k$ features are remaining.\n",
    "\n",
    "Note that this is **not** the same as just removing all the less important features in one shot!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [],
   "source": [
    "scaler = StandardScaler()\n",
    "X_train_scaled = scaler.fit_transform(X_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([16, 12, 19, 13, 23, 20, 10,  1,  9, 22,  2, 25,  5,  7, 15,  4, 26,\n",
       "       18, 21,  8,  1,  1,  1,  6, 14, 24,  3,  1, 17, 11])"
      ]
     },
     "execution_count": 53,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_selection import RFE\n",
    "\n",
    "# create ranking of features\n",
    "rfe = RFE(LogisticRegression(), n_features_to_select=5)\n",
    "rfe.fit(X_train_scaled, y_train)\n",
    "rfe.ranking_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[False False False False False False False  True False False False False\n",
      " False False False False False False False False  True  True  True False\n",
      " False False False  True False False]\n"
     ]
    }
   ],
   "source": [
    "print(rfe.support_)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "selected features:  ['mean concave points' 'worst radius' 'worst texture' 'worst perimeter'\n",
      " 'worst concave points']\n"
     ]
    }
   ],
   "source": [
    "print(\"selected features: \", cancer.feature_names[rfe.support_])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- How do we know what value to pass to `n_features_to_select`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "- Use `RFECV` which uses cross-validation to select number of features. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {
    "slideshow": {
     "slide_type": "-"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[False  True False  True False False  True  True  True False  True False\n",
      "  True  True False  True False False False  True  True  True  True  True\n",
      " False False  True  True False  True]\n",
      "['mean texture' 'mean area' 'mean concavity' 'mean concave points'\n",
      " 'mean symmetry' 'radius error' 'perimeter error' 'area error'\n",
      " 'compactness error' 'fractal dimension error' 'worst radius'\n",
      " 'worst texture' 'worst perimeter' 'worst area' 'worst concavity'\n",
      " 'worst concave points' 'worst fractal dimension']\n"
     ]
    }
   ],
   "source": [
    "from sklearn.feature_selection import RFECV\n",
    "\n",
    "rfe_cv = RFECV(LogisticRegression(max_iter=2000), cv=10)\n",
    "rfe_cv.fit(X_train_scaled, y_train)\n",
    "print(rfe_cv.support_)\n",
    "print(cancer.feature_names[rfe_cv.support_])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       0.365138\n",
       "score_time     0.002752\n",
       "test_score     0.943609\n",
       "train_score    1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "rfe_pipe = make_pipeline(\n",
    "    StandardScaler(),\n",
    "    RFECV(LogisticRegression(max_iter=2000), cv=10),\n",
    "    RandomForestClassifier(n_estimators=100, random_state=42),\n",
    ")\n",
    "\n",
    "pd.DataFrame(cross_validate(rfe_pipe, X_train, y_train, return_train_score=True)).mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- Slow because there is cross validation within cross validation \n",
    "- Not a big improvement in scores compared to all features on this toy case"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "skip"
    }
   },
   "source": [
    "<br><br>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Search and score\n",
    "\n",
    "- Define a **scoring function** $f(S)$ that measures the quality of the set of features $S$. \n",
    "- Now **search** for the set of features $S$ with the best score."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### General idea of search and score methods \n",
    "\n",
    "- Example: Suppose you have three features: $A, B, C$\n",
    "    - Compute **score** for $S = \\{\\}$\n",
    "    - Compute **score** for $S = \\{A\\}$\n",
    "    - Compute **score** for $S= \\{B\\}$\n",
    "    - Compute **score** for $S = \\{C\\}$\n",
    "    - Compute **score** for $S = \\{A,B\\}$    \n",
    "    - Compute **score** for $S = \\{A,C\\}$\n",
    "    - Compute **score** for $S = \\{B,C\\}$\n",
    "    - Compute **score** for $S = \\{A,B,C\\}$    \n",
    "- Return $S$ with the best score.  \n",
    "- How many distinct combinations we have to try out? "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Forward or backward selection \n",
    "\n",
    "- Also called wrapper methods\n",
    "- Shrink or grow feature set by removing or adding one feature at a time \n",
    "- Makes the decision based on whether adding/removing the feature improves the CV score or not"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/forward_selection.png)\n",
    "\n",
    "<!-- <img src='img/forward_selection.png' width=\"1000\" height=\"1000\" /> -->"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       1.586542\n",
       "score_time     0.002640\n",
       "test_score     0.933020\n",
       "train_score    1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 58,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.feature_selection import SequentialFeatureSelector\n",
    "\n",
    "pipe_forward = make_pipeline(\n",
    "    StandardScaler(),\n",
    "    SequentialFeatureSelector(LogisticRegression(max_iter=1000), \n",
    "                              direction=\"forward\", \n",
    "                              n_features_to_select='auto', \n",
    "                              tol=None),\n",
    "    RandomForestClassifier(n_estimators=100, random_state=42),\n",
    ")\n",
    "pd.DataFrame(\n",
    "    cross_validate(pipe_forward, X_train, y_train, return_train_score=True)\n",
    ").mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "fit_time       2.017284\n",
       "score_time     0.002670\n",
       "test_score     0.950627\n",
       "train_score    1.000000\n",
       "dtype: float64"
      ]
     },
     "execution_count": 59,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pipe_forward = make_pipeline(\n",
    "    StandardScaler(),\n",
    "    SequentialFeatureSelector(\n",
    "        LogisticRegression(max_iter=1000), \n",
    "                           direction=\"backward\", \n",
    "                           n_features_to_select=15),\n",
    "    RandomForestClassifier(n_estimators=100, random_state=42),\n",
    ")\n",
    "pd.DataFrame(\n",
    "    cross_validate(pipe_forward, X_train, y_train, return_train_score=True)\n",
    ").mean()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Other ways to search \n",
    "\n",
    "- Stochastic local search\n",
    "    - Inject randomness so that we can explore new parts of the search space\n",
    "    - Simulated annealing\n",
    "    - Genetic algorithms"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### Warnings about feature selection \n",
    "\n",
    "- A feature's relevance is only defined in the context of other features.\n",
    "    - Adding/removing features can make features relevant/irrelevant.\n",
    "- If features can be predicted from other features, you cannot know which one to pick. \n",
    "- Relevance for features does not have a causal relationship. \n",
    "- Don't be overconfident. \n",
    "    - The methods we have seen probably do not discover the ground truth and how the world really works.\n",
    "    - They simply tell you which features help in predicting $y_i$ for the data you have."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### (iClicker) Exercise 13.1 \n",
    "\n",
    "**iClicker cloud join link: https://join.iclicker.com/3DP5H**\n",
    "\n",
    "**Select all of the following statements which are TRUE.**\n",
    "\n",
    "- (A) Simple association-based feature selection approaches do not take into account the interaction between features.\n",
    "- (B) You can carry out feature selection using linear models by pruning the features which have very small weights (i.e., coefficients less than a threshold).\n",
    "- (C) Forward search is guaranteed to find the best feature set.  \n",
    "- (D) The order of features removed given by `rfe.ranking_` is the same as the order of original feature importances given by the model.  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "### (Optional) Problems with feature selection \n",
    "\n",
    "- The term 'relevance' is not clearly defined.\n",
    "- What all things can go wrong with feature selection?\n",
    "- Attribution: From CPSC 340. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Example: Is \"Relevance\" clearly defined?\n",
    "\n",
    "- Consider a supervised classification task of predicting whether someone has particular genetic variation (SNP)\n",
    "\n",
    "<img src='img/sex_mom_dad.png' width=\"600\" height=\"600\" />\n",
    "\n",
    "- True model: You almost have the same value as your biological mom."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- True model: You almost have the same value for SNP as your biological mom.\n",
    "    - (SNP = biological mom) with very high probability \n",
    "    - (SNP != biological mom) with very low probability \n",
    "    \n",
    "\n",
    "<img src='img/SNP.png' width=\"400\" height=\"400\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- What if \"mom\" feature is repeated?\n",
    "- Should we pick both? Should we pick one of them because it predicts the other? \n",
    "- Dependence, collinearity for linear models\n",
    "    - If a feature can be predicted from the other, don't know which one to pick. \n",
    "\n",
    "<img src='img/sex_mom_mom2_dad.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- What if we add (maternal) \"grandma\" feature?\n",
    "- Is it relevant? \n",
    "    - We can predict SNP accurately using this feature\n",
    "- Conditional independence\n",
    "    - But grandma is irrelevant given biological mom feature\n",
    "    - Relevant features may become irrelevant given other features\n",
    "\n",
    "<img src='img/sex_mom_dad_grandma.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- What if we do not know biological mom feature and we just have grandma feature\n",
    "- It becomes relevant now. \n",
    "    - Without mom feature this is the best we can do. \n",
    "- General problem (\"taco Tuesday\" problem)\n",
    "    - Features can become relevant due to missing information \n",
    "\n",
    "\n",
    "<img src='img/sex_dad_grandma.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- Are there any relevant features now?\n",
    "- They may have some common maternal ancestor.   \n",
    "- What if mom likes dad because they share SNP? \n",
    "- General problem (Confounding)\n",
    "    - Hidden features can make irrelevant features relevant.\n",
    "\n",
    "<img src='img/sex_dad.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- Now what if we have \"sibling\" feature? \n",
    "- The feature is relevant in predicting SNP but not the cause of SNP. \n",
    "- General problem (non causality)\n",
    "    - the relevant feature may not be causal \n",
    "\n",
    "<img src='img/sex_dad_sibling.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Is \"Relevance\" clearly defined?\n",
    "\n",
    "- What if you are given \"baby\" feature?\n",
    "- Now the sex feature becomes relevant. \n",
    "    - \"baby\" feature is relevant when sex == F\n",
    "    \n",
    "- General problem (context specific relevance)\n",
    "    - adding a feature can make an irrelevant feature relevant\n",
    "\n",
    "<img src='img/sex_dad_baby.png' width=\"600\" height=\"600\"/>\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Warnings about feature selection \n",
    "\n",
    "- A feature is only relevant in the context of other features.\n",
    "    - Adding/removing features can make features relevant/irrelevant.\n",
    "- Confounding factors can make irrelevant features the most relevant.\n",
    "- If features can be predicted from other other features, you cannot know which one to pick. \n",
    "- Relevance for features does not have a causal relationship. \n",
    "\n",
    "- Is feature selection completely hopeless?\n",
    "    - It is messy but we still need to do it. So we try to do our best! "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### General advice on finding relevant features\n",
    "\n",
    "- Try forward selection. \n",
    "- Try other feature selection methods (e.g., `RFE`, simulated annealing, genetic algorithms)\n",
    "- Talk to domain experts; they probably have an idea why certain features are relevant.\n",
    "- Don't be overconfident. \n",
    "    - The methods we have seen probably do not discover the ground truth and how the world really works.\n",
    "    - They simply tell you which features help in predicting $y_i$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "slideshow": {
     "slide_type": "slide"
    }
   },
   "source": [
    "#### Relevant resources \n",
    "- [Genome-wide association study](https://en.wikipedia.org/wiki/Genome-wide_association_study)\n",
    "- [sklearn feature selection](https://scikit-learn.org/stable/modules/feature_selection.html)\n",
    "- [PyData: A Practical Guide to Dimensionality Reduction Techniques](https://www.youtube.com/watch?v=ioXKxulmwVQ)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Slideshow",
  "kernelspec": {
   "display_name": "Python [conda env:cpsc330]",
   "language": "python",
   "name": "conda-env-cpsc330-py"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}