ML - Mistakes to avoid in Machine Learning/01_00_Mistakes to avoid in Machine Learning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Mistakes to avoid in Machine Learning"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Always check to avoid the following mistakes in ML.\n",
    "\n",
    "+ [ ] [Assuming Data is good to go](#1)\n",
    "+ [ ] [Neglecting to consult subject matter experts](#2)\n",
    "+ [ ] [Overtiffing your models](#3)\n",
    "+ [ ] [Not standardizing your data](#4)\n",
    "+ [ ] [Focusing on Wrong Factors](#5)\n",
    "+ [ ] [Data Leakage](#6)\n",
    "+ [ ] [Forgetting traditional statistics tools](#7)\n",
    "+ [ ] [Assuming Deployment is a breeze](#8)\n",
    "+ [ ] [Assuming Machine Learning is the answer](#9)\n",
    "+ [ ] [Developing in a silo](#10)\n",
    "+ [ ] [Not treating for imbalanced sampling](#11)\n",
    "+ [ ] [Interpreting your coefficients without properly treating for multicollinearity](#12)\n",
    "+ [ ] [Evaluating by accuracy alone](#13)\n",
    "+ [ ] [Giving overly technical presentations](#14)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='1'>1) Assuming Data is good to go</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualize your data\n",
    "+ use `describe()` functions in pandas\n",
    "+ use `pandas-profiling`\n",
    "\n",
    "### Check for Duplicate\n",
    "+ use pandas `duplicated` function\n",
    "+ use `drop_duplicates()`\n",
    "\n",
    "### Beware of Missing Values\n",
    "+ use `isnull().sum()` on a dataframe to determine missing values.\n",
    "+ use `.dropna()` to remove records with null values\n",
    "    + replace null values with `0` using `.fillna()` function.\n",
    "+ Estimate null values with **imputing** data\n",
    "    + use scikit learn's SimpleImputer\n",
    "    + fill missing values with mean, median, mode"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='2'>2) Neglecting to consult subject matter experts</a>\n",
    "\n",
    "+ SMEs can be Product Mangager or customers whom you are creating a model for\n",
    "+ always check with your customers for\n",
    "    + Are there any known issues with the data?\n",
    "    + Have there been any prior issues with modelling?\n",
    "    + Are there any common hang-ups?\n",
    "    + Are there any additional concerns\n",
    "+ get their feedbacks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='3'>3) Overtiffing your models</a>\n",
    "\n",
    "### **Overfitting**<br> \n",
    "When your model captures patterns in your training data too well - meaning it doesn't generalize well to unseen data.\n",
    "\n",
    "## **Preventing Overfitting** \n",
    "\n",
    "**Regularization:** Introducing a penalty for overly complex features that reduces - or eliminates - their weight in our model.\n",
    "\n",
    "Two common types of regularization include \n",
    "+ **Lasso or L1 regularization**\n",
    "+ **Ridge or L2 regularization**.\n",
    "\n",
    "\n",
    "Each of these will shrink the weights of coefficients in the model. But L1 can reduce the weight for some features to zero, thereby removing them entirely from the model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='4'>4) Not standardizing your data</a>\n",
    "\n",
    "\n",
    "### **Why do we need to scale features?**\n",
    "\n",
    "* Many machine learning techniques will incorrectly assign a **higher weight to features of a higher magnitude**.\n",
    "\n",
    "* There are wo common approaches for scaling.\n",
    "    + **MinMaxScaler**\n",
    "    + **StandardScaler**\n",
    "\n",
    "\n",
    "**NOTE: tree based algorithms don't require Scaling**\n",
    "\n",
    "-------\n",
    "\n",
    "## Min Max Scaler\n",
    "\n",
    "+ Min-max scaling involves scaling your feature to `a range between 0 and 1`, as defined by the `min and max of your feature`.\n",
    "+ is recommended when your algorithm `doesn't require assumptions about the distributions` of your variables, as in the case of KNN.\n",
    "\n",
    "\n",
    "## Standard Scaler\n",
    "\n",
    "+ The StandardScaler will scale features to be the `standard deviation from the mean for that feature`. Thus we have a `range of values both positive and negative`.\n",
    "+ this approach assumes a `bell curve distribution (Normal Distribution)` for your variables and it's most effective when it's the case.\n",
    "\n",
    "\n",
    "-------\n",
    "\n",
    "### Before we scale, we need to perform the train-test split.\n",
    "\n",
    "The reason we do scaling is that we will actually derive the scaling bounds from the training set, then apply it to test set.\n",
    "\n",
    "Remember in machine learning, it's important that anything our model learns must come from training set, not the test set."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "# <a name='5'>5)  Focusing on Wrong Factors</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes Datascientists hit the wall of performance after tuning and refining the model.\n",
    "\n",
    "If we want to avoid this outcome,\n",
    "+ take a step back and ask yourselves and other around you, is there any other datasets that the model can be benefit from?\n",
    "+ revisit the data that we collected at the start of the project.\n",
    "+ there is a good chance that incorporating more data sources into your model, we can significantly improve the model.\n",
    "\n",
    "--\n",
    "\n",
    "## Suggested Approach\n",
    "\n",
    "+ Make a **wish list** of the data you want. Then map this wishlist to the data sources that exist within your business.\n",
    "\n",
    "+ Find the **avaliable data sources** and incorporate **relevant new variables**. If they meet the feature selection criteria, measure their feature importance and overall impact on the model's performance.\n",
    "\n",
    "+ Reach out to **other data team**. Ask what data they would think to incorporate. Mostly data scientists work in silo, so approach other team who focus on other data sources. The other team may be able to provide us with starter script or query which can save us time on data prep. Put aside the bias on the data first and let the evaluation criteria be the judge.\n",
    "\n",
    "+ More is not always better. Adding more data may require additional work on feature selection and may lead to overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='6'>6)  Data Leakage</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **What is data leakage?** ##\n",
    "\n",
    "Data leakage can be thought of as anytime information from outside your training set enters your model.\n",
    "\n",
    "\n",
    "It is especially prevlant when working with time series data and in environments where there are data cleaniness issues. The end result maybe fool you into thinking your model generalizes much better than it really does.\n",
    "\n",
    "Mindful of this and remember if your results look too good to be true, pump the brakes and follow those important steps before sharing out the models results.\n",
    "\n",
    "\n",
    "## **How to detect and prevent data leakage** ##\n",
    "+ **Are any features surprisingly highly correlated with your target variable?**\n",
    "    + use corr() to find out\n",
    "+ Similarly, after training your model - **review the feature importance to see if anything stands out.**\n",
    "+ **If using time series data, train-test split along your date variable.**\n",
    "    + it is not appropriate to do a random test split as you normally would.\n",
    "    + sort by data first\n",
    "    + then split along the date variable.\n",
    "    \n",
    "+ When **Scaling**, fit your scaler to your training group only, then transform both training and test group\n",
    "\n",
    "+ when using **K-fold cross validation**, repeat the preprocessing steps within each fold separately to prevent data leakage.\n",
    "    + use the pipeline to handle preprocessing steps and use it via GridSearchCV, RandomizedSearchCV.\n",
    "    + https://towardsdatascience.com/pre-process-data-with-pipeline-to-prevent-data-leakage-during-cross-validation-e3442cca7fdc\n",
    "    \n",
    "![cv_pipeline.png](cv_pipeline.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### [INCORRECT WAY] AVOID SCALING and passing those data into CV. \n",
    "\n",
    "![avoidthis.png](avoidthis.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='7'>7)  Forgetting traditional statistics tools</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "famaliarize yourself with traditional regression techniques if we trying to explain the past, rather than trying to predict the future.\n",
    "\n",
    "## Regression Approach\n",
    "+ R-squared value\n",
    "+ Variable coefficients\n",
    "+ P-values\n",
    "+ Interpretability\n",
    "\n",
    "Always be mindful of statistics methods of when using Regression such as...\n",
    "+ treating Multicollinearityi if you are intent to interpreting the output.\n",
    "\n",
    "\n",
    "## A/B Testing\n",
    "+ uses t-tests\n",
    "+ Randomized Experiment: We determine if a treament yields any statistically significant impact in a randomized experiment.\n",
    "+ Casual Interference: this can yield the covted causal inference in which we can reasonably say X caused Y."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='8'>8)  Assuming Deployment is a breeze</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Depending on the usecase, deployment can be complex.\n",
    "\n",
    "\n",
    "## Start with the end in mind\n",
    "+ Plan your deployment startegy **at the beginning** of the project.\n",
    "    + this will help to illuminate the limitations you'll need to consider when you are creating the model. (example: if you are planning for real-time predictions in your deployment, check to see all the data you have will actually be available to generate predictions.\n",
    "+ Will you be scheduling batch predictions or predicting in real time?\n",
    "    + if so, you'll need to look into things like Flask and deploying APIs.\n",
    "+ What are the compute requirements?\n",
    "+ How will you monitor performance over time?\n",
    "+ Will you be updating and re-deploying your model? If so, How?\n",
    "+ Is your model driving behaviour that you intended it to?\n",
    "    + use A/B testing to evaluate its significance.\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='9'>9)  Assuming Machine Learning is the answer</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Access whether the project requires machine learning upfront.\n",
    "\n",
    "Use the **following criterias whether the machine learning is necessary and likely to succeed for your use case**:\n",
    "+ Do I have a large and diverse set of data to start with?\n",
    "+ How well-defined is the problem that I am trying to solve?\n",
    "+ Do I have a clear outcome that I am trying to predict?\n",
    "+ Do I have hypothesis?\n",
    "+ Will Quick Ad-hoc analysis or Full-fledged machine learning model require?\n",
    "    + often quick descriptive statistics can provide the insights.\n",
    "+ For classification problem, is your data label?\n",
    "    + prehaps data cleaning exercises is needed, before any modelling can be performed.\n",
    "+ If successful, will my results drive meaningful action?\n",
    "    + don't predict for the predicting sake"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='10'>10)  Developing in a silo</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With so much time spent heads down in your code, it's easy to lose perspective about some of the intangibles that go into a successful machine learning project. \n",
    "\n",
    "So avoid the mistake of developing in a silo through these tips:\n",
    "\n",
    "+ **Invite others** to look at your code.\n",
    "    + don't worry about being judged. Maybe you will get some recommendations or will even get questions that will prepare you to socialize your work to a broader audience\n",
    "+ Reach out to **established data scientists** in your field.\n",
    "+ Thoroughly **version and document** your code so it can be reproduced.\n",
    "+ **Communicate reguarly** with your subject matter experts.\n",
    "    + most importantly, regularly communicate your progress to your managers or customers and diplomatically share any roadblocks you're encountering. You want to avoid disappearing into your script after those initial meetings. Your customers may be left wondering, what's taking him so long? Or did she incorporate that feedback I gave her?\n",
    "    + don't share your script with nontechnical audiences, but **communicating your progress and initial findings through visualizations will help set expectations and create happier customers**. \n",
    "+ Take a break form the screen and **go for a walk**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='11'>11)  Not treating for imbalanced sampling</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## **Imbalanced Data**\n",
    "Encountered in a classification problem in which the **number of observations per class are disproportionately distributed.**\n",
    "\n",
    "\n",
    "## **How to treat for Imbalanced Data?**\n",
    "+ use `imbalanced-learn` (imblearn) package.\n",
    "+ it can provide various sampling techniques.\n",
    "\n",
    "-------\n",
    "\n",
    "# 1) Over-Sampling Approach\n",
    "\n",
    "\n",
    "## 1.1) naive approach known as Random Over-Sampling\n",
    "+ We will upsample our minority classes, that is sample with replacement until the number of observations is uniform across all classes.\n",
    "+ As we can imagine this approach should give us a pause depending on the scale of upsampling we'll be doing.\n",
    "+ `from imblearn.over_sampling import RandomOverSampler`\n",
    "\n",
    "## 1.2) another approach is SMOTE (Synthetic Minority Oversampling Technique)\n",
    "+ in the case, we generate new observations within the existing feature space over our minority classes.\n",
    "\n",
    "\n",
    "\n",
    "--------\n",
    "\n",
    "# 2) Under-Sampling Technique\n",
    "\n",
    "## 2.1) Naive approach to randomly under-sample our majority class\n",
    "+ this time we actually throwing out data in our majority class until the number of observations is uniform.\n",
    "+ `from imblearn.under_sampling import RandomUnderSampler`\n",
    "+ always check number of observations per class after resampled. **Because of the infrequency of our smallest minority class, we threw out a huge percentage**. If that is the case and we lost a lot of data, we might want to consider other methods for this kind of dataset (like `k-means` and `near-miss`)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='12'>12)  Interpreting your coefficients without properly treating for multicollinearity</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "+ **Multicollinearity** is when one predictor variable in your regression model can be accurately predicted from the others.\n",
    "\n",
    "-----\n",
    "\n",
    "## Use Logit (https://pypi.org/project/statsmodels/)\n",
    "\n",
    "+ `import statsmodels.api as sm`, `from statsmodels.tools.tools import add_constant` (provide full summary of the data)\n",
    "+ check for `R-Squared` value (Pseudo R-squ).\n",
    "+ check for `Coefficient` of independent variables (coef). These tell us how you can expect the likelihood of being one class to respond to changes in features.\n",
    "+ check for `P-values` which tell us the relative statistical significance of each variables. ( P>|z|)\n",
    "\n",
    "**Before you interpret those values, we need to understand if multicollinearity is present in our data.**\n",
    "\n",
    "Multicollinearity won't affect the quality of predictions in our model, but only our abiblity to intrepret individual coefficients in p-values. This brings us to `Variance Inflation Factor(VIF)`.\n",
    "\n",
    "-----\n",
    "## Variance Inflation Factor (VIF)\n",
    " \n",
    "+ **Variance Inflation Factor (VIF)** tells us the extent to which we have multicollinearity in our result\n",
    "+ a `factor of 5 to 10 or more is considered high` and tells us to be wary of the model coefficients.\n",
    "\n",
    "`from statsmodels.stats.outliers_influence import variance_inflation_factor`\n",
    "`vif = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]`\n",
    "\n",
    "+ To treat for multicollinearity, you can remove those variables with the high VIF. \n",
    "\n",
    "**Note this will affect the explanatory power of your overall model.**"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "-----\n",
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='13'>13)  Evaluating by accuracy alone</a>\n",
    "\n",
    "## **Beyond Accuracy**\n",
    "\n",
    "+ **Accuracy**: The share of all total predictions that were correct.\n",
    "    + As accuracy alone doesn't tell the whole story, check the following.\n",
    "    + **True Positive Rate (Sensitivity)**: this tell us how well our predictions in the positive subset of data.\n",
    "    + **True Negative Rate (Specificity)**: this tell us how well our predictions in the negative subset of data.\n",
    "+ **Recall** is the ability of the classifier to find all the positive samples\n",
    "+ **Precision** tells us how relevant our result is\n",
    "+ **F1 Score:** the weighted average of precision and recall. We want this to be as close to 1 as possible.\n",
    "\n",
    "------\n",
    "+ use **Confusion matrix** and **Classification Report**\n",
    "    + Confusion Matrix: this matrix will reveal how well our predictions line up with the actuals across the positive and negative subsets of the data.\n",
    "    \n",
    "------\n",
    "\n",
    "### ROC Curve and AUC\n",
    "\n",
    "+ A ROC curve is the most commonly used way to visualize the performance of a binary classifier\n",
    "+ AUC is the best way to summarise its performance in a single number.\n",
    "\n",
    "\n",
    "+ **ROC curve (receiving operator characteristic curve):**\n",
    "    + This is a great way to evaluate your model when you have actual predicted probabilities instead of just zeros and ones. \n",
    "    + The ROC curve allows you to see the true positive rate plotted against the false positive rate across varying thresholds for deeming a prediction positive or negative. \n",
    "    + The path of this curve tells us how well your model performs. \n",
    "    + And we can actually sum the area under this curve to give us a clean metric to work with. This is known as a **AUC, area under the curve**, and it varies from 0.5 to one. \n",
    "        + we can think of this as a **letter grade**. \n",
    "        + So 0.85 is like a B, but the actual interpretation depends on your use case. \n",
    "        \n",
    "Take note that accuracy as a metric will only tell you part of the picture, particularly when you have imbalanced data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# <a name='14'>14)  Giving overly technical presentations</a>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Art of story telling\n",
    "+ Use **Data Visualizations** instead of showing your code.\n",
    "+ stick to summaries of your results.\n",
    "+ Make data visualizations **relevant and easy to read**.\n",
    "+ **Lead with the results**. Not the technicalities.\n",
    "    + sell the orgianizaion's benfits like \"What time will be saved?\" Or value created above the current baseline.\n",
    "    + you can use confusion matrix to get your point across.\n",
    "+ Speak to how your approach **addresses the problem to be solved.**\n",
    "+ learn to use story telling techniques to convey why you work is going to have a big impact and drive success.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "------"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "venv-datascience",
   "language": "python",
   "name": "venv-datascience"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}