Student_MLE_MiniProject_Deep_Learning.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# Mini Project: Deep Learning with Keras\n",
        "\n",
        "In this mini-project we'll be building a deep learning classifier using Keras to predict income from the popular [Adult Income dataset](http://www.cs.toronto.edu/~delve/data/adult/adultDetail.html).\n",
        "\n",
        "Predicting income from demographic and socio-economic information is an important task with real-world applications, such as financial planning, market research, and social policy analysis. The Adult dataset, sometimes referred to as the \"Census Income\" dataset, contains a vast amount of anonymized data on individuals, including features such as age, education, marital status, occupation, and more. Our objective is to leverage this data to train a deep learning model that can effectively predict whether an individual's income exceeds $50,000 annually or not.\n",
        "\n",
        "Throughout this Colab, we will walk you through the entire process of building a deep learning classifier using Keras, a high-level neural network API that runs on top of TensorFlow. Keras is known for its user-friendly and intuitive interface, making it an excellent choice for both beginners and experienced deep learning practitioners.\n",
        "\n",
        "Here's a brief outline of what we will cover in this mini-project:\n",
        "\n",
        "1. **Data Preprocessing:** We will start by loading and exploring the Adult dataset.\n",
        "\n",
        "2. **Building the Deep Learning Model:** We will construct a neural network using Keras, where we'll dive into understanding the key components of a neural network, including layers, activation functions, and optimization algorithms.\n",
        "\n",
        "3. **Model Training:** With our model architecture in place, we will split the data into training and validation sets and train the neural network on the training data. We will monitor the training process to prevent overfitting and enhance generalization.\n",
        "\n",
        "4. **Model Evaluation:** After training, we'll assess the performance of our model on the test dataset.\n",
        "\n",
        "By the end of this tutorial, you will not only have a functional deep learning classifier for income prediction but also gain valuable insights into how to leverage the power of neural networks for solving real-world classification tasks.\n"
      ],
      "metadata": {
        "id": "fyXucUekO19i"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install scikeras"
      ],
      "metadata": {
        "id": "rAGzXpBhHLPJ"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "kLWR1DfQPakn"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import OneHotEncoder, LabelEncoder\n",
        "from sklearn.preprocessing import MinMaxScaler\n",
        "from sklearn.compose import ColumnTransformer\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import roc_auc_score\n",
        "from sklearn.metrics import RocCurveDisplay\n",
        "from keras.models import Sequential\n",
        "from keras.layers import Dense\n",
        "from scikeras.wrappers import KerasClassifier\n",
        "from sklearn.pipeline import Pipeline"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can download the Adult data from the link [here](https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data).\n",
        "\n",
        "Here are your tasks:\n",
        "\n",
        "  1. Load the Adult data into a Pandas Dataframe.\n",
        "  2. Ensure the dataset has properly named columns. If the columns are not read in, assign them by referencing the dataset documentation.\n",
        "  3. Display the first five rows of the dataset."
      ],
      "metadata": {
        "id": "5ymxgnyq86hE"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "DATA_PATH = 'https://archive.ics.uci.edu/ml/machine-learning-databases/adult/adult.data'\n",
        "\n",
        "# Download the dataset and load it into a pandas DataFrame"
      ],
      "metadata": {
        "id": "QmwdQy7pShig"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Display the first few rows of the DataFrame"
      ],
      "metadata": {
        "id": "X1wSIzVtPrfL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "If you're not already familiar with the Adult dataset, it's important to do some exploratory data analysis.\n",
        "\n",
        "Here are your tasks:\n",
        "\n",
        "  1. Do exploratory data analysis to give you some better intuition for the dataset. This is a bit open-ended. How many rows/columns are there? How are NULL values represented? What's the percentage of positive cases in the dataset?\n",
        "\n",
        "  2. Drop all rows with NULL values.\n",
        "\n",
        "  3. Use Scikit-Learn's [LabelEncoder](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.LabelEncoder.html) to convert the `income` column with a data type string to a binary variable."
      ],
      "metadata": {
        "id": "5fHLuKZl9ivm"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Do some exploratory analysis. How many rows/columns are there? How are NULL\n",
        "# values represented? What's the percentrage of positive cases in the dataset?"
      ],
      "metadata": {
        "id": "fc_s4kRKTloe"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Find all NULL values and drop them"
      ],
      "metadata": {
        "id": "pZW7GRw3P0dT"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Use Scikit-Learn's LabelEncoder to convert the income column with a data type\n",
        "# string to a binary variable."
      ],
      "metadata": {
        "id": "BZ_mJT_DLZ-L"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "1. Split the data into training and test sets. Remember not to include the label you're trying to predict, `income`, as a column in your training data."
      ],
      "metadata": {
        "id": "ibK0DxJsA1JH"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Split dataset into training and test sets"
      ],
      "metadata": {
        "id": "1whzL6K7J-zq"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "In machine learning, the Receiver Operating Characteristic (ROC) curve and the Area Under the Curve (AUC) metric are commonly used to evaluate the performance of binary classification models. These are valuable tools for understanding how well a model can distinguish between the positive and negative classes in a classification problem.\n",
        "\n",
        "Let's break down each concept:\n",
        "\n",
        "1. ROC Curve:\n",
        "The ROC curve is a graphical representation of a binary classifier's performance as the discrimination threshold is varied. It is created by plotting the True Positive Rate (TPR) against the False Positive Rate (FPR) at different threshold values. Here's how these rates are calculated:\n",
        "\n",
        "- True Positive Rate (TPR), also called Sensitivity or Recall, measures the proportion of actual positive instances that are correctly identified by the model:\n",
        "   TPR = True Positives / (True Positives + False Negatives)\n",
        "\n",
        "- False Positive Rate (FPR) measures the proportion of actual negative instances that are incorrectly classified as positive by the model:\n",
        "   FPR = False Positives / (False Positives + True Negatives)\n",
        "\n",
        "The ROC curve is useful because it shows how well a classifier can trade off between sensitivity and specificity across different threshold values. The ideal ROC curve hugs the top-left corner, indicating a high TPR and low FPR, meaning the classifier is excellent at distinguishing between the two classes.\n",
        "\n",
        "2. AUC (Area Under the Curve):\n",
        "The AUC is a scalar metric derived from the ROC curve. It represents the area under the ROC curve, hence its name. The AUC ranges from 0 to 1, where 0 indicates a very poor classifier (always predicting the opposite class) and 1 signifies a perfect classifier (making all correct predictions).\n",
        "\n",
        "The AUC metric is beneficial because it provides a single value to summarize the classifier's overall performance across all possible threshold values. It is particularly useful when dealing with imbalanced datasets, where one class significantly outnumbers the other. In such cases, accuracy alone might not be a reliable evaluation metric, and AUC can provide a more robust performance measure.\n",
        "\n",
        "A quick rule of thumb for interpreting AUC values:\n",
        "- AUC ≈ 0.5: The model performs no better than random guessing.\n",
        "- 0.5 < AUC < 0.7: The model has poor to fair performance.\n",
        "- 0.7 < AUC < 0.9: The model has good to excellent performance.\n",
        "- AUC ≈ 1: The model is close to or has a perfect performance."
      ],
      "metadata": {
        "id": "HMsXM6B_BX5k"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here are your tasks:\n",
        "\n",
        "  1. Use Scikit-Learn's [roc_auc_score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.roc_auc_score.html) to calculate the AUC score for a method that always predicts the majority class.  "
      ],
      "metadata": {
        "id": "NDGgBVEKEYKi"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use Scikit-Learn's roc_auc_score to calculate the AUC score for a method that\n",
        "# always predicts the majority class."
      ],
      "metadata": {
        "id": "s00Xs8bqUZnn"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Now, let's do a little feature engineering.\n",
        "\n",
        "1. Use Scikit-Learn's [ColumnTransformer](https://scikit-learn.org/stable/modules/generated/sklearn.compose.ColumnTransformer.html) to apply One Hot Encoding to the categorical variables in `workclass`, `education`, `marital-status`, `occupation`, `relationship`, 'race', `sex`, and `native-country`. Also, apply [MinMaxScaler](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.MinMaxScaler.html) to the remaining continuous features. How many columns will the dataframe have after these columns transformations are applied?"
      ],
      "metadata": {
        "id": "uWSiYNarF2t_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Use Scikit-Learn's ColumnTransformer to apply One Hot Encoding to the\n",
        "# categorical variables in workclass, education, marital-status, occupation,\n",
        "# relationship, 'race', sex, and native-country. #Also, apply MinMaxScaler to\n",
        "# the remaining continuous features."
      ],
      "metadata": {
        "id": "4DybgGJyW-3Q"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# How many columns will the dataframe have after these columns transformations are applied?"
      ],
      "metadata": {
        "id": "emfaqHwvKfLU"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "Keras is an open-source deep learning library written in Python. It was developed to provide a user-friendly, high-level interface for building and training neural networks. The library was created by François Chollet and was first released in March 2015 as part of the Deeplearning4j project. Later, it became part of the TensorFlow ecosystem and is now the official high-level API for TensorFlow.\n",
        "\n",
        "Keras is designed to be modular, user-friendly, and easy to extend. It allows researchers and developers to quickly prototype and experiment with various deep learning models. One of the primary goals of Keras is to enable fast experimentation, making it simple to build and iterate on different architectures.\n",
        "\n",
        "Key features of Keras include:\n",
        "\n",
        "1. User-friendly API: Keras provides a simple and intuitive interface for defining and training deep learning models. Its design philosophy focuses on ease of use and clarity of code.\n",
        "\n",
        "2. Modularity: Models in Keras are built as a sequence of layers, and users can easily stack, merge, or create complex architectures using a wide range of predefined layers.\n",
        "\n",
        "3. Extensibility: Keras allows users to define custom layers, loss functions, and metrics. This flexibility enables researchers to experiment with new ideas and algorithms seamlessly.\n",
        "\n",
        "4. Backends: Initially, Keras supported multiple backends, including TensorFlow, Theano, and CNTK. However, as of TensorFlow version 2.0, TensorFlow has become the primary backend for Keras.\n",
        "\n",
        "5. Multi-GPU and distributed training: Keras supports training models on multiple GPUs and in distributed computing environments, making it suitable for large-scale experiments.\n",
        "\n",
        "6. Pre-trained models: Keras includes a collection of pre-trained models for common tasks, such as image classification (e.g., VGG, ResNet, MobileNet) and natural language processing (e.g., Word2Vec, GloVe).\n",
        "\n",
        "The integration of Keras into TensorFlow as its official high-level API has solidified its position as one of the most popular deep learning libraries in the machine learning community. Its ease of use and versatility have contributed to its widespread adoption in both academia and industry for a wide range of deep learning tasks."
      ],
      "metadata": {
        "id": "AtoqTz5rGuET"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Here are your tasks:\n",
        "\n",
        "1. Create your own model in Keras to predict income in the Adult training data. Remember, it's always better to start simple and add complexity to the model if necessary. What's a good loss function to use?\n",
        "\n",
        "2. Keras can be integrated with Scitkit-Learn using a wrapper. Use the [KerasClassifier wrapper](https://adriangb.com/scikeras/stable/generated/scikeras.wrappers.KerasClassifier.html) to integrate your Keras model with the ColumnTransformer from previous steps using a [Pipeline](https://scikit-learn.org/stable/modules/generated/sklearn.pipeline.Pipeline.html) object.\n",
        "\n",
        "3. Fit your model.\n",
        "\n",
        "4. Calculate the AUC score of your model on the test data. Does the model predict better than random?\n",
        "\n",
        "5. Generate an ROC curve for your model using [RocCurveDisplay](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.RocCurveDisplay.html). What would the curve look like if all your predictions were randomly generated? What would the curve look like if it you had a perfect model?"
      ],
      "metadata": {
        "id": "HVUa0h83HU24"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Define the Keras model"
      ],
      "metadata": {
        "id": "h2xIpLlXQEcx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create a Keras classifier"
      ],
      "metadata": {
        "id": "Rz-m2LhrQGud"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Create the scikit-learn pipeline"
      ],
      "metadata": {
        "id": "VKxkil7QQJ6n"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Fit the pipeline on the training data"
      ],
      "metadata": {
        "id": "25O8ZLleGQnk"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Calculate the AUC score of your model on the test data.\n",
        "# Does the model predict better than random?"
      ],
      "metadata": {
        "id": "SLcNQGVqNYbB"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Generate an ROC curve for your model."
      ],
      "metadata": {
        "id": "prJG9pr7PYIc"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}