sa_gdp_matricpassrate.ipynb

{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "private_outputs": true,
      "provenance": [],
      "authorship_tag": "ABX9TyOf2p4vldkIMkae2+nLDplU",
      "include_colab_link": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    }
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "view-in-github",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/JonasGiven/GDP-PREDICTION-IN-2030-IN-SA/blob/main/sa_gdp_matricpassrate.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# **Predicting GDP Growth Based on Educational Pass Rates: An Analytical Approach Using Historical Data**"
      ],
      "metadata": {
        "id": "XxlkfuSJ1MJK"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Problem Statement**\n",
        "\n",
        "Despite significant investments in education, the relationship between educational outcomes and economic performance remains complex and not fully understood. This project aims to investigate the correlation between historical pass rates and GDP growth from 1995 to 2023. By analyzing these trends, the project seeks to develop a predictive model that can estimate future GDP changes based on variations in educational pass rates. Specifically, it aims to predict the impact on GDP if the pass rate increases by 10% by 2030.\n"
      ],
      "metadata": {
        "id": "ApcJz-Cl1O5O"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Context**\n",
        "\n",
        "Education is widely regarded as a key driver of economic development. A well-educated population is often linked to higher productivity, innovation, and overall economic growth. However, quantifying this relationship poses a challenge due to the myriad of influencing factors. This project leverages historical data on pass rates and GDP from 1995 to 2023 to explore this relationship in a structured and empirical manner. By employing machine learning techniques, the project will provide insights into how improvements in education can translate into economic gains. This analysis will aid policymakers and stakeholders in making informed decisions to foster both educational and economic advancements."
      ],
      "metadata": {
        "id": "vHbKgQqU1X7r"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Import and install necessary libraries**"
      ],
      "metadata": {
        "id": "NE39QMvb1urI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "!pip install mlflow"
      ],
      "metadata": {
        "id": "mRTlbyx11g91"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "import mlflow\n",
        "import os\n",
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import seaborn as sns\n",
        "import mlflow.sklearn\n",
        "from sklearn.ensemble import RandomForestRegressor\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.metrics import mean_squared_error, r2_score"
      ],
      "metadata": {
        "id": "uIjqAtmo23UX"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Load dataset**"
      ],
      "metadata": {
        "id": "QpUw-1Wz3cA1"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "df_gdp = pd.read_csv(\"https://raw.githubusercontent.com/JonasGiven/Datasets/main/SA_matricratepass_gdp%202024-07-04%2012_46_25.csv\")"
      ],
      "metadata": {
        "id": "C6XGBsqm3mjR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "df_gdp.head(10)"
      ],
      "metadata": {
        "id": "W1MiQa623_65"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Data exploration**"
      ],
      "metadata": {
        "id": "sEC5FUlxC58K"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Drop 2023 it's an outlier\n",
        "# Assuming df_gdp is your DataFrame containing GDP data\n",
        "df_gdp = df_gdp[df_gdp['Year'] != 2023]\n",
        "\n",
        "# Display the tail of the DataFrame to verify\n",
        "df_gdp.tail()"
      ],
      "metadata": {
        "id": "LkXI-oN5626T"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Box and whisker diagram of GDP and official matric pass rate\n",
        "# Create a figure with two subplots\n",
        "fig, axes = plt.subplots(1, 2, figsize=(14, 6))\n",
        "\n",
        "# Box plot for the official matric pass rate\n",
        "\n",
        "sns.boxplot(data=df_gdp, x='Official matric pass rate', ax=axes[0], color ='cyan')\n",
        "axes[0].set_title('Box and Whisker Plot for Official Matric Pass Rate')\n",
        "axes[0].set_xlabel('Official matric pass rate (%)')\n",
        "\n",
        "# Calculate quartiles and min/max for pass rate\n",
        "q1_pass = df_gdp['Official matric pass rate'].quantile(0.25)\n",
        "median_pass = df_gdp['Official matric pass rate'].median()\n",
        "q3_pass = df_gdp['Official matric pass rate'].quantile(0.75)\n",
        "min_pass = df_gdp['Official matric pass rate'].min()\n",
        "max_pass = df_gdp['Official matric pass rate'].max()\n",
        "\n",
        "# Annotate quartiles and min/max for pass rate below the plot\n",
        "axes[0].text(0.5, -0.15, f'Min: {min_pass:.2f}\\nQ1: {q1_pass:.2f}\\nMedian: {median_pass:.2f}\\nQ3: {q3_pass:.2f}\\nMax: {max_pass:.2f}',\n",
        "             transform=axes[0].transAxes, ha='center', va='top', fontsize=10, bbox=dict(facecolor='white', alpha=0.5))\n",
        "\n",
        "# Box plot for GDP in billion $\n",
        "sns.boxplot(data=df_gdp, x='GDP in billion $', ax=axes[1])\n",
        "axes[1].set_title('Box and Whisker Plot for GDP in Billion $')\n",
        "axes[1].set_xlabel('GDP (in billion $)')\n",
        "\n",
        "# Calculate quartiles and min/max for GDP\n",
        "q1_gdp = df_gdp['GDP in billion $'].quantile(0.25)\n",
        "median_gdp = df_gdp['GDP in billion $'].median()\n",
        "q3_gdp = df_gdp['GDP in billion $'].quantile(0.75)\n",
        "min_gdp = df_gdp['GDP in billion $'].min()\n",
        "max_gdp = df_gdp['GDP in billion $'].max()\n",
        "\n",
        "# Annotate quartiles and min/max for GDP below the plot\n",
        "axes[1].text(0.5, -0.15, f'Min: {min_gdp:.2f}\\nQ1: {q1_gdp:.2f}\\nMedian: {median_gdp:.2f}\\nQ3: {q3_gdp:.2f}\\nMax: {max_gdp:.2f}',\n",
        "             transform=axes[1].transAxes, ha='center', va='top', fontsize=10, bbox=dict(facecolor='white', alpha=0.5))\n",
        "\n",
        "\n",
        "# Display the plots\n",
        "plt.tight_layout()\n",
        "plt.show()"
      ],
      "metadata": {
        "cellView": "form",
        "id": "aW0Lww5GOHWx"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Observational Report\n",
        "\n",
        "The box and whisker plots reveal important insights about the official matric pass rate and GDP. For the pass rate, the minimum was 47.40%, with Q1 at 61.43%, a median of 69.60%, Q3 at 75.27%, and a maximum of 81.30%. This shows a generally high pass rate, with most values clustered around the median to upper quartile range. For GDP, the minimum was \\$129.09 billion, Q1 was \\$171.05 billion, the median was \\$326.67 billion, Q3 was \\$392.22 billion, and the maximum was $458.20 billion. The GDP values indicate significant economic growth over the years, with a wide range from the lower quartile to the maximum. These observations suggest a positive correlation where higher pass rates are associated with higher GDP, reflecting the potential impact of education on economic performance."
      ],
      "metadata": {
        "id": "WVFqv5L7WVUO"
      }
    },
    {
      "source": [
        "# @title Official matric pass rate vs GDP in billion $\n",
        "df_gdp.plot(kind='scatter', x='Official matric pass rate', y='GDP in billion $', s=32, alpha=.8,color = 'green', marker = '+')\n",
        "plt.gca().spines[['top', 'right',]].set_visible(False)"
      ],
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "metadata": {
        "cellView": "form",
        "id": "rMp9SCt1489w"
      }
    },
    {
      "source": [
        "# @title Year vs GDP in billion $\n",
        "\n",
        "def _plot_series(series, series_name, series_index=0, color ='red'):\n",
        "  palette = list(sns.palettes.mpl_palette('Dark2'),)\n",
        "  xs = series['Year']\n",
        "  ys = series['GDP in billion $']\n",
        "\n",
        "  plt.plot(xs, ys, label=series_name, color=palette[series_index % len(palette)])\n",
        "\n",
        "fig, ax = plt.subplots(figsize=(10, 5.2), layout='constrained')\n",
        "df_sorted = df_gdp.sort_values('Year', ascending=True)\n",
        "_plot_series(df_sorted, '')\n",
        "sns.despine(fig=fig, ax=ax)\n",
        "plt.xlabel('Year')\n",
        "_ = plt.ylabel('GDP in billion $')"
      ],
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "metadata": {
        "cellView": "form",
        "id": "7v1pI9x54q1c"
      }
    },
    {
      "source": [
        "# @title Year vs Official matric pass rate\n",
        "def _plot_series(series, series_name, series_index=0, color = 'green'):\n",
        "  palette = list(sns.palettes.mpl_palette('Dark2'))\n",
        "  xs = series['Year']\n",
        "  ys = series['Official matric pass rate']\n",
        "\n",
        "  plt.plot(xs, ys, label=series_name, color=palette[series_index % len(palette)])\n",
        "\n",
        "fig, ax = plt.subplots(figsize=(10, 5.2), layout='constrained')\n",
        "df_sorted = df_gdp.sort_values('Year', ascending=True)\n",
        "_plot_series(df_sorted, '')\n",
        "sns.despine(fig=fig, ax=ax)\n",
        "plt.xlabel('Year')\n",
        "_ = plt.ylabel('Official matric pass rate')"
      ],
      "cell_type": "code",
      "execution_count": null,
      "outputs": [],
      "metadata": {
        "cellView": "form",
        "id": "EDq17whI4fwA"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title GDP vs official matric pass rate from 1995 - 2022\n",
        "# Create figure and axis objects\n",
        "fig, ax1 = plt.subplots(figsize=(10, 6))\n",
        "\n",
        "# Plot Pass Rate on the left y-axis\n",
        "\n",
        "ax1.plot(df_gdp['Year'], df_gdp['Official matric pass rate'], label='Official matric pass rate', marker='s', color='green')\n",
        "ax1.set_xlabel('Year')\n",
        "ax1.set_ylabel('Official matric pass rate\t', color='green')\n",
        "ax1.tick_params(axis='y', labelcolor='green')\n",
        "\n",
        "# Create a second y-axis for GDP\n",
        "ax2 = ax1.twinx()\n",
        "ax2.plot(df_gdp['Year'], df_gdp['GDP in billion $'], label='GDP (in billion $)', marker='o', color='red')\n",
        "ax2.set_ylabel('GDP (in billion $)', color='red')\n",
        "ax2.tick_params(axis='y', labelcolor='red')\n",
        "\n",
        "# Title and legend\n",
        "plt.title('GDP vs official matric pass rate over Years')\n",
        "ax1.legend(loc='upper left')\n",
        "ax2.legend(loc='upper right')\n",
        "\n",
        "# Grid\n",
        "ax1.grid(True)\n",
        "\n",
        "plt.show()"
      ],
      "metadata": {
        "cellView": "form",
        "id": "CokYBRsRARDp"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "### Observations\n",
        "\n",
        "My analysis reveals a clear correlation between South Africa’s GDP and the official matric pass rate over the years. When the pass rate increases, the GDP also tends to rise, and when the pass rate drops, the GDP tends to fall. For instance, from around 1998 to 2003, there was a significant increase in the pass rate, accompanied by a rise in GDP. However, from 2003 to 2010, the pass rate declined sharply, which slowed the GDP growth. Conversely, from 2009 to 2014, we observed a sharp increase in the pass rate, and the GDP grew as well. This pattern continues, with the pass rate drop in 2020 and 2021 due to COVID-19 causing a slight decrease in GDP. These trends suggest a strong link between educational outcomes and economic performance in South Africa."
      ],
      "metadata": {
        "id": "l5ZLSZs9JUtr"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Model training**"
      ],
      "metadata": {
        "id": "U5iuulmhTEJs"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# Data labeling\n",
        "y = df_gdp['GDP in billion $']\n",
        "X = df_gdp[['Year', 'Official matric pass rate']]\n",
        "\n",
        "# Data splitting\n",
        "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=100)\n",
        "\n",
        "# Model building\n",
        "rf = RandomForestRegressor(max_depth=2, random_state=100)\n",
        "rf.fit(X_train, y_train)"
      ],
      "metadata": {
        "id": "SHv8hW67TJ9D"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Ensure that X_train and X_test are DataFrames with the correct column names\n",
        "print(\"Training features:\", X_train.columns)\n",
        "print(\"Testing features:\", X_test.columns)"
      ],
      "metadata": {
        "id": "kFUzpdV5e9bH"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Apply model to make predictions**"
      ],
      "metadata": {
        "id": "iB3fwBT6UjJd"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "y_rf_train_pred = rf.predict(X_train)\n",
        "y_rf_test_pred = rf.predict(X_test)"
      ],
      "metadata": {
        "id": "k3xzWd_dUo-G"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "# Print the predictions\n",
        "print(\"Training predictions:\", y_rf_train_pred)\n",
        "print(\"Testing predictions:\", y_rf_test_pred)"
      ],
      "metadata": {
        "collapsed": true,
        "id": "_xNpNLlXfKXz"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "#@title Predicted gdp in 2030 if the official pass rate increase by 10% from 2022\n",
        "#in 2022 the official pass rate is 80.1% in 2030 it will be 90.1%\n",
        "\n",
        "# Define the input data for prediction\n",
        "input_data = pd.DataFrame({\n",
        "    'Year': [2030],\n",
        "    'Official matric pass rate': [90.1]\n",
        "})\n",
        "\n",
        "# Predict\n",
        "pred_GDP_2030 = rf.predict(input_data)\n",
        "\n",
        "print(\"Predicted GDP for 2030:\", pred_GDP_2030[0])"
      ],
      "metadata": {
        "cellView": "form",
        "id": "ahKGDMoGashR"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Evaluate model perfomance**"
      ],
      "metadata": {
        "id": "2U-UiT8UU3fp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "rf_train_mse = mean_squared_error(y_train, y_rf_train_pred)\n",
        "rf_train_r2 = r2_score(y_train, y_rf_train_pred)\n",
        "\n",
        "rf_test_mse = mean_squared_error(y_test, y_rf_test_pred)\n",
        "rf_test_r2 = r2_score(y_test, y_rf_test_pred)"
      ],
      "metadata": {
        "id": "VxI0bbxDU-QL"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## **Configure and run mlflow**"
      ],
      "metadata": {
        "id": "BTYo2KfFVVUV"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "# @title mlflow, dagshub\n",
        "\n",
        "os.environ['MLFLOW_TRACKING_USERNAME'] = 'Jonas'\n",
        "os.environ['MLFLOW_TRACKING_PASSWORD'] = '6a23abfadd67ec688cfc99d274ca11cbec50bb1a'\n",
        "os.environ['MLFLOW_TRACKING_PROJECTNAME'] = 'ml002_SA_matricpassrate_gdp'\n",
        "mlflow.set_tracking_uri('https://www.dagshub.com/Jonas/ml002_SA_matricpassrate_gdp.mlflow')\n",
        "\n",
        "# Log results with MLflow\n",
        "if mlflow.active_run():\n",
        "    mlflow.end_run()\n",
        "\n",
        "with mlflow.start_run(run_name='gdp_matricpassrate_rf'):\n",
        "\n",
        "    # Log parameters\n",
        "    mlflow.log_param('train_test_split', 0.2)\n",
        "    mlflow.log_param('input_data', 'SA_matricratepass_gdp')\n",
        "    mlflow.log_param('random_state', 100)\n",
        "    mlflow.log_param('model', 'random forest regressor')\n",
        "    mlflow.log_param('max_depth', 2)\n",
        "\n",
        "    # Log metrics\n",
        "    mlflow.log_metric('train_mse', rf_train_mse)\n",
        "    mlflow.log_metric('train_r2', rf_train_r2)\n",
        "    mlflow.log_metric('test_mse', rf_test_mse)\n",
        "    mlflow.log_metric('test_r2', rf_test_r2)\n",
        "    mlflow.log_metric('pred_GDP_2030', pred_GDP_2030)\n",
        "\n",
        "    # Log model\n",
        "    mlflow.sklearn.log_model(rf, 'model')\n",
        "\n",
        "    # Plotting\n",
        "    plt.figure(figsize=(5, 5))\n",
        "    plt.scatter(y_train, y_rf_train_pred, c=\"#7CAE00\", alpha=0.3)  # alpha makes regions that are highly overlapping darker\n",
        "\n",
        "    z = np.polyfit(y_train, y_rf_train_pred, 1)\n",
        "    p = np.poly1d(z)\n",
        "\n",
        "    plt.plot(y_train, p(y_train), '#F8766D')\n",
        "    plt.ylabel('Predicted GDP')\n",
        "    plt.xlabel('Experimental GDP')\n",
        "    plt.title('Experimental vs Predicted GDP')\n",
        "\n",
        "    # Save plot\n",
        "    plot_filename = 'rf_experimental_predicted.png'\n",
        "    plt.savefig(plot_filename)\n",
        "    mlflow.log_artifact(plot_filename)\n",
        "    plt.show()\n",
        "\n",
        "print('MLflow run completed successfully')"
      ],
      "metadata": {
        "cellView": "form",
        "id": "RNNDECE_VaJ5"
      },
      "execution_count": null,
      "outputs": []
    }
  ]
}