session-three/session-three-project-social-sciences-questions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "33961a91-219f-4864-bdc5-9a1bf7b46526",
   "metadata": {},
   "source": [
    "# Python for Data Science Project Session 3: Social Sciences"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14d8ea0d-1f5a-4de0-a790-a6f225ac593d",
   "metadata": {},
   "source": [
    "The combination of social sciences and maths is not really any social sciences kid's fave combo (perhaps besides the econ kids) ... which is why NumPy is a useful tool to help simplfy our lives!  \n",
    "\n",
    "We will cover two sub-projects in this workbook.  Firstly, we will look at happiness data, practicing techniques to manage and alter arrays.  Secondly, we will look at population growth modelling for India, making use of NumPy's mathematical operations to extrapolate the data.  We will then plot these extrapolation outcomes - a nice little refresher on Session 2 data visualisation."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "96eaa88a-3f49-4052-a7e8-0b7d99e857c1",
   "metadata": {},
   "source": [
    "Start by importing the following packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "00f01690-7a5c-4593-bb67-4452d0842af6",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "import matplotlib.pyplot as plt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ce7719b-6254-4974-b94c-a296576807a5",
   "metadata": {},
   "source": [
    "## Section 1: Happiness data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcbcd2b0-fefd-452f-ab3f-d653ed441794",
   "metadata": {},
   "source": [
    "Import the data file `Happiness_survey_2019.csv` and name the dataset `happy_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4796ded-e658-40b1-88a3-27d09a0775a6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c11dde86-01f9-4d39-8908-363247ae3b70",
   "metadata": {},
   "source": [
    "We will now get the dataframe ready to convert into an array.  We will be analysing the data on a global scale and doing our own data analysis, so will hence  drop the `Overall rank` column.  What other column do you also need to drop in order for the array to be viable?  Do this below.\n",
    "\n",
    "Hint: look at datatypes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "60f49f8b-f1e1-443f-b5b7-6c46b66e64df",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "affc2fab-4ba5-4b01-bbed-4abd07e331ea",
   "metadata": {},
   "source": [
    "Now that the dataframe is ready, convert it into an array.  Name this array `happy`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "fdd02edd-9444-4cd0-acf7-0b6272dc745b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "77f6419f-0c03-4d77-9a31-4a262e2b6dc3",
   "metadata": {},
   "source": [
    "When analysing data, it is important to get a general understanding of the distribution of the variables in question. Calculate the mean of each variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae4ca4b2-d0a7-4c36-a1ec-c0b4d13acac1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "54e838fa-458c-4393-86e4-ccbbf2e03291",
   "metadata": {},
   "source": [
    "The data was already ordered from the highest to lowest happiness score, the former being 7.769, the latter being 2.853.  Therefore, a mean happiness score of 5.407 seems to indicate the data is upwardly skewed.  \n",
    "\n",
    "A reason for this may be the divide between developping and developped nations.  In order to investigate this further, we will use a second data set called `Country_income_category.csv`.  Upload this and call it `income_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ccabbebd-6fa5-446b-91eb-d336c96d1301",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "cb6c8e7e-8527-4fa1-aa57-95ac8af4f666",
   "metadata": {},
   "source": [
    "The countries in this data set have already been ordered to match the countries of the happiness survey.  The key is the following:\n",
    "\n",
    "* High income country = 3\n",
    "* Upper middle income country = 2\n",
    "* Lower middle income country = 1\n",
    "* Low income country = 0\n",
    "\n",
    "First, do the necessary action to convert this data set into an array called `income` (don't forget about data types). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0026da73-a984-4d97-9611-71aee12559ed",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c44c79d4-5ca7-416c-9207-a81deddacbd2",
   "metadata": {},
   "source": [
    "Now, combine the `income` array relating to country income categories with the array `happy`.  Name this new array `data`.\n",
    "\n",
    "Hint: You cannot use a traditional concatenate method as the arrays have different dimensions.  Look at the documentation for `hstack` or `column_stack` below.\n",
    "\n",
    "* hstack = https://numpy.org/doc/stable/reference/generated/numpy.hstack.html\n",
    "* column stack = https://numpy.org/doc/stable/reference/generated/numpy.column_stack.html#numpy.column_stack"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3affc2cc-3cb3-496d-89e4-85eed57e6fe7",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "94c24754-c602-46b4-9667-a08285b1f351",
   "metadata": {},
   "source": [
    "Now, we will split the large data matrix we have into 4 subset matrices so that each one is grouped by the country income category. \n",
    "\n",
    "To do this, we will use the `np.split` function.  This requires distinct indices at which to split the array.  In order to do so, we will sort the `data` array by the income categorisation column so that it is in ascending order.  Then, we will identify which indices said income categorisation column changes values (from 0 to 1, 1 to 2 etc).  Using this info, we will split the array into 4 sub-matrices.  We will do this step by step below based on the following example = https://stackoverflow.com/questions/31863083/python-split-numpy-array-based-on-values-in-the-array\n",
    "\n",
    "First, we are going to use `argsort` to sort the `data` array by the income categories (0 = low income, 1 = lower mid etc).  Sorting the income category column in acending order will mean that all the countries will be grouped by their associated categorical values (zeroes together, then ones, twos, and finally threes).  Name this array `data_sorted`, and refer to the following example for guidance = https://www.kite.com/python/answers/how-to-sort-the-rows-of-a-numpy-array-by-a-column-in-python\n",
    "\n",
    "Hint: the income category info is in the last column of the array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8dde057e-a65c-4783-bd2e-30cbce909a80",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c36229ba-a85a-4c44-87c6-fc0804066975",
   "metadata": {},
   "source": [
    "Now, using `np.diff(arr[index])` we will identify the indices at which the `data_sorted` array changes income category."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d9b11fd7-c90f-4a1e-9e50-7598344e90ff",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "cc99cef5-365d-4f5d-8195-1aae151dd6b3",
   "metadata": {},
   "source": [
    "The array above has a \"1\" wherever the country income category changes value in the sorted array.  This therefore indicates a change from 0 to 1, or 1 to 2 etc.  \n",
    "\n",
    "Using `np.where(index)[0]+1`, we will now identify the indices for the points at which the value changes, so that we can know where to split our array. The `[0]+1` adds 1 to it because as per how indexing works, the actual index of an item is one more than the returned index."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "be1b0d2a-8dd5-4427-b402-61698b4445f6",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c5566237-ba64-4e85-8e48-be3907ae881b",
   "metadata": {},
   "source": [
    "We now have all the elements we need to split our array into 4 sub-matrices using `np.array_split(arr, index)`.\n",
    "\n",
    "Do so below, once again reffering to the example.  Call this new matrix `data_split`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b29ef105-1d80-404c-a222-999b4d949b03",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "e75bcbe9-f646-4505-b8b3-911c32fc0510",
   "metadata": {},
   "source": [
    "Now that we have our 4 distinct matrices, we can find the mean values per country income category.  Do so using a for loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b297b034-d97d-4baf-bf0e-cfd79ac12ba0",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d186e680-9c7c-4a8c-b685-39c48ad96209",
   "metadata": {},
   "source": [
    "As can be seen, there is a distinct difference between the means in each income category group.  How does the previous overall mean compare to this breakdown? What conclusions can you draw regarding developement level and mean happiness?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d700a344-aa28-4421-8844-9116f6dd966e",
   "metadata": {},
   "source": [
    "## Section 2 : Population growth model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78bb8ac2-b3f5-47d3-81c8-52d5d86eefa1",
   "metadata": {},
   "source": [
    "In this second section, we will look at India's population data and attempt to predict the population in 1991-2020 based on 1960-1990 data.  We will then compare our predictions to the real 1991-2020 data we have from our data set as a comparison.  I know maths x humanities is not the best combo, so don't worry, this will be really broken down and easy to follow.  We will use use two different methods of extrapolation:\n",
    "\n",
    "* A standard population growth equation \n",
    "* Using NumPy to identify the pattern in number increase and continue it \n",
    "\n",
    " First, load the `India_population.csv` data set and call it `pop_df`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b4487154-6f4d-4f3c-99c5-13c1b42d6d84",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c7d20922-bce3-4220-86e4-edbd7796440d",
   "metadata": {},
   "source": [
    "Notice that the datapoints in the dataset at reversed - as in the first column is 2020 and the last column in 1960 opposed to being in ascending order.  Transform `pop_df` into an array called `pop_all` and reverse it so that the years are in increasing order using `np.flip(arr)`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e41fc22-2468-4d7b-891a-defa6da41c96",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "7234671f-0deb-4997-9e0e-b9d4461b1639",
   "metadata": {},
   "source": [
    "### Method 1: Population growth equation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a7be10f2-5e44-4f4d-85c4-e3801fb6ba00",
   "metadata": {},
   "source": [
    "The equation for population growth is the following: \n",
    "\n",
    "$$\n",
    "P = P_0e^{λ(t_1-t_0)}\n",
    "$$\n",
    "\n",
    "The main thing to understand is that P is the population number, T refers to the year, and that λ is the growth rate - which dicates how fast the population is growing.  It is an unknown that we need to figure out from our data set.  The equation above can be rearanged into the following which will allow us to calculate λ.\n",
    "$$\n",
    "λ = \\frac{log(\\frac{P_1}{P_0})}{t_1-t_0}\n",
    "$$\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4b3bd66f-0ac5-4ae3-b972-2f6d8c8c1eb2",
   "metadata": {},
   "source": [
    "Create a variable called `rate` which reflects the equation above.  As we are going to be extrapolating based on values from 1960 to 1990, these will be the values we refer to for `P` and `T`.  Accordingly, let the values that refer to subscript 0 be 1960 and those with subscript 1 refer to 1990.  Use `np.log` to conduct the log in the numerator.\n",
    "\n",
    "Hint: the index for the population in 1960 will be 0, and that for 1990 will be 31"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c6c7ce7b-cfbd-4694-b877-5048daf45996",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "13a8ba1a-7d26-4072-a7d7-86a4c2217578",
   "metadata": {},
   "source": [
    "Having calculated the growth rate, we can now put together the complete equation for population growth, which we will call `pop_equation`, do extrapolate for the years 1991-2020.  Therefore, $P_0$ will be 1990, as this is the last point of the data set.\n",
    "\n",
    "Use `np.exp` for the exponential aspect of the equation, and opposed to finding the difference between the $t_1 $ and $ t_0$ every time, multiply the `rate` variable by `n`.  \n",
    "\n",
    "Insert this equation into the allocated place in the for loop below, which will allow the `n` variable to cycle through numbers 1-30, allowing to extrapolate 30 years of population values - 1991 to 2020!  The number range 1-30 reflects the difference [ex] 1991-1990 = 1 ; 1992-1990 = 2 ... 2020-1990 = 30.  These values will be appended to the array `pop_predict_1` - containing our first method of extrapolation.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "3c0d227b-e6cb-4a1c-8ea3-ebb1d1257d97",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'pop_equation' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-2-d8679852771c>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      2\u001b[0m \u001b[0;32mfor\u001b[0m \u001b[0mn\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mrange\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;36m30\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      3\u001b[0m     \u001b[0;31m#insert equation\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 4\u001b[0;31m     \u001b[0mpop_predict_1\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mappend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mpop_equation\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      5\u001b[0m \u001b[0mpop_predict_1\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'pop_equation' is not defined"
     ]
    }
   ],
   "source": [
    "pop_predict_1 = []\n",
    "for n in range(30):\n",
    "    #insert equation\n",
    "    pop_predict_1.append(pop_equation)\n",
    "pop_predict_1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fda0506c-88a3-4e1e-bb24-967b0ebc8e98",
   "metadata": {},
   "source": [
    "### Method 2: NumPy extrapolation"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b70825b-5ffa-43fc-8e28-596f3c43309a",
   "metadata": {},
   "source": [
    "For the second approach, we will refer to the following method to extrapolate data by letting python identify the pattern in the values and continuing it: https://stackoverflow.com/questions/19406049/extrapolating-data-with-numpy-python/19432618.\n",
    "\n",
    "Essentially, using  the data from 1960 to 1990, python will extrapolate the years 1991-2020 based on the tendency of increase between the numbers. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa601c79-bbff-4372-8eb0-bab7299f75cc",
   "metadata": {},
   "source": [
    "First, create an array called `pop_1990` that has data from 1960 to 1990.  To do this, you will have to use `np.delete(arr,obj)`.  \n",
    "\n",
    "`obj` indicates the indexes of the part of the array we want to delete.  In our case we will use the function `slice(start,end,intervals)` in order to specify that we want the data from 1990 onwards (not including 1990) to be deleted.\n",
    "\n",
    "Hint: the index for the 1990 will be (0,31)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3526b5a-af6e-47e2-bb36-cd3407812503",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "132de676-e953-4831-9d11-cba01ee0fc66",
   "metadata": {},
   "source": [
    "Now create an array called `years` containing the numbers 1960-1990.  Use the function `np.arange(start, stop, interval)`to do this.  Double check that the array runs to 1990 inclusive!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "772b8fb7-5491-49ef-a337-6073462c96e8",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "0713004b-0869-4eb6-bae8-cb22000d131f",
   "metadata": {},
   "source": [
    "We will now create a polynomial fit based on our data from 1960 to 1990 to understand the growth trend in the population.  Create the variable `fit` using the function `np.polyfit(x, y, degree)`.  Population data usually follows an exponential growth model, so use degree 2, and for the x and y variables input the relevant arrays created above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f2918a86-f678-4e91-9454-b5e2e0b0ab84",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8c6c7c4e-b3c5-43a2-ba92-e30481682d71",
   "metadata": {},
   "source": [
    "Next, we will prepare to apply the function to find population predictions for the years 1991 to 2020.  Run the code below to create the variable `line`.  `np.poly1d` allows us to turn the `fit` into a function that returns the value of the polynomial at any given x value - thereby allowing for extrapolation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cac33090-075a-4b68-b3af-f59c62891e1d",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "c3c1dc51-d00f-4f88-beff-5c2d11c4214e",
   "metadata": {},
   "source": [
    "Now, create an array titled `new_years` that contains the values 1991 to 2020.  Use the same `np.arange` function detailed previously.  Once again, make sure it includes the year 2020!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7527b843-7877-4427-8e16-0c8ae7527222",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "b974dc6e-f409-49c9-969b-b6a6f827d072",
   "metadata": {},
   "source": [
    "Finally, we can apply our polynomial to find our extrapolated population values.  Name this array `pop_predict_2`, applying the `line` function to the `new_years` variable."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f6a5eae3-c94b-42c1-81b4-1e31ee1752ff",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "d856709d-839f-49b7-8aef-8117d23a795a",
   "metadata": {},
   "source": [
    "### Graphing the Results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7b0d14b2-5606-4158-a0b1-8e36538204a8",
   "metadata": {},
   "source": [
    "Now that we have our 2 population extrapolation methods completed, we shall plot them along side the actual 1991-2020 data and observe how good a extrapolation it is.  Create an array with this 1991-2020 data called `pop_2020` using the `slice` method described for the `pop_1990` array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "66da8362-3f9a-4e0e-a031-e17c8c7aeded",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "df7cd2f5-9662-4d66-8d16-5c76f99bcd72",
   "metadata": {},
   "source": [
    "We will now draw on our knowlege from the previous data visualisation session using `matplotlib` to scatter the three arrays with the following labels:\n",
    "* `pop_predict_1`  = \"method 1\"\n",
    "* `pop_predict_2` = \"method 2\"\n",
    "* `pop_2020`  = \"reality\"\n",
    "\n",
    "Don't forget to include a legend.  To display the graph, `plt.show(block=False)` is required - the `block` element helping circumnavigate python's desire in this case to overprocess the command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "08c7666e-dcf0-43fc-b2bc-22835addb568",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "09930d70-f5bb-44a2-a6d9-927ee3af1c46",
   "metadata": {},
   "source": [
    "Compare the results from the two population extrapolations to reality.  What do you think about them? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b9581d6f-fa3f-4cb0-a18b-49eb50b93c7a",
   "metadata": {},
   "source": [
    "Hope you enjoyed this notebook, and it offered a glimpse into what NumPy can do to hopefully make your life a bit easier when it comes to maths :)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python for Data Science",
   "language": "python",
   "name": "pyds"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}