session-two/session-two-project-social-sciences-questions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a479d151-9a7c-472d-b186-6d7618dcae64",
   "metadata": {},
   "source": [
    "# Python for Data Science Project Session 2: Social Sciences"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca0d8f81-5598-4066-bb15-b0657f88f2bb",
   "metadata": {},
   "source": [
    "The Sustainable Development Goals (SDGs) are a UN initiative consisting of 17 goals.  Their aim is to continue striving towards development in a more environmentally conscious manner.  \n",
    "\n",
    "In this notebook, we will investigate data regarding the aim of increasing access to electricity.  We will create three main graphs – first looking at data on a global scale, then focusing on Africa, and finally looking at if progress has been made in the 10 countries with the lowest access.  \n",
    "\n",
    "We will be using World Bank data spanning from 2000-2019 relating to Goal 7 (Affordable and Clean Energy)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c6952cd3-7448-43d8-89f4-b4ee99377b5f",
   "metadata": {},
   "source": [
    "First, import the necessary packages:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c946b6fc-bf58-4790-9665-8bdc7ca7e030",
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "import pandas as pd\n",
    "import pycountry \n",
    "import seaborn as sns"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "113021ef-0f10-4b4b-a3fb-7a89757643c7",
   "metadata": {},
   "source": [
    "Then, run the code below to import the dataset `GoalData7.csv` (find the file on the session 2 page) to access the data and extract the variables that are of relevance.  This will create a data frame called `sdg` that contains our 6 variables of interest:\n",
    "* `GeoAreaCode`\n",
    "* `GeoAreaName`\n",
    "* `TimePeriod`\n",
    "* `Value`\n",
    "* `Location`\n",
    "* `Nature`\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c1d552ef-bfb8-46bf-9459-11157dfef6c9",
   "metadata": {},
   "outputs": [],
   "source": [
    "raw_sdg = pd.read_csv(\"Goal7data.csv\")\n",
    "sdg = raw_sdg[[\"GeoAreaCode\", \"GeoAreaName\", \"TimePeriod\", \"Value\", \"Location\", \"Nature\"]].copy()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7649bf5d-7bb7-457e-8ed1-4490405091f8",
   "metadata": {},
   "source": [
    "Have a look at the data frame `sdg`, calling up random samples. Notice how the data is split up into `ALLAREA`, `RURAL` and `URBAN` data points, along with values for countries and wider regions.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54eb7905-6804-4137-a948-789973eebf74",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "8f48ab6a-2721-4c53-8aee-b50d4fadf961",
   "metadata": {
    "tags": []
   },
   "source": [
    "## A Global Look"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7025ff28-8243-48e9-b743-1c808c14f6da",
   "metadata": {},
   "source": [
    "In this first section, we will look at the data on a global scale.  Run the code below to create the data frame `df_world` that consists of the `ALLAREA` data from the geographical area called `World`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "12a57105-09a7-4cc3-93b9-4cfd8bcb5e5e",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_world = sdg[(sdg[\"GeoAreaName\"] == \"World\") & (sdg[\"Location\"] == \"ALLAREA\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7c273f7e-f00c-413b-8ff6-3a4c794b22ee",
   "metadata": {},
   "source": [
    "Then, create a line plot from this data frame, with the x-axis being the `TimePeriod`, and the y-axis being the access to electricity  `Value`.  Specify a colour, and add the parameter `label = \"All Country Data\"` to the plot specifications in order to help clarify the legend.\n",
    "\n",
    "Format the figure, reffering to the documentation (https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.html) for help.  Ensure that the:\n",
    "* x axis ticks are correctly spaced [hint: use splicing], and rotated so they can be read \n",
    "* both axes are labelled\n",
    "* there is a grid (and it is behind the line plot)\n",
    "* the plot has a title\n",
    "* the plot has a legend\n",
    "* colour documentation = https://matplotlib.org/stable/gallery/color/named_colors.html"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3f838e44-f260-458a-8a74-f3865a73cd4a",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f8704064-aa12-4858-86fc-5e7669735e51",
   "metadata": {},
   "source": [
    "Now, we will add a scatter of all countries data.  First, run the code below to manipulate `sdg` such that a data frame `df_c` is created.  It consists of the `ALLAREA` data for just countries - thereby removing values for regions and urban/rural specifics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3cd0c4cb-868c-4e60-95be-392b7bd9577e",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Create list of countries\n",
    "countries_list = []\n",
    "for country in pycountry.countries:\n",
    "    countries_list.append(country.name)\n",
    "    \n",
    "#Create data frame without regions\n",
    "df_c = sdg[(sdg[\"Location\"] == \"ALLAREA\")]\n",
    "df_c = df_c[df_c.GeoAreaName.isin(countries_list)]\n",
    "df_c"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9fa64d3f-922c-4778-a66e-d93e841d1057",
   "metadata": {},
   "source": [
    "Now, add a scatter of all countries data using the data frame `df_c` to the line plot created above. Change the transparency for clarity using `alpha`, and the make the colour something neutral so that the world average stands out ontop.  Also include a `label` parameter for clarity in the legend."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d6f4b5b0-7fd4-4da0-a396-025713d3d26e",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "922716b5-bf65-40ff-aaf7-db4925dd4cac",
   "metadata": {},
   "source": [
    "The scatter plot has a lot of data, and so is not very clear to read.  However, creating a colour distinction between rural and urban data points would illustrate if there are any significant divisions between access to electricity."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8c165254-f1fc-4154-9731-d7ec6ba15cd2",
   "metadata": {},
   "source": [
    "Run and observe the code below.  It is an example which illustrates how to plot seperate series (in the example, the seperate series being different flower species) using a `for` loop."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e84868eb-35f6-463f-9c20-f0e831adfbfb",
   "metadata": {},
   "outputs": [],
   "source": [
    "iris = pd.read_csv(\"https://raw.githubusercontent.com/mwaskom/seaborn-data/master/iris.csv\")\n",
    "iris.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "946e4c89-f8f4-43a7-82db-fc9e924ecfac",
   "metadata": {},
   "outputs": [],
   "source": [
    "fig, ax = plt.subplots()\n",
    "\n",
    "for species in iris.species.unique():\n",
    "    iris_subset = iris.loc[iris.species == species]\n",
    "    ax.scatter(iris_subset.sepal_length, iris_subset.sepal_width,\n",
    "               label=species)\n",
    "ax.legend()\n",
    "ax.set_xlabel(\"Sepal Length\")\n",
    "ax.set_ylabel(\"Sepal Width\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dfc8c9c8-583c-4ed6-974e-e64c10b7862d",
   "metadata": {},
   "source": [
    "Now, we will use this `for` loop method to plot the rural and urban aspects of our data frame as separate series.  Run the cell below to create the data frame `df_c2` which will have all the countries rural and urban data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "71901ba9-1a88-4f3c-904b-e104a6211e68",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_c2 = sdg[(~(sdg[\"Location\"] == \"ALLAREA\")) & (sdg.GeoAreaName.isin(countries_list))]\n",
    "df_c2"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4ff11707-9542-4769-a9cb-b8dc4ebcb914",
   "metadata": {},
   "source": [
    "Then, replace the previous scatter plot by using the `for` loop to create a two tone scatter chart distinguishing between rural and urban areas.  Still keep the world average in the figure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58bcad3c-26a7-43bc-86b6-732a3ce40d58",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "de007939-9fd8-46e3-8ee7-36e143839822",
   "metadata": {},
   "source": [
    "What do you notice about the distrubution of rural vs urban data points ?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b626989c-3c60-4e8a-bc2f-a2cfbcac2734",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Deep Dive: A look into Africa"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17ebee21-8d38-41fb-ad5b-17267dce15b6",
   "metadata": {},
   "source": [
    "Research generally points to Africa being the continent that is the most energy deprived.  We will therefore focus the next step of our analysis on this continent.\n",
    "\n",
    "The data includes a breakdown of 6 african regions:\n",
    "* North Africa\n",
    "* South Africa\n",
    "* Middle Africa\n",
    "* East Africa\n",
    "* West Africa\n",
    "* Sub-Saharan Africa\n",
    "\n",
    "The aim is to creat two line plots side by side, one with the urban data of said regions, and the other with rural data.  Line graphs have less flexibility compared to scatter plots in terms of determining colours and series.  It is therefore easiest to split the rural and urban data.\n",
    "\n",
    "Observe and run the code for the creation of the urban african regions data frame.  Note that it is first filtered to include any `GeoAreaName` that contains the word `Africa`, along with selecting the `Locatio` for `URBAN`.  A look at the `value_counts()` shows that there are some undersireable regions that have been included, such as \"South Africa\".  These are then removed to create the final `df_africa_u` data frame."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "4eb45fb7-595d-4f9b-93cc-e84b175167c4",
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'sdg' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-1-c6046854827a>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[1;32m      1\u001b[0m \u001b[0;31m#africa regions urban\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 2\u001b[0;31m \u001b[0mdf_africa_rawu\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0msdg\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0msdg\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGeoAreaName\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontains\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Africa\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m&\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0msdg\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m\"Location\"\u001b[0m\u001b[0;34m]\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m'URBAN'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      3\u001b[0m \u001b[0mdf_africa_rawu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGeoAreaName\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalue_counts\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      4\u001b[0m \u001b[0mdf_africa_u\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mdf_africa_rawu\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mdf_africa_rawu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGeoAreaName\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;34m\"Northern Africa and Western Asia\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m&\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mdf_africa_rawu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGeoAreaName\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;34m\"South Africa\"\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;34m&\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mdf_africa_rawu\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mGeoAreaName\u001b[0m \u001b[0;34m!=\u001b[0m \u001b[0;34m\"Central African Republic\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mNameError\u001b[0m: name 'sdg' is not defined"
     ]
    }
   ],
   "source": [
    "#africa regions urban\n",
    "df_africa_rawu = sdg[sdg.GeoAreaName.str.contains(\"Africa\") & (sdg[\"Location\"] == 'URBAN')] \n",
    "df_africa_rawu.GeoAreaName.value_counts() \n",
    "df_africa_u = df_africa_rawu[(df_africa_rawu.GeoAreaName != \"Northern Africa and Western Asia\") & (df_africa_rawu.GeoAreaName != \"South Africa\") & (df_africa_rawu.GeoAreaName != \"Central African Republic\")]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c7d01d30-a407-4e0a-a217-5d85256ac699",
   "metadata": {},
   "source": [
    "Follow the above methodology to create a data frame for the rural regions of Africa:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6bb372f6-4e79-4471-abe3-3465cf8298f9",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "72c048b3-eaa8-42e2-9959-7a3162edeaef",
   "metadata": {},
   "source": [
    "Look at either the `df_africa_u` or `df_africa_r` data frame.  All the regions are listed together under the `GeoAreaName` column.  This makes it tricky to create a line plot."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e710fb7d-63c6-4c14-a7ee-0a66ef081a88",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "85413513-b7a6-4a56-bd73-bda1dda58640",
   "metadata": {},
   "source": [
    "To help plot each region as a seperate line, we will create something known as a 'pivot table'.  This will reorder the data so that each region will be its own column. \n",
    "\n",
    "Referring to the documentation here (https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.pivot.html), create two new dataframes: `pivot_u` and `pivot_r` with the urban and rural data for the regions.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c0452d4b-de7c-465e-9640-bfa393249707",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "5587c65f-c6e9-4a89-b2bf-4b833678465f",
   "metadata": {},
   "source": [
    "Having reorganised the data, now create two subplots next to each other, one with the African regions urban data, and the other with the rural.  The pivot table will automatically plot each series in a different colour, so that is the only input needed in `.plot()`.\n",
    "\n",
    "Each subplot has to be formatted seperately, so a trick to apply the same conditions to both is using `plt.setp((ax1,ax2), function = [])`.  It doesn't work in all cases (in this case doesn't seem to work for adding grid lines), but helps speed things up often. \n",
    "\n",
    "Look at the documentation (https://matplotlib.org/stable/api/_as_gen/matplotlib.pyplot.htmlto), and find the necessary functions to:\n",
    "* set x axis from 2000 to 2019\n",
    "* set y axis from 0 to 100\n",
    "* correct x axis ticks (splicing won't work here unfortunately - I suggest jumps of 2)\n",
    "* add grid lines\n",
    "* add a legend to the figure\n",
    "* set titles for individual plots\n",
    "* set an overal title for the figure"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dbf2cab2-8c13-4744-b031-7699b38ef207",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "03c983cb-32c5-4713-8a4b-3097cb5e8f6a",
   "metadata": {
    "tags": []
   },
   "source": [
    "## Progress or not?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1ebe97cf-752e-4eab-82fb-6da5619f7821",
   "metadata": {},
   "source": [
    "With the aim of the SDGs being to help countries develop, seeing if there is progress in acheiving said goals is an important element of analysis.\n",
    "\n",
    "We will therefore find the 10 countries that had the lowest access to electricity in 2000, and compare it to the data for them in 2019. This will be done using a more advanced plotting tool called \"Seaborn\" to create a grouped bar graph.\n",
    "\n",
    "Run the cells below to create a dataframe of all countries in 2000 (`df_2000`) and 2019 (`df_2019`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "443e479f-f411-4777-8506-5ff6c796b8bd",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_2000 = df_c[(df_c[\"Location\"] == \"ALLAREA\") & (df_c[\"TimePeriod\"] == 2000)]\n",
    "df_2019 = df_c[(df_c[\"Location\"] == \"ALLAREA\") & (df_c[\"TimePeriod\"] == 2019)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d3b1069-2cc5-4c9b-b6fe-a01540d685fe",
   "metadata": {},
   "source": [
    "Sort the values colomn to find the 10 countries with the lowest access to electricity in 2000.  Call this new data frame `df_2000_10`. \n",
    "\n",
    "Hint = use the `.nsmallest(#, colomn)` function to find the bottom 10 countries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b2722810-0b9f-422f-8812-8c7cf8e1ffba",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "f9da927c-7a7e-4155-be2f-13bc276331b6",
   "metadata": {},
   "source": [
    "We will now create a data frame of the same 10 countries, but with their 2019 data.\n",
    "\n",
    "First, create a list titled `c_10` of the name in the GeoAreaName colomn so we can use it to filter the 2019 data.\n",
    "\n",
    "Hint = use `.tolist()` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0e55268b-6f3c-4650-b15b-db0f34a07729",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "399e1e1b-bcbe-4e6c-bf88-2285e38d2431",
   "metadata": {},
   "source": [
    "Now, using this list as a filter, create a dataframe `df_2019_10` from the dataframe `df_2019` which has all countries data in 2019.\n",
    "\n",
    "Hint = use the `.isin()` function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "72fe98cd-e64c-49d6-b5fd-658086f52d6f",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "1e40cd1c-57de-4bc2-89b1-24819def4dfc",
   "metadata": {},
   "source": [
    "Seaborn requires that the data is in one single data frame, so run the code below to combine the two data frames (learn more about such dataframe manipulations in session 4).  This will create a final dataframe called `df_10`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8b4e4522-4563-436d-b7d6-96350ccae4c1",
   "metadata": {},
   "outputs": [],
   "source": [
    "frames = [df_2000_10, df_2019_10]\n",
    "\n",
    "df_10 = pd.concat(frames)\n",
    "df_10"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "79031001-ff9f-4f5e-af4d-e6b09e25fc06",
   "metadata": {},
   "source": [
    "Now, we will use Seaborn to create a grouped bar graph.  Look at the example at this following link = https://seaborn.pydata.org/examples/grouped_barplot.html,  and use it as a guide to plot the `df_10` dataframe below. \n",
    "\n",
    "Not all the elements from the example are necessary, and in addition, rotate the x-axis ticks 90 degrees."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1ecd5e7f-fd7d-49c6-b786-fbb71fa734a1",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "id": "a22dc005-8cc6-439b-9b89-a3ed934c0889",
   "metadata": {},
   "source": [
    "You can find alot more information on Seaborn plot customisation here = https://s3.amazonaws.com/assets.datacamp.com/production/course_15192/slides/chapter4.pdf\n",
    "\n",
    "Python has loads of powerful methods to visualise data, and hopefully this notebook has given you a taste for what is possible.  Hope you enjoyed!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python for Data Science",
   "language": "python",
   "name": "pyds"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}