session-one/session-one-project-maths-stats-questions.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "friendly-trout",
   "metadata": {},
   "source": [
    "# Python for Data Science Practice Session 1 : Mathematics and Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "impressive-township",
   "metadata": {},
   "source": [
    "## A walk through history"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "liable-voice",
   "metadata": {},
   "source": [
    "In this session, we are going to analyse a dataset containing Mathematicians that have helped shape what Maths - as well as many other fields - have become today. This is a great chance for you to learn more about previous Mathematicians starting from where they're from, the fields they were interested in, the occupations they had and many other stuff that you can explore.\n",
    "\n",
    "We start by importing the libraries that we are going to need for this project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "above-matthew",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Import pandas  \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "positive-geometry",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This is a library that is used in one of the pre-written codes\n",
    " "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "confirmed-bedroom",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Upload the file named mathematicians.csv into a dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "touched-illness",
   "metadata": {},
   "outputs": [],
   "source": [
    "# View the dataframe\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "formal-cliff",
   "metadata": {},
   "source": [
    "You might be wondering what NaN stands for. Well, NaN is an abbreviation for \"Not a Number\", and it is a member of the numeric data type but it is undefined or unrepresentable on the computer. Take 0/0 as an example, which is undefined. NaNs are treated as missing values by Python. \n",
    "\n",
    "This dataset is a great example on how real life datasets might look like, where datasets are not perfect and you will have to find a way to deal with those missing values.\n",
    "\n",
    "Anyways, let's proceed for now with getting a taste of what the dataset looks like and we'll get back to dealing with missing values in a bit."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "creative-people",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the dimensions of the dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "junior-strategy",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the data of 10 random mathematicians\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "secure-bikini",
   "metadata": {},
   "source": [
    "Not all the column headings are visible in the previous output due to the large amount of columns. \n",
    "\n",
    "Let us output the column headings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "fresh-kidney",
   "metadata": {},
   "outputs": [],
   "source": [
    "# View the column headings\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "threatened-geometry",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Drop the following columns: 'Erdős number' , 'instance of', 'approx. date of birth' and 'approx. date of death'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "severe-variation",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the number of columns\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "featured-minneapolis",
   "metadata": {},
   "source": [
    "- - - - "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spatial-bristol",
   "metadata": {},
   "source": [
    "## Let's talk about missing values"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "spiritual-efficiency",
   "metadata": {},
   "source": [
    "So we have a total of 25 columns. Let's check how many missing values each column has individually."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "stupid-alcohol",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the total number of missing values per column\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bridal-request",
   "metadata": {},
   "source": [
    " - - - - - -\n",
    "As you can see, most of the columns have a large amount of missing values which is an issue. After all, the analysis you make is as good as the data that you have. What this means in our case is that due to the large amount of missing values per column, the analysis that we do might not be of the best quality.\n",
    "\n",
    "One of the problems that we could face is having a pattern in the missing values. An example would be that older Mathematicians might have less information associated with them compared to the most recent ones (I will leave this for you to explore on your own to check and see if this is the case here if you are interested). As a result, this might introduce bias in your analysis, where the outputs you get from your analysis are not the best representatives for the older mathematicians because most of the data used in your analysis will be associated with recent Mathematicians, hence biased towards recent Mathematicians.\n",
    "\n",
    "What we would like to have is a missing value free dataset with a broad variety of Mathematicians. \n",
    "- - - - - - - - - - - - - - -\n",
    "In the next task, I would like for you to pick a random mathematician and fill in the missing values by doing a quick research. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "abroad-hopkins",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The following steps that you should follow are: \n",
    "\n",
    "# 1) Pick a random mathematician \n",
    "# 2) Do a quick research on them\n",
    "# 3) Fill in their missing data manually in the dataframe\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "homeless-factor",
   "metadata": {},
   "source": [
    "In a real world case scenario, we will not fill in each missing value manually as it is extremely in-efficient.\n",
    "\n",
    "A solution to this is called Web Scraping, which is the technique used to automatically scrape data from the internet using Python libraries, with the most popular one being called Beautiful Soup. It is extremely powerful in the sense that it is able to collect large sums of data by the click of a button, saving the massive amount of time that is needed to collect the data manually.\n",
    "\n",
    "Even though Web Scraping is beyond the scope of this course, I would recommend reading more about it on your own. You can find more about Web Scraping and Beautiful Soup here: https://realpython.com/beautiful-soup-web-scraper-python/ \n",
    "\n",
    "Keep in mind that there are different types of missing values (eg: numeric missing values) and different ways of dealing with them. We will return back to this topic in the upcoming sessions.\n",
    "- - - - -"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "challenging-floating",
   "metadata": {},
   "source": [
    "----\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "least-faculty",
   "metadata": {},
   "source": [
    "## List of occupations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "widespread-contrast",
   "metadata": {},
   "outputs": [],
   "source": [
    "# View the occupation column\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ruled-clause",
   "metadata": {},
   "source": [
    "Lots of the Mathematicians worked multiple occupations as you can see. Let us analyse the different occupations in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "conservative-programming",
   "metadata": {},
   "outputs": [],
   "source": [
    "# View the unique entries in the occupation column, then save it in a list called unique_occ\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "weighted-complement",
   "metadata": {},
   "outputs": [],
   "source": [
    "# View unique_occ\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "complimentary-absolute",
   "metadata": {},
   "source": [
    "the occupation \"mathematician\" is repeated in both ['mathematician'] and ['publisher', 'mathematician'] in unique_occ. This is because the entries are different, and so they are treated as unique results. This deems the <b>unique()</b> function useless in outputting a list of the unique occupations. \n",
    "\n",
    "In the following tasks, we will work towards outputting a list of the unique occupations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "continental-venezuela",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This code created a list called flat_list which takes each entry in df.occupation.unique(), splits each occupation  \n",
    "# from the other occupations and puts it in its own entry\n",
    "item_list = []\n",
    "for item in unique_occ:\n",
    "    item = item.split(\",\")\n",
    "    item_list.append(item)\n",
    "flat_list = itertools.chain(*item_list)\n",
    "flat_list = list(flat_list)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "corrected-folks",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print the first 30 entries of flat_list\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "industrial-message",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Length of unique_occ is _____ while length of flat_list is _____\n"
     ]
    }
   ],
   "source": [
    "# Compare the length unique_occ with flat_list \n",
    "print(\"Length of unique_occ is\", '_____' , \"while length of flat_list is\", '_____' )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "compatible-residence",
   "metadata": {},
   "source": [
    "As you can see in the first 30 entries of flat_list, the format still has a few issues. For example, we have 'mathematician', ['mathematician'] and 'mathematician'] which all represent the same thing but are written in different ways. They are going to be treated by Python as distinct entries when trying to filter out the distinct values from the list.\n",
    "\n",
    "To solve this, we will loop over the entries in flat_list and use a function called <b>replace</b> that will help remove unwanted characters in order for all of the entries to have a unified format. Read more about how to use it here: https://www.w3schools.com/python/ref_string_replace.asp \n",
    "\n",
    "I would recommend solving this issue by removing each unwanted character individually, checking out how the output looks like afterwards, identifying another character that needs to be removed and then repeating the process until we end up with a unified format for all the entries. This helps in making the process much simpler. \n",
    "\n",
    "One thing to keep in mind is that Python is really sensitive to cases and spacings. So, 'Mathematician' isn't the same as 'mathematician', and   '[' is not the same as  ' [' .\n",
    "\n",
    "\n",
    "<b> HINT:</b> Writing ' ' in the second parameter of the function <b>replace</b> will remove the character instead of replacing it with another character."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "opponent-double",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Loop through flat_list and use replace to remove unwanted characters, then save every new entry\n",
    "# in the new list called flat_list2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "formal-mathematics",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the first 30 entries of flat_list2\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "encouraging-slovenia",
   "metadata": {},
   "source": [
    "Problem solved! Now use the function called <b>set</b> to retrieve the unique occupations from flat_list2. Read more about how to use it from here: https://www.geeksforgeeks.org/python-set-method/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "aging-midwest",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Retrieve the unique occupations from flat_list2\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "removable-bangkok",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the length of the list of unique occupations\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "strategic-locator",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the list of unique occupations\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "published-memory",
   "metadata": {},
   "source": [
    "Apart from having some unusual entries in the list of unique occupations (which we will ignore for the sake of time), it has a broad variety of occupations which shows how flexible Mathematicians are.\n",
    "- - - - -"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "laughing-representative",
   "metadata": {},
   "source": [
    "## Analysing gender distribution"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "celtic-might",
   "metadata": {},
   "source": [
    "Now let's take a look into the difference between the number of male and female mathematicians:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "atmospheric-salem",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Print out the unique entries in the \"sex or gender\" column\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "upset-latvia",
   "metadata": {},
   "source": [
    "It seems to produce some weird entries like ['male', 'Swedish Wikipedia', 'Virtual International Authority File', 'Italian Wikipedia'] containing info that should not be in the gender column. This is definitely not the format that we want the gender column to have.\n",
    "\n",
    "In the following tasks, we are going to modify the dataset to match what we need."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "proved-phrase",
   "metadata": {},
   "source": [
    "Looking closely at the unique entries in \"sex or gender\", we either have <b>male</b>, <b>female</b>, <b>intersex</b> or <b>NaN</b>. Some of the entries have those words with some other unwanted entries, but we know that each entry contains one of those 4 words.  This makes it easy to change every entry in the column so that it is either \"male\", \"female\", \"intersex\" or \"not specified\". \n",
    " \n",
    "We use a function called <b>str.contains</b> which checks every string entry in a list on whether it contains a specific word (or generally a combination of characters of interest) and outputs a list with each entry being either a True or False, where True represents the existence of the word in the entry, and False the absence.\n",
    "\n",
    "Read more about it here: https://pandas.pydata.org/docs/reference/api/pandas.Series.str.contains.html and use it in order to create four new boolean columns in the dataframe: \"male\", \"female\", \"intersex\" and \"not specified\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "colonial-device",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Create the three new columns specified above in the dataframe\n",
    "df['sex or gender'] = df['sex or gender'].fillna('') #This replaces the NaN values with empty values in order to be able to apply str.contains"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "retained-replication",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count the number of males\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "silver-steel",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count the number of females\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "interstate-break",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count the number of intersex\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "transsexual-classroom",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count the number of not specified\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "renewable-teens",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The proportions are:\n",
      "Male = _____\n",
      "Female = _____\n"
     ]
    }
   ],
   "source": [
    "# Calculate the proportions of male and female mathematicians in the dataset\n",
    "print('The proportions are:')\n",
    "print('Male =', '_____')\n",
    "print('Female =', '_____' )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "federal-tiffany",
   "metadata": {},
   "source": [
    "Now that is a big difference between the number of male and female Mathematicians. This is a clear indication that the data is biased towards male Mathematicians. \n",
    "\n",
    "Do you think that this might be a problem when using the data for predictions and drawing conclusions? Have a think about it.\n",
    "\n",
    "We now create a new column called gender that has one of the following values: male, female, intersex or not specified."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "alone-awareness",
   "metadata": {},
   "outputs": [],
   "source": [
    "# This code creates the gender column with the format that we want\n",
    "gender = []\n",
    "x = range(df.shape[0])\n",
    "for i in x:\n",
    "    if df['sex or gender'].str.contains(\"'male'\")[i] == True:\n",
    "        gender.append('male')\n",
    "    elif df['sex or gender'].str.contains(\"'intersex'\")[i] == True: \n",
    "        gender.append('intersex')\n",
    "    elif df['sex or gender'].str.contains(\"'female'\")[i] == True:\n",
    "        gender.append('female')\n",
    "    else: \n",
    "        gender.append('not specified')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "advance-thing",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace the 'sex or gender' column with 'gender'\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "secure-moral",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Preview the 'sex or gender' column\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "offensive-adjustment",
   "metadata": {},
   "source": [
    "All solved! Now that we formatted the 'sex or gender' column, we can get rid of the boolean columns 'male', 'female', 'intersex' and 'not-specified'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "numerical-buddy",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove the following columns: 'male', 'female', 'intersex' and 'not-specified'\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "disciplinary-stake",
   "metadata": {},
   "source": [
    "## Analysing the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "african-merchandise",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Are there any rows that do not have any missing values? (Output True or False)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "super-porcelain",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the rows that have at least 10 missing values\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "acute-afternoon",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the rows of the mathematicians that have the year of birth filled in, name the list of rows x\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cellular-muscle",
   "metadata": {},
   "source": [
    "The following tasks are going to require the knowledge of a function called <b>str.contains</b> which is mentioned and explained in the section named <b>Analysing gender distribution</b> in this notebook. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "covered-geometry",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Check the datatype of the 'year of birth' column\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "christian-discipline",
   "metadata": {},
   "source": [
    "We would like to have the data type of the entries of the 'year of birth' column to be numeric in order to be able to perform mathematical operations that will be needed in the following tasks.\n",
    "\n",
    "In the next task, convert the data type of the column 'year of birth' to numeric. Do you run into a problem? Inspect the column entries to find out what is causing it and fix it. \n",
    "\n",
    "Note: You can either edit the entries that are causing the problems, or remove them entirely."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "bacterial-heater",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Convert the datatype of the column 'year of birth' to numeric\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "dimensional-sending",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the row of the oldest mathematician in x\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "buried-thousand",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the row of the most recently born mathematician in x\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "female-disability",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the range of the year of birth column\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "contained-george",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Output the rows of the mathematicians that have studied at the University of Cambridge\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "surface-directive",
   "metadata": {},
   "outputs": [],
   "source": [
    "# How many mathematicians were interested in any of the fields of Analysis?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "pursuant-architecture",
   "metadata": {},
   "source": [
    "## Extra task"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "through-cartoon",
   "metadata": {},
   "source": [
    "Here is a link for you to check out that shows the investigations that the owner of this dataset performed: https://www.kaggle.com/joephilleo/investigating-the-mathematicians-of-wikipedia"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}