21_python_data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data science in Python\n",
    "\n",
    "- Course GitHub repo: https://github.com/pycam/python-data-science\n",
    "- Python website: https://www.python.org/ \n",
    "\n",
    "## Session 2.1: Working with Pandas\n",
    "\n",
    "- [Reading CSV Data Using Pandas](#Reading-CSV-Data-Using-Pandas)\n",
    "- [Exploring our data](#Exploring-our-data)\n",
    "- [DataFrame manipulation](#DataFrame-manipulation)\n",
    "- [Exercise 2.1.1](#Exercise-2.1.1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Mind map\n",
    "\n",
    "<img src=\"img/mind_maps/mind_maps.004.jpeg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Reading CSV Data Using Pandas\n",
    "\n",
    "### Importing the `pandas` package\n",
    "\n",
    "[`pandas`](http://pandas.pydata.org/) is a widely-used external Python package for data analysis, particularly useful for tabular data (see [documentation](http://pandas.pydata.org/pandas-docs/stable/)).\n",
    "\n",
    "The `pandas` DataFrame object borrows many features from R's `data.frame`. Both are 2-dimensional tables whose columns can contain different data types (e.g. boolean, integer, float, categorical/factor). Both the rows and columns are indexed, and can be referred to by number or name.\n",
    "\n",
    "Because `pandas` is an external third-party package, it is not included in Python by default. You can install it using `pip install pandas`, or if using conda, `conda install pandas`. \n",
    "\n",
    "Once installed, we load the package with the `import` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Reading CSV data\n",
    "\n",
    "To read a Comma Separated Values (CSV) data file with `pandas`, we use the `.read_csv()` command.\n",
    "\n",
    "We are going to load a slightly different Gapminder dataset for Oceania, where each column represents the GDP per capita for different years and each row represents a country in Oceania. In this case, our file is in a sub-directory called `data/`, and the name of the file is `gapminder_gdp_oceania.csv`. \n",
    "\n",
    "We store the resulting DataFrame object and give it the variable name `data`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "data = pandas.read_csv('data/gapminder_gdp_oceania.csv')\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you forget to include `data/`, or if you include it but your copy of the file is saved somewhere else, you will get a runtime error that ends with a line like this:\n",
    "```\n",
    "FileNotFoundError: File b'gapminder_gdp_oceania.csv' does not exist\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that `pandas` uses a backslash `\\` to show wrapped lines when the output is too wide to fit on the screen. Looking at the index column on the far left, you can see that our DataFrame has two rows (automatically 0 and 1, since Python uses 0-based indexing).\n",
    "\n",
    "Generally, columns in a DataFrame are the observed variables, and the rows are the observations. \n",
    "\n",
    "Instead of treating the `'country'` column like any other observed variable, we can tell `pandas` to use this column as the row index. We can then refer to the rows by country name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pandas.read_csv('data/gapminder_gdp_oceania.csv', index_col='country')\n",
    "print(data)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the `.read_csv()` command is not limited to reading CSV files. For example, you can read Tab Separated Value (TSV) files by adding the argument `sep='\\t'`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exploring our data\n",
    "\n",
    "A `pandas` DataFrame is a 2-dimensional object that can store columns with different data types (including string, boolean, integer, float, categorical/factor, etc). It is similar to a spreadsheet or an SQL table or the `data.frame` structure in R. \n",
    "\n",
    "A DataFrame always has an index (0-based). An index refers to the position of an element in the data structure. Using the `.info()` method, we can view basic information about our DataFrame object:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As expected, our object is a `DataFrame` (or, to use the full name that Python uses to refer to it internally, a `pandas.core.frame.DataFrame`).\n",
    "\n",
    "It has 2 rows (named Australia and New Zealand) and 12 columns. The columns consist of 64-bit floating point values. It uses about 200 bytes of memory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As mentioned, a DataFrame is a Python object, which means it can have **object attributes** and **object methods**.\n",
    "\n",
    "**Object attributes** contain information about the object. You can access [DataFrame attributes](https://pandas.pydata.org/pandas-docs/stable/reference/series.html#attributes) to learn more about the contents of your DataFrame. To do this, use the object variable name followed by the attribute name, separated by a `.`. Do not use any `()` to access attributes. \n",
    "\n",
    "For example, let's access our column data types, which are stored in the `.dtypes` attribute:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can access the dimensions of your DataFrame using the `.shape` attribute. The first value is the number of rows, and the second the number of columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The row and column names can be accessed using the attributes `.index.values` and `.columns.values`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.index.values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.columns.values"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Object methods** are functions that are associated with an object. Because they are functions, you do use `()` to call them, and can add arguments inside the parentheses to control the behaviour of the method. For example, the `.info()` command we executed previously was a method. \n",
    "\n",
    "Let's now download the European Gapminder data and explore a few of these (see the [DataFrame API](https://pandas.pydata.org/pandas-docs/stable/reference/index.html) for more)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data = pandas.read_csv('data/gapminder_gdp_europe.csv', index_col='country')\n",
    "print(eu_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.head()` method prints the first few rows of the table, while the `.tail()` method prints the last few lines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data.head(3)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data.tail(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `.describe` method computes some summary statistics for the columns (including the count, mean, median, and std):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We often want to calculate summary statistics grouped by subsets or attributes within fields of our data. For example, we might want to calculate the average GDP per capita for 1962.\n",
    "\n",
    "There are two ways to access columns in a DataFrame. The first is using a `.` followed by the name of the column, while the second is using square brackets:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data.gdpPercap_1962.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data['gdpPercap_1962'].head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also compute metrics on specific columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data['gdpPercap_1962'].mean()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "eu_data['gdpPercap_1962'].std()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: these statistics can only be computed on numeric columns. They can be particularly useful when your DataFrame contains thousands of entries."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## DataFrame manipulation\n",
    "\n",
    "### Pandas Cheat Sheet\n",
    "\n",
    "The [Pandas Cheat Sheet](https://github.com/pandas-dev/pandas/blob/master/doc/cheatsheet/Pandas_Cheat_Sheet.pdf) can be very helpful for recalling basic `pandas` operations. \n",
    "\n",
    "Let's start by creating a small DataFrame object and storing it using the variable name `df`. One way to do this is from a Python dictionary:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas\n",
    "df = pandas.DataFrame({'gene': ['BRCA2', 'TNFAIP3', 'TCF7'], \n",
    "                       'chrom': ['13', '6', '5'],\n",
    "                       'length': [84195, 16099, 37155]}\n",
    ")\n",
    "print(df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Selecting columns and rows\n",
    "To select rows and columns in `pandas`, we use square brackets `[]`. There are several ways to select or \"slice\" a DataFrame objects. It's important to distinguish between **positional indexing**, which uses index numbers, and **label-based indexing** which uses column or row names.\n",
    "\n",
    "Let's start by selecting one column, using its name:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['length']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we select the first two rows using their numeric index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[:2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can combine these to get the first 2 rows of the column `'length'`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df[:2]['length']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To do **positional indexing** for both rows and columns, use `.iloc[]`. The first argument is the numeric index of the rows, and the second the numeric index of the columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[:2]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[:2,2]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also select the first 2 rows, and all of the columns from index 1 until the end, as follows:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.iloc[:2,1:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For **label-based indexing**, use `.loc[]` with the column and row names:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df.loc[:1,'chrom':]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note: because the rows have numeric indices in this DataFrame, we might assume that selecting rows with `.iloc[]` and `.loc[]` would be the same. As we see below, this is not the case."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Sorting columns\n",
    "To sort the entire DataFrame according to one of the columns, we can use the `.sort_values()` method. Note that this method returns a DataFrame object, so we need to store that object using a new variable name such as `sorted_df`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_df = df.sort_values('length')\n",
    "print(sorted_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Going back to positional versus label-based indexing, we see that `.iloc[0]` will return the first row in the DataFrame, while `.loc[0]` will return the row with index 0:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_df.iloc[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_df.loc[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also sort the DataFrame in descending order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_df = df.sort_values('length', ascending=False)\n",
    "print(sorted_df)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you would like to reset the row indices after sorting, use `.reset_index()` and store the result in a new variable. Adding the argument `drop=True` will prevent `pandas` from adding the old index as a column:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sorted_df_reindexed = sorted_df.reset_index(drop=True)\n",
    "print(sorted_df_reindexed)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Exercise 2.1.1\n",
    "\n",
    "- Read the data in `gapminder_gdp_americas.csv` (which should be in the same directory as `gapminder_gdp_oceania.csv`) into a variable called `americas_data` and display its summary statistics.\n",
    "- As well as the `.read_csv()` function for reading data from a file, Pandas provides a `.to_csv()` function to write DataFrames to files. Applying what you’ve learned about reading from files, write one of your DataFrame to a file called `processed.csv`. You can use help to get information on how to use `.to_csv()`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Manipulating data with Pandas (live coding session)\n",
    "\n",
    "Let's now open a new Jupyter notebook, and explore another dataset `GRCm38.gff3` from the `data/` folder.\n",
    "\n",
    "[GFF is a standard file format](http://gmod.org/wiki/GFF3) for storing genomic features in a text file. GFF stands for Generic Feature Format. GFF files are plain text, 9 column, tab-delimited files. \n",
    "\n",
    "The 9 columns of the annotation section are as follows:\n",
    "\n",
    "- Column 1: \"seqid\" - The ID of the landmark used to establish the coordinate system for the current feature, a.k.a. chromosome name.\n",
    "- Column 2: \"source\" - The algorithm or operating procedure that generated the feature. \n",
    "- Column 3: \"type\" - The type of feature.\n",
    "- Columns 4 & 5: \"start\" and \"end\" - The start and end of the feature.\n",
    "- Column 6: \"score\" - The score of the feature, a floating point number.\n",
    "- Column 7: \"strand\" - The strand of the feature.\n",
    "- Column 8: \"phase\" - For features of type \"CDS\", the phase indicates where the feature begins with reference to the reading frame. \n",
    "- Column 9: \"attributes\" - A list of feature attributes in the format tag=value. \n",
    "\n",
    "We have modified these files and added a 10th column \"gbid\" which is the GenBank ID of each feature, and taken a random subset of these features for each species."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Next session\n",
    "\n",
    "Go to our next notebook: [Session 2.2: Data visualisation with Matplotlib](22_python_data.ipynb)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}