data-wrangling-in-pandas.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data wrangling in pandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import all the tools..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "from glob import glob\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "import re\n",
    "import seaborn as sns\n",
    "import matplotlib.pyplot as plt\n",
    "from patsy import dmatrix\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Read in the data\n",
    "Let's play with some behavioral data from the Human Connectome Project. We'll look at data from the N-back task for the first 100 subjects. The E-Prime logs are contained in a `data` folder below this notebook. We'll loop over them and read each one into a pandas DataFrame; then we'll concatenate them all into one big DataFrame to rule them all."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Read all individual run E-Prime outputs and merge into one big DataFrame\n",
    "files = glob('data/WM*TAB.txt')\n",
    "dfs = []\n",
    "\n",
    "for f in files:\n",
    "\n",
    "    # Read in each tab-delimited file with pandas\n",
    "    _df = pd.read_csv(f, sep='\\t')\n",
    "    \n",
    "    # contents don't include subject, so we extract it from the filename\n",
    "    # and append it to the DataFrame\n",
    "    subject = re.search('.*(\\d{6})', f).group(1)\n",
    "    _df['Subject'] = subject\n",
    "\n",
    "    dfs.append(_df)\n",
    "\n",
    "# Concatenate all DFs together along the row axis\n",
    "data = pd.concat(dfs, axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Inspecting the data\n",
    "The first thing we should do with any new dataset is inspect it in various ways to get a better sense of what it contains. Some things we might want to know:\n",
    "* How big is the dataset?\n",
    "* What do the first few rows look like?\n",
    "* How many columns contain meaningful information?\n",
    "* How are values distributed?\n",
    "* Are there missing values?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# Insert code here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preparing the data\n",
    "Most datasets require some amount of cleaning/wrangling/munging/reshaping/sanitizing/[insert your favorite verb]ing before they're ready for analysis. Some things we should consider:\n",
    "* Are missing or obviously erroneous values present, and if so, how should we handle them?\n",
    "* Are there subsets of the data we might not want to include?\n",
    "* Are there variables we want to drop or need to transform in some way?\n",
    "    * Are are our variables on reasonable scales?\n",
    "* Does the format of the data match what our analysis or visualization tools expect?\n",
    "    * E.g., should the data be in \"wide\" or \"long\" format?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Insert code here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# A deeper look\n",
    "In practice, data exploration and preparation is an iterative process: we typically clean up the data to address some obvious problems, then take another look, find new problems, and rinse and repeat. This cycle can last for a long time, which is why many data scientists \"joke\" that actual data analysis is the easy (and smaller) part of their job.\n",
    "\n",
    "Now that we've ensured basic sanity (hopefully!), we can start to probe the data in ways that might be a bit more scientifically interesting. At this point, the questions start to become increasingly domain-specific.\n",
    "\n",
    "Things we can ask about the HCP N-back data:\n",
    "* How does performance vary across different experimental conditions?\n",
    "* How does it covary across subjects?\n",
    "* How do the different performance metrics relate--e.g., RT and accuracy?\n",
    "* Are there outliers--either in terms of subjects or in terms of stimuli?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Insert code here!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Statistical inference\n",
    "Researchers often rush to get to statistical inference. That's probably a bad idea. Statistical analysis should ideally be conducted only after we feel we have a reasonable handle on some of the basic qualitative patterns present in our data. If we can't answer questions like \"is there much variation in performance across conditions\" *before* running a regression analysis, we should consider taking a step back. (Note that everything we're talking about in this notebook here falls squarely under the heading of exploratory analysis--and should be presented as such in talks, manuscripts, etc. If we want to claim that you're doing hypothesis testing, our hypotheses and analysis plan should all be written down in detail *before* we ever see the data. We can't decide how to clean and analyze your data after looking at it without running a serious risk of overfitting.)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "# Guess what this box is for? That's right! Insert code here!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}