05-aggregate_wozwaarde.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Journalism\n",
    "## Example \"WOZ-waarde\"\n",
    "\n",
    "*Damian Trilling and Penny Sheets*\n",
    "\n",
    "This week, we will particular look at techniques for aggregating data and for joining datasets.\n",
    "We use data on housing prices from https://data.amsterdam.nl/ ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As always, we first import some modules we need and load our data file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "%matplotlib inline"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv('wozwaarde.csv', delimiter=';')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Cleaning up and recoding\n",
    "When we inspect the data, we see that each \"wijk\" seems to be represented by a code (a letter plus two numbers). Essentially, the code is the first word of the \"wijk\" column. Let's put it into a new coloumn.\n",
    "\n",
    "The lambda function says: Take each cell, call the value `x`, split `x` into a list of words (we did that in the week on analyzing text), and then return the 0st element. We then put the result into a new column called `code`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df['code'] = df['wijk'].map(lambda x: x.split()[0])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If we now inspect the rows closer, we can see that the \"wijken\" have a letter and a two-digit numerical code, whereas some rows (e.g., row 10) contain information on the \"stadsdeel\" level, signified by only the letter.\n",
    "\n",
    "This tells us something about the data structure.\n",
    "\n",
    "It seems problematic that both are in the same tabel, given that the \"stadsdeel\" rows are essentially contain aggregated data from the \"wijken\". We could check that, but it seems very reasonable to assume this, based on just looking at the dataframe.\n",
    "\n",
    "Probably, it's a good idea to seperate our dataframe into two different dataframes.\n",
    "\n",
    "We can do so by just checking the length of the value in the `code` column that we created."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select only the \"stadsdelen\" and put them into a new dataframe\n",
    "stadsdelen = df[df['code'].map(lambda x: len(x)==1)]\n",
    "stadsdelen['wijk'] = stadsdelen['wijk'].map(lambda x: x[2:])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's read the information which letter is associated with which \"stadsdeel\" into a dictionary, that we can later use for recoding."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stadsdeelcodes = {}\n",
    "for k, v in stadsdelen[['wijk','code']].to_dict(orient='index').items():\n",
    "    stadsdeelcodes.update({v['code']: v['wijk']})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "stadsdeelcodes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# put all \"wijken\" (which have a code that is longer than 1) into a new dataframe, \n",
    "# and remove their code (the first 4 characters) from their name\n",
    "wijken = df[df['code'].map(lambda x: len(x)>1)]\n",
    "wijken['wijk'] = wijken['wijk'].map(lambda x: x[4:])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now use the dictionry that we made above to automatically code in which stadsdeel a wijk is located (by looking up the first character (i.e., the letter) of their code in the dictionary `stadsdeelcodes` and putting the corresponding value in a new column, `stadsdeel`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken['stadsdeel'] = wijken['code'].map(lambda x: stadsdeelcodes[x[:1]])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken.columns"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## From wide to long\n",
    "If you look at the dataframe `wijken`, you will see that it is structured in a so-called *wide* format. \n",
    "That means that you have multiple measurements of the same thing (the house vlaues) in different columns, depending on in which year it was measured. \n",
    "\n",
    "In other words: there is no column (variable) `year` that would tell you when a measurement has taken place, but this information is essentially encoded in the column names.\n",
    "\n",
    "For many analyses, this is quite unfortunate. After all, we cannout do sth like `.groupby('year')` in a dataset that is formatted this way.\n",
    "\n",
    "We will therefore transform it into a more tidy format, a *long* format. The `.melt()` method allows us to do so. We need to specify which variables stay the same and identify the cases (`id_vars`), which columns contain the values (`value_vars`), and how the two new variables to store the old column names and the cell entries in (`var_name` and `value_name`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken_long = wijken.melt(id_vars=['wijk','stadsdeel'], \n",
    "                          value_vars=['2014', '2015', '2016', '2017', '2018'],\n",
    "                          value_name='woz-waarde',\n",
    "                          var_name = 'year')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken_long"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's save it for future usage (and for some other notebooks in the next weeks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken_long.to_csv('wijken_long.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Some analysis with `.groupby()` and `.agg()`\n",
    "\n",
    "Have a look at the slides for more info on `.groupby()` and `.agg()`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "#wijken_long.index = pd.DatetimeIndex(wijken_long.year.map(lambda x: \"1-1-{}\".format(x)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken_long.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken_long.groupby('year').agg(np.mean).plot(xticks=[0,1,2,3,4])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken_long.groupby(['year','stadsdeel']).agg(np.mean).unstack().plot(\n",
    "    figsize=[10,7], xticks=range(5))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As also explained on the slides, the `.unstack()` part is needed to flatten the hierarchical index that grouping by *two* variables creates. If we want to read the table, we don't need to unstack it, but we cannot directly plot the stacked table. Try it with and without! "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wijken_long.groupby(['year','stadsdeel']).agg(np.mean).unstack()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oost = wijken_long[wijken_long['stadsdeel']=='Oost']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oost.groupby(['year','wijk']).agg(np.mean).unstack().plot(figsize=[20,15])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oost[oost.wijk=='Dapperbuurt']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "oost[oost.wijk.str.startswith('Indisch')]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "wijken_long.groupby(['year','stadsdeel'])['woz-waarde'].agg([min,max])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Difference between most expensive and least expensive buurt within stadsdeel')\n",
    "wijken_long.groupby(['year','stadsdeel'])['woz-waarde'].agg(lambda x: max(x)-min(x)).unstack().plot()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print('Difference between most expensive and least expensive buurt within stadsdeel')\n",
    "wijken_long[wijken_long['stadsdeel']!= 'Westpoort'].groupby(\n",
    "    ['year','stadsdeel'])['woz-waarde'].agg(lambda x: max(x)-min(x)).unstack().plot()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We see that differences within stadsdeel rise."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Now it's your turn.\n",
    "\n",
    "Think of other aggregations, either with this or with other datasets!"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.8"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}