Ex5.Predicting-house-prices.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://www.th-koeln.de/img/logo.svg\" style=\"float: right;\" width=\"200\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# 5th exercise: <font color=\"#C70039\">Predicting house prices by a regression</font>\n",
    "* Course: DIS21a.1\n",
    "* Lecturer: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Author of notebook modifications and adaptations: <a href=\"https://www.gernotheisenberg.de/\">Gernot Heisenberg</a>\n",
    "* Date: 12.06.2020\n",
    "\n",
    "<img src=\"https://miro.medium.com/max/876/0*YnFjHIvXcDv8oL3q.\" style=\"float: center;\" width=\"300\">\n",
    "\n",
    "**GENERAL NOTE 1**: \n",
    "Please make sure you are reading the entire notebook, since it contains a lot of information on your tasks (e.g. regarding the set of certain paramaters or a specific computational trick), and the written mark downs as well as comments contain a lot of information on how things work together as a whole. \n",
    "\n",
    "**GENERAL NOTE 2**: \n",
    "* Please, when commenting source code, just use English language only. \n",
    "* When describing an observation (for instance, after you have run through your test plan) you may use German language.\n",
    "This applies to all exercises in DIS 21a.1.  \n",
    "\n",
    "--------------------\n",
    "\n",
    "### <font color=\"ce33ff\">DESCRIPTION</font>:\n",
    "The previous exercises were dealing with classification problems, where the goal was to predict a single discrete/categorial label. As you have learned already, another common type of machine learning problem is *regression*, which consists of predicting a continuous value instead of a discrete label. \n",
    "For instance, predicting the temperature tomorrow, given meteorological data, or predicting the time that a \n",
    "software project will take to complete, given its specifications. \n",
    "Make sure, you do not mix up \"regression\" with the algorithm \"logistic regression\": confusingly, \"logistic regression\" is not a regression algorithm, it is a classification algorithm. :-)\n",
    "\n",
    "This notebook allows you for predict the median price of homes in a given Boston suburb in the mid-1970s, given a few data points about the suburb at the time, such as the crime rate, the local property tax rate, etc.\n",
    "The dataset we will be using has another interesting difference from our two previous examples: it has very few data points, only 506 in total, split between 404 training samples and 102 test samples, and each \"feature\" in the input data (e.g. the crime rate is a feature) has a different scale. For instance some values are proportions, which take a values between 0 and 1, others take values between 1 and 12, others between 0 and 100...\n",
    "\n",
    "-----------------------\n",
    "\n",
    "### <font color=\"FFC300\">TASKS</font>:\n",
    "The tasks that you need to work on in this notebook are always indicated below as bullet points. \n",
    "If a task is more challenging and consists of several steps, this is indicated as well. \n",
    "Make sure you have worked down the task list and commented your doings. \n",
    "This should be done by using markdown.<br> \n",
    "<font color=red>Make sure you don't forget to specify your name and your matriculation number in the notebook before submitting it.</font>\n",
    "\n",
    "**YOUR TASKS in this exercise are as follows**:\n",
    "1. import the notebook to Google Colab.\n",
    "2. make sure you specified you name and your matriculation number in the header below my name and date.\n",
    "    * set the date too and remove mine.\n",
    "3. read the entire notebook carefully\n",
    "    * add comments whereever you feel it necessary for better understanding\n",
    "    * run the notebook for the first time and note the result in your markdown result table (your test plan). \n",
    "4. go into the section 'building the ANN' \n",
    "    * add the missing code that does create a network as shown in the image in the lecture slides on page 179 (File: 'DIS21a.1-7.HANDS_ON.First.DLNetwork.Architectures.for.Solving.Three.Interesting.Problems.pdf')\n",
    "    * set the activation function to ReLu\n",
    "    * set the correct activation function in the last layer/the output. What is correct when doing a regression?\n",
    "    * add the missing code for compiling the network by setting\n",
    "        * the loss function (keep in mind you are building a regression model)\n",
    "        * the optimizer\n",
    "        * the evaluation metric (keep in mind you are building a regression model)\n",
    "5. optimize the hyperparameters of the model as you are still some good way off the actual price\n",
    "    * compute the minimum of the number of epochs which you can see in the chart\n",
    "    * increase the batch size\n",
    "    * adjust the size of the hidden layers\n",
    "6. build a production model and test on the test data\n",
    "\n",
    "----------------- "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "## START OF THE NOTEBOOK CODE\n",
    "----------------------------------------------------------------------------------------------------------------------"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import tensorflow\n",
    "tensorflow.keras.__version__"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### loading the house price data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras.datasets import boston_housing\n",
    "(train_data, train_targets), (test_data, test_targets) =  boston_housing.load_data()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_data.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "--------------\n",
    "\n",
    "### data preparation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, there are 404 training samples and 102 test samples. The data consists of 13 features. The 13 features in the input data are as follows:\n",
    "\n",
    "1. Per capita crime rate. (Pro-Kopf-Verbrechensrate)\n",
    "2. Proportion of residential land zoned for lots over 25.000 square feet.\n",
    "3. Proportion of non-retail business acres per town.\n",
    "4. Charles River dummy variable (= 1 if tract bounds river; 0 otherwise).\n",
    "5. Nitric oxides concentration (parts per 10 million).\n",
    "6. Average number of rooms per dwelling.\n",
    "7. Proportion of owner-occupied units built prior to 1940.\n",
    "8. Weighted distances to five Boston employment centres.\n",
    "9. Index of accessibility to radial highways.\n",
    "10. Full-value property-tax rate per US$ 10.000.\n",
    "11. Pupil-teacher ratio by town.\n",
    "12. 1000 * (Bk - 0.63) ** 2 where Bk is the proportion of black people by town. \n",
    "    * Unbelievable this attribute was taken into account (Anmerk. Gernot Heisenberg)\n",
    "13. % lower status of the population.\n",
    "\n",
    "If you want to read more about the Boston housing data set click here:\n",
    "<a href=\"https://www.kaggle.com/c/boston-housing\">https://www.kaggle.com/c/boston-housing</a>\n",
    "\n",
    "The targets are the median values of owner-occupied homes, in thousands of dollars. \n",
    "The prices are typically between US$ 10.000 - 50.000."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_targets.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_targets[0:100]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you have learned in the lecture it is going to be problematic to feed into an ANN values that all take wildly different ranges. The ANN might be able to automatically adapt to such heterogeneous data but it would definitely make learning more difficult since attributes are not comparable. \n",
    "Hence, standardization of the attributes is needed: \n",
    "* For each attribute in the input data (a column in the input data matrix),  subtract the mean of the attribute and divide by the standard deviation, so that the attribute will be centered around 0 and will have a unit standard deviation. \n",
    "* This is called z-score-standardization (see lecture slides) and can be easily done using python's standard lib:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute mean and stddev\n",
    "mean = train_data.mean(axis=0)\n",
    "train_data -= mean\n",
    "std = train_data.std(axis=0)\n",
    "train_data /= std\n",
    "\n",
    "test_data -= mean\n",
    "test_data /= std"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the quantities that are used for standardizing the test data have been computed using the training data. \n",
    "Never, never, never ever in your life, compute any quantity on test data, even not for something as simple as data standardization (compare lecture slides)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### building the ANN\n",
    "\n",
    "Because so few samples are available, it will be best using a small network. \n",
    "In general, the less training data there is, the worse overfitting will be, and using a small network is one (of several) ways to fight overfitting."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from tensorflow.keras import models\n",
    "from tensorflow.keras import layers\n",
    "\n",
    "# Because you will need to instantiate the same model multiple times, \n",
    "# it is better to implement a build function to construct the model.\n",
    "# Also compile the network here in this function.\n",
    "\n",
    "def build_model(be_smart_and_hand_over_all_parameters):\n",
    "    model = models.Sequential()\n",
    "\n",
    "    '''ADD THE MISSING CODE HERE'''\n",
    "    '''LOOK AT THE TASK LIST TO SEE WHAT ARCHITECTURE THE NETWORK SHALL HAVE '''\n",
    "\n",
    "    # your code\n",
    "\n",
    "    return model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### validating the model using k-fold validation\n",
    "\n",
    "Since you have learned during the lectures, that simple hold-out validation, is the way to go, but not, when data is really small. The best practice in such situations is to use k-fold cross-validation. It consists of splitting the available data into k partitions (typically k= 4 or 5), then instantiating k identical models, and training each one on k-1 partitions while evaluating on the remaining partition. The validation score for the model used would then be the average of the k validation scores obtained.\n",
    "\n",
    "In terms of code, this is straightforward!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "\n",
    "k = 4 \n",
    "num_val_samples = len(train_data) // k # floor/integer division (see https://www.programiz.com/python-programming/operators also)\n",
    "num_epochs = 100\n",
    "all_scores = []\n",
    "\n",
    "# loop over all folds, k-times\n",
    "for i in range(k):\n",
    "    print('processing ' + str(i+1) + '.fold')\n",
    "    \n",
    "    # Prepare the validation data: data from partition # k\n",
    "    val_data = train_data[i * num_val_samples: (i + 1) * num_val_samples]\n",
    "    val_targets = train_targets[i * num_val_samples: (i + 1) * num_val_samples]\n",
    "\n",
    "    # Prepare the training data: data from all other partitions\n",
    "    partial_train_data = np.concatenate(\n",
    "                            [train_data[:i * num_val_samples],\n",
    "                             train_data[(i + 1) * num_val_samples:]],\n",
    "                            axis=0)\n",
    "    \n",
    "    partial_train_targets = np.concatenate(\n",
    "                            [train_targets[:i * num_val_samples],\n",
    "                             train_targets[(i + 1) * num_val_samples:]],\n",
    "                            axis=0)\n",
    "\n",
    "    # Build the model (each model should be build with the same parameters)\n",
    "    model = build_model(be_smart_and_hand_over_all_parameters) # forgotten ??? :-)\n",
    "    \n",
    "    # Train the model (set the silent mode by verbose=0)\n",
    "    model.fit(partial_train_data, partial_train_targets, epochs=num_epochs, batch_size=1, verbose=0)\n",
    "    \n",
    "    # Evaluate the model on the validation data\n",
    "    val_mse, val_mae = model.evaluate(val_data, val_targets, verbose=0)\n",
    "    all_scores.append(val_mae) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# show the scores each fold has predicted on the validation data\n",
    "all_scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# take the average of all scores\n",
    "np.mean(all_scores)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, the different runs do indeed show rather different validation scores, from 2.0 to 2.77. \n",
    "Their average (2.4) is a much more reliable metric than any single of these scores -- that's the entire point of K-fold cross-validation. \n",
    "\n",
    "In this case, we are off by approximately USD 2.400 on average, which is still significant considering that the prices range from US$ 10.000-50.000. \n",
    "\n",
    "-------------"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### new training and validation\n",
    "Let's train the network for a bit longer: 500 epochs. \n",
    "To keep a record of how well the model did at each epoch, we will modify our training loop to save the per-epoch validation score log:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Do some memory clean up\n",
    "from tensorflow.keras import backend as K\n",
    "K.clear_session()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "num_epochs = 500\n",
    "all_mae_histories = []\n",
    "\n",
    "for i in range(k):\n",
    "    print('processing ' + str(i+1) + '.fold')\n",
    "\n",
    "    # Prepare the validation data: data from partition # k\n",
    "    val_data = train_data[i * num_val_samples: (i + 1) * num_val_samples]\n",
    "    val_targets = train_targets[i * num_val_samples: (i + 1) * num_val_samples]\n",
    "\n",
    "    # Prepare the training data: data from all other partitions\n",
    "    partial_train_data = np.concatenate(\n",
    "                        [train_data[:i * num_val_samples],\n",
    "                         train_data[(i + 1) * num_val_samples:]],\n",
    "                        axis=0)\n",
    "    \n",
    "    partial_train_targets = np.concatenate(\n",
    "                        [train_targets[:i * num_val_samples],\n",
    "                         train_targets[(i + 1) * num_val_samples:]],\n",
    "                        axis=0)\n",
    "\n",
    "    # Build the model (each model should be build with the same parameters)\n",
    "    model = build_model(be_smart_and_hand_over_all_parameters)\n",
    "    \n",
    "    # Train the model (in silent mode, verbose=0)\n",
    "    '''Write everything to the history object and explore it afterwards (see below)'''\n",
    "    history = model.fit(partial_train_data, partial_train_targets,\n",
    "                        validation_data=(val_data, val_targets),\n",
    "                        epochs=num_epochs, batch_size=1, verbose=0)\n",
    "    \n",
    "    print(\"history.history.keys = \", history.history.keys())\n",
    "    print(\"--------------------------------------------------\")\n",
    "    mae_history = history.history['val_mae']\n",
    "    all_mae_histories.append(mae_history)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Compute the average of the per-epoch MAE scores for all folds"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "average_mae_history = [np.mean([x[i] for x in all_mae_histories]) for i in range(num_epochs)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's plot the average_mae_history scores using pyplot"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "plt.plot(range(1, len(average_mae_history) + 1), average_mae_history)\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation MAE score')\n",
    "plt.title(\"Averaged validation MAE score per epoch\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "It may be a bit hard to see the plot due to scaling problems (high value at the beginning) \n",
    "and the high variance of the data.\n",
    "\n",
    "For improving this, let's do the following:\n",
    "\n",
    "1. Omit the first 10 data points, which are on a different scale from the rest of the curve.\n",
    "2. Replace each point with an exponential moving average of the previous points, to obtain a smooth curve.\n",
    "    * this is a common technique for dealing with signals (i.e. time series) of any kind, especially financial stock data.\n",
    "\n",
    "Even though, you can also scale the chart bigger if you want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def smooth_curve(points, factor=0.9):\n",
    "  smoothed_points = []\n",
    "  for point in points:\n",
    "    if smoothed_points:\n",
    "      previous = smoothed_points[-1]\n",
    "      smoothed_points.append(previous * factor + point * (1 - factor))\n",
    "    else:\n",
    "      smoothed_points.append(point)\n",
    "  return smoothed_points\n",
    "\n",
    "smooth_mae_history = smooth_curve(average_mae_history[10:]) # get rid of the first 10 data points\n",
    "\n",
    "plt.plot(range(1, len(smooth_mae_history) + 1), smooth_mae_history)\n",
    "plt.xlabel('Epochs')\n",
    "plt.ylabel('Validation MAE score')\n",
    "plt.title(\"Smoothed (f=0.9) average validation MAE score per epoch\")\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "According to this plot, it seems that validation MAE stops improving significantly after some epochs, and the model starts overfitting."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Your  task now (see task list items 5 and 6 also)\n",
    "\n",
    "5. optimize the hyperparameters of your model\n",
    "    * try to compute the minimum number of epochs after the model starts overfitting\n",
    "    * adjust the batch size\n",
    "    * adjust the number of hidden layers\n",
    "    * <font color=\"#C70039\">Work again with a table to not get lost and reflect a good test plan</font>\n",
    "    \n",
    "6. Now, you can training a final \"production\" model on all of the training data, with the best parameters, then look at its performance on the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Get a fresh, compiled model and optimize all hyperparameters\n",
    "model = build_model(be_smart_and_hand_over_all_parameters)\n",
    "\n",
    "# Train it on the entirety of the data.\n",
    "'''ADD THE MISSING CODE HERE'''\n",
    "'''LOOK AT THE TASK LIST TO SEE WHAT YOU HAVE TO DO '''\n",
    "\n",
    "# your code\n",
    "\n",
    "test_mse_score, test_mae_score = model.evaluate(test_data, test_targets)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "test_mae_score # print out the score"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}