Bonus - Classify Tumors with Machine Learning.ipynb

{
    "nbformat_minor": 1, 
    "cells": [
        {
            "source": "# Classify Tumors with Machine Learning\n \nThis notebook explains how to use machine learning to classify tumor data. You develop your solution in three parts as follows:\n\n* An intuitive introduction to supervised learning concepts\n\n* A basic example of a machine learning model. \n\n* A deep dive into model stacking and parameter tuning, both of which are used in practice to significantly improve predictive accuracy\n\nSome guidelines for reading this notebook:\n\n* If you have no experience with machine learning: follow the entire notebook for a comprehensive walkthrough.\n\n* If you would like to read about decision trees, go to [section 2](#bullet-8).\n\n* If you would like to read about using XGBoost in practice, go to [section 3](#bullet-13).\n\nSome familiarity with Python is recommended. This notebook runs on Python 2 with Spark 2.0.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Table of contents\n1.0 [Introduction to supervised learning](#bullet-1)<br/>\n2.0 [Basic model: decision trees](#bullet-8)<br/> \n3.0 [Ensemble model: gradient boosting](#bullet-13)  \n4.0 [XGBoost: parameter tuning](#bullet-17)\n    ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## 1.0 Introduction to supervised learning<a class=\"anchor\" id=\"bullet-1\"></a>\n\n1.1 [What is machine learning?](#bullet-2)<br/>\n    1.2 [Defining the task](#bullet-3)<br/>\n    1.3 [How does an algorithm learn?](#bullet-4)<br/>\n    1.4 [Data preview](#bullet-5)<br/>\n    1.5 [Pre-processing](#bullet-6)<br/>\n    1.7 [Create train and test sets](#bullet-7)<br/>\n\n### 1.1 What is machine learning?<a class=\"anchor\" id=\"bullet-2\"></a>\n\nYou use an algorithm (code) to create a model (mathematical function). The algorithm tries to find patterns in a sample of **training data** that can be used on future data. The model itself is a summary of these patterns, distilled into mathematical relationships between variables. The algorithm depends on **hyperparameters** that control how it looks for these patterns (that is, how it learns). \n\nThe difficult part is to find parameters that balance accuracy and precision of the model with its ability to generalize.\n\n**Note:**\n* Accuracy and precision of predictions depend on the complexity of the model\n* Ability to generalize depends on how conservative the model is (how quickly it concludes that something is a pattern)\n\nThis balancing act is called the **bias-variance tradeoff**, because a very complex model will represent all the specific nuances of its training data, therefore failing to generalize. In the opposite case, a simple model does not pick up enough of a pattern to ensure accuracy and precision of future predictions. \n\nYour job* is to pick an efficient algorithm and suitable model for our data set and learning task, and fine-tune its parameters to produce a balanced model.\n\n**apart from the arduous task of data pre-processing, which this notebook skims over for the sake of brevity (and sanity!)*\n\n\n### 1.2 Defining the task<a class=\"anchor\" id=\"bullet-3\"></a>\n\nThe goal is to use an augmented version <a href=\"https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29\" target=\"_blank\" rel=\"noopener noreferrer\">this data set</a> to develop a predictive model which classifies breast tumors as malignant or benign depending on measurements of the tumor cells. The [augmented data set](https://apsportal.ibm.com/exchange/public/entry/view/c173693bf48aeb22e41bbe2b41d79c1f) includes statistical analysis values, such as the mean, for some of the measurements. In machine learning terms, the **target** (what you want to predict) is the diagnosis, and the **features** are the measurements. The data sample contains **observations** of tumor cases described by these two components.\n\n\n### 1.3 How does an algorithm learn?<a class=\"anchor\" id=\"bullet-4\"></a>\n\n1. Initialize the algorithm to produce a default model, and give it the feature data. The algorithm makes its first set of predictions (close to a random guess).\n2. The algorithm measures the error between its previous prediction and the true value of the targets. \n3. The algorithm adjusts its model-building to make the error smaller. Each adjustment represents a part of the pattern it is learning, and is used to process future data.\n4. It continues to predict, calculate error, and adjust.\n\nRemember that the algorithm is only using a training sample to perform step 3. If the algorithm makes an adjustment so that its predictions are *exactly* the targets, then it is **overfitted** to the training data. This means that it has lost the ability to generalize to new (unseen) observations, because it picked up a pattern that only applies to the data points it learned from.\n\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "#### Load data\nTo load the data:\n1. Go to the [data set](https://apsportal.ibm.com/exchange/public/entry/view/c173693bf48aeb22e41bbe2b41d79c1f) on the DSX Community. \n1. Click the download icon and save the data set to your computer.  \n1. Load the `BreastCancerWisconsonDataSet.csv` file into your notebook. Click the **Find and Add Data** icon on the notebook action bar. Drop the file into the box or browse to select the file. The file is loaded to your object storage and appears in the Data Assets section of the project. For more information, see <a href=\"https://datascience.ibm.com/docs/content/analyze-data/load-and-access-data.html\" target=\"_blank\" rel=\"noopener noreferrer\">Load and access data</a>.\n1. To load the data from the `BreastCancerWisconsonDataSet.csv` file into a pandas DataFrame, click in the next code cell and select **Insert to code > pandas DataFrame** under the file name.\n1. Make sure the object is named `df` instead of `df_data_1` in the last two lines of the **Insert to Code** block and then run the cell.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [], 
            "source": "# The code was removed by DSX for sharing."
        }, 
        {
            "source": "### 1.4 Data preview<a class=\"anchor\" id=\"bullet-5\"></a>\n\nIn the following data table, each row corresponds to one observation of a tumor. The columns contain the measurement features, with one column containing the diagnosis target.", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [], 
            "source": "from IPython.display import display\nprint df.shape\ndisplay(df[200:210])"
        }, 
        {
            "source": "The data contains features extracted from 569 diagnostic images of breast tumors. The diagnosis column indicates whether the mass was benign (B), or malignant (M). The rest of the columns contain features which are structured as follows:\n\n10 variables describe the cell nuclei of each mass, and for each variable, the mean, standard deviation, and 'worst' (mean of three largest measurements) are calculated. The variables are briefly described on the <a href=\"https://archive.ics.uci.edu/ml/datasets/Breast+Cancer+Wisconsin+%28Diagnostic%29\" target=\"_blank\" rel=\"noopener noreferrer\">original data set page</a>.\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 1.5 Pre-processing<a class=\"anchor\" id=\"bullet-6\"></a>\n\nThe following code cell gently cleans the data and generates a summary that can be used to check for outliers. This data set is well prepared for analysis and does not require further manipulation.\n\nIf you don't have scikit-learn library installed, install it.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!pip install -U scikit-learn"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [], 
            "source": "from sklearn.preprocessing import LabelEncoder\n\n# checking for missing values\ndf.isnull().any()\n\n# dropping Unnamed column\n# df2 = df.drop(['Unnamed: 32'], axis=1)\n\ndf2 = df\n\n# converting diagnosis M/B to numerical\nlenc = LabelEncoder()\nlenc.fit(df['diagnosis'])\ndf2['diagnosis'] = lenc.transform(df2['diagnosis'])\n\n# overview of data sample\ndf2.describe()"
        }, 
        {
            "source": "### 1.6 Create train and test sets<a class=\"anchor\" id=\"bullet-7\"></a>\n\nTake 80% of the total data to train the model, and reserve 20% of it for testing. After the model is built and tuned upon the training set, you can use the testing set to observe how well the model is able to generalize to new 'unseen' data.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [], 
            "source": "from sklearn.model_selection import train_test_split\n\n## X are all the features (columns) that might be useful to the model\n## y is the target (diagnosis column)\nX, y = df2.ix[:,'radius_mean':], df2[['diagnosis']]\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0, stratify=y)"
        }, 
        {
            "source": "## 2.0 Basic model: decision trees<a class=\"anchor\" id=\"bullet-8\"></a>\n\n2.1 [Mimicking human decision-making](#bullet-9)<br/>\n    2.2 [Interpreting a tree graphic](#bullet-10)<br/>\n    2.3 [How a decision tree learns](#bullet-11)<br/>\n    2.3 [Evaluating the model](#bullet-12)<br/>\n\n### 2.1 Mimicking human decision-making<a class=\"anchor\" id=\"bullet-9\"></a>\n\nA decision tree is an algorithm that can be used for machine learning. It is analogous to the game <a href=\"http://www.wikihow.com/Play-20-Questions\" target=\"_blank\" rel=\"noopener noreferrer\">20 questions</a> - each 'question' is a **splitting feature** chosen from the data based on how useful it is for identifying the target. The algorithm decides on these features by optimizing a variety of mathematical functions that quantify prediction error (<a href=\"https://en.wikipedia.org/wiki/Decision_tree_learning#Metrics\" target=\"_blank\" rel=\"noopener noreferrer\">for the curious</a>).\n\nTrees are considered weak learners, because their predictions are usually only slightly better than chance. The following code builds a tree based on the tumor data, and displays the process graphically.\n\nIf you don't have pydotplus library installed, install it.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!pip install pydotplus"
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from sklearn import tree\nimport pydotplus\nfrom IPython.display import Image \n\n# list of features we want to consider\nsplitting_features = [x for x in X_train.columns if x not in ['id']]\n\n# initializing the tree model and training it\ntree_model = tree.DecisionTreeClassifier()\ntree_model = tree_model.fit(X_train, y_train)\n\n# generating a graphic for the tree\ndot_data = tree.export_graphviz(tree_model, out_file=None, \n                         feature_names=splitting_features,\n                         class_names = ['Benign', 'Malignant'],\n                         filled=True, rounded=True,  \n                         special_characters=True)  \ngraph = pydotplus.graph_from_dot_data(dot_data)  \nImage(graph.create_png())"
        }, 
        {
            "source": "### 2.2 Interpreting a tree graphic<a class=\"anchor\" id=\"bullet-10\"></a>\nEach rectangle represents a **node** of the tree.\n* The first line in each node identifies the splitting feature and decision being made upon it.\n* The observations are separated based on this decision into **children** nodes (left and right).\n* **Samples**: how many observations have been filtered by that decision.\n* **Value**: where how many observations are being split into the left and right children respectively.\n* **Gini**: a measurement of incorrect classification. As the algorithm learns, the gini inequality decreases to 0.\n* **Class**: the target that the observations have been grouped into: benign or malignant.\n* The rectangles at the bottom of the diagram are called **leaves**, which contain the final predictions (observations classified by diagnosis).\n\n### 2.3 How a decision tree learns<a class=\"anchor\" id=\"bullet-11\"></a>\n\nApplying the general learning outline ([section 1.3](#bullet-4)) to a decision tree:\n\n*1. Initialize the algorithm to produce a default model, and give it the feature data. The algorithm makes its first set of predictions (close to a random guess).*<br/>\nThe default model is the very first splitting feature - the algorithm chose 'concave points mean <= 0.0492'. The observations are divided into benign or malignant based on this decision, making up the first set of predictions!\n\n*2. The algorithm measures the error between its previous prediction and the true value of the targets.*<br/> \nThis error is quantified by the gini value that appears in the first node of the tree.\n\n*3. The algorithm adjusts its model-building to make the error smaller. Each adjustment represents a part of the pattern it is learning, and is used to process future data.*<br/>\nThe adjustment is the next splitting feature that is chosen for each subgroup that resulted from the concave points mean decision. The algorithm chose 'radius worst' for observations that had a concave points mean less than the threshold, and 'concavity worst' for observations with concave points mean greater than the threshold.\n\n*4. It continues to predict, calculate error, and adjust.*<br/>\nThe next set of predictions are the classifications that result from the second round of decision making. Each time, a gini value is calculated to indicate how far the classifications are from the true targets, and another set of splitting features are chosen to further refine the class groupings. \n\n### 2.4 Evaluating the model<a class=\"anchor\" id=\"bullet-12\"></a>\nPick a metric to measure predictive accuracy or performance of your model, so that you can compare different models. \n\nModels can differ in two ways:\n\n1. The algorithm used for learning (for example, the difference between a decision tree and gradient boosting)\n2. The **parameter** selection within a single algorithm (for example, a model with a learning rate set to 0.1 or 0.0)\n\nThis case refers to (1), you want to know how the tree model performs, so that it can be compared to the gradient boosting model in the next section.\n\nThe choice of evaluation metric depends on the type of learning problem. For binary classification, you can use a metric called the <a href=\"https://www.quora.com/Machine-Learning-What-is-an-intuitive-explanation-of-AUC\" target=\"_blank\" rel=\"noopener noreferrer\">AUC score</a>, which measures the probability that the model correctly identifies *malignant tumors only*. This metric is used as an indication of performance because misclassifying a malignant tumor as benign is the worst prediction scenario (compared to correct classification of benign/malignant and misclassification of benign).\n\nYou also calculate the general accuracy of the model - the percentage of observations that have been correctly classified overall. The following code uses your simple tree model to predict on the testing set, and evaluates the accuracy and AUC score of its predictions:", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from sklearn import metrics\n\n#Predict test set:\ntest_predictions = tree_model.predict(X_test[splitting_features])\ntest_predprob = tree_model.predict_proba(X_test[splitting_features])[:,1]\n        \n#Print model report:\nprint \"Accuracy : %.4g\" % metrics.accuracy_score(y_test['diagnosis'].values, test_predictions)\nprint \"AUC Score: %f\" % metrics.roc_auc_score(y_test['diagnosis'], test_predprob)"
        }, 
        {
            "source": "The model gives ~92% accuracy for all cases, and a ~93% probability of catching malignant cases in the data. Each time the code is run, the algorithm may learn slightly differently (that is, select different splitting features), which results in slight variations in accuracy and AUC scores. In the following section, you work on improving the model by stacking together many trees, and tuning parameters for better generalization ability.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## 3.0 Ensemble model: gradient tree boosting<a class=\"anchor\" id=\"bullet-13\"></a>\n\nStacking, or meta-ensembling, is a method of joining multiple predictive models so that the strengths of each can cover the weaknesses of others. Gradient boosting is one technique used to stack weak learners together in an **ensemble** to achieve better predictions with each additional model.\n\n3.1 [An intuitive explanation](#bullet-14)<br/>\n    3.2 [XGBoost](#bullet-15)<br/>\n    3.3 [K-fold cross validation](#bullet-16)<br/>\n\n### 3.1 An intuitive explanation of gradient boosting<a class=\"anchor\" id=\"bullet-14\"></a>\n\n1. A tree is used to learn from the data as described in [section 2.2](#bullet-10). This first model is called the **base learner**.\n2. The algorithm calculates the error of the base learner's final predictions. These error measurements are called **pseudo-residuals**. \n3. A second decision tree, called the **booster**, is stacked on top of the base tree by using the pseudo-residuals as input. Its output is a prediction of error for each prediction of the base tree. \n4. The algorithm adjusts the base prediction by adding the predicted amount of error.\n*Note: target - prediction = error, so by adding 'predicted error' to the base prediction, you are getting closer to the target.*\n5. Another set of pseudo-residuals are calculated for the predictions of the updated model (base tree + first boosting tree). Each round of boosting corrects the base prediction by a predicted amount of error, gradually inching towards the true value.\n\n\n### 3.2 XGBoost<a class=\"anchor\" id=\"bullet-15\"></a>\n\nYou use the XGBoost library, an implementation of gradient tree boosting popularized by usage in machine learning competitions. It is a valuable addition to any machine learning toolkit, as it significantly outperforms other algorithms in speed, accuracy, and flexibility. \n\nIf you don't have xgboost library installed, install it.", 
            "cell_type": "markdown", 
            "metadata": {
                "collapsed": true
            }
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "!pip install xgboost"
        }, 
        {
            "source": "**Note:** The xgboost package uses an older version of sklearn. When you run import xgboost, ignore the DeprecationWarning. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import xgboost as xgb\nfrom xgboost.sklearn import XGBClassifier"
        }, 
        {
            "source": "### 3.3 K-fold cross validation<a class=\"anchor\" id=\"bullet-16\"></a>\n\nCross-validation is used to measure how well a model generalizes using the training set (no need to bring in the testing set yet!). The k-fold method divides the training data into even smaller train/test sets to gauge how the model performs on 'unseen' data.\n\n1. The data is divided into k subsamples. You use 5 subsamples: you can call them A, B, C, D, E.\n2. The model is trained on 4 subsamples (for example, A, B, C, D).\n3. The model makes predictions on the subsample that was held out from training (E), and you score it based on an evaluation metric.\n4. The model is then trained on another combination of subsamples (for example, B, C, D, E), and tested on the one that was held out (A).\n5. This repeats for every possible combination of k-folds. You take the average of the evaluation metrics at each iteration to give an idea of how the model performs on actual unseen data.\n\nYou use cross-validation during parameter tuning so that you can directly observe the effect of a parameter on a model's generalization ability.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from sklearn import cross_validation"
        }, 
        {
            "source": "## 4.0 XGBoost: Parameter tuning<a class=\"anchor\" id=\"bullet-17\"></a><a href=\"#footnote-1\"><sup>[1]</sup></a>\n\n4.1 [Default model](#bullet-18)<br/>\n    4.2 [Tuning number of estimators](#bullet-19)<br/>\n    4.3 [Evaluating default model](#bullet-20)<br/>\n    4.4 [Interpreting evaluation](#bullet-21)<br/>\n    4.5 [Grid search](#bullet-22)<br/>\n    4.6 [Tuning tree depth and min child weight](#bullet-23)<br/>\n    4.7 [Tuning gamma](#bullet-24)<br/>\n    4.8 [Evaluating updated model](#bullet-25)<br/>\n    4.9 [Tuning sampling parameters](#bullet-26)<br/>\n    4.10 [Tuning lambda & alpha](#bullet-27)<br/>\n    4.11 [Evaluating final model](#bullet-28)<br/>\n    4.12 [Predict on the test set](#bullet-29)<br/>\n\n### 4.1 Setting up a default model<a class=\"anchor\" id=\"bullet-18\"></a>\nYou need to initialize a model with some default parameters as a starting point. The selection here depends on the nature of your data and your experience. Recall that parameters tell the algorithm how to learn in one of two ways: how complex the model will be, and how quickly it should conclude on a pattern in the data.\n\nLuckily, XGBoost performs well with its default values, so you only need to define the following three parameters: \n\n1. objective: Depends on the target type - use binary:logistic because you have a binary classification problem.\n2. learning rate: This parameter is always chosen first, as the optimal values of all other parameters are dependent upon it. It directly controls how conservative the model is in picking up patterns - a smaller value makes the model more conservative. A good rule of thumb is to initialize the learning rate at 0.1.  \n3. number of estimators: The maximum number of boosting trees to be added. Pick a number that is a bit larger than necessary for our data set size. \n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# initializing our first model with an objective and learning rate\nxgb0 = XGBClassifier(\n objective= 'binary:logistic',\n learning_rate = 0.1, \n n_estimators = 30)"
        }, 
        {
            "source": "### 4.2 Tuning number of estimators<a class=\"anchor\" id=\"bullet-19\"></a>\n\nThe 'number of estimators' parameter in XGBoost refers to the maximum number of boosting trees (or rounds) to be used in building the model. Intuitively, the more trees that are added to the model, the more complex it is. This parameter is tuned first, so you can broadly adjust the complexity of your model before making adjustments that have smaller impacts on conservative learning.\n\nThe function below performs the following actions to find the best number of boosting trees to use on your data:\n\n1. Trains an XGBoost model using features of the data.\n2. Performs k-fold cross validation on the model, using accuracy and AUC score as the evaluation metric.\n3. Returns output for each boosting round so you can see how the model is learning. You will look at the detailed output in the next section.\n4. It stops running after the cross-validation score does not improve significantly with additional boosting rounds, giving you an optimal number of estimators for the model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "import matplotlib.pylab as plt\n%matplotlib inline\n\ndef evaluate_model(alg, train, target, predictors, cv_folds=5, early_stopping_rounds=1):\n    \n    xgb_param = alg.get_xgb_params()\n    xgtrain = xgb.DMatrix(train[predictors].values, target['diagnosis'].values)\n    cvresult = xgb.cv(xgb_param, xgtrain, num_boost_round=alg.get_params()['n_estimators'], nfold=cv_folds,\n        metrics='auc', early_stopping_rounds=early_stopping_rounds, verbose_eval=True)\n    alg.set_params(n_estimators=cvresult.shape[0])\n    \n    #Fit the algorithm on the data\n    alg.fit(train[predictors], target['diagnosis'], eval_metric='auc')\n        \n    #Predict training set:\n    dtrain_predictions = alg.predict(train[predictors])\n    dtrain_predprob = alg.predict_proba(train[predictors])[:,1]\n        \n    #Print model report:\n    print \"\\nModel Report\"\n    print \"Accuracy : %.4g\" % metrics.accuracy_score(target['diagnosis'].values, dtrain_predictions)\n    print \"AUC Score (Train): %f\" % metrics.roc_auc_score(target['diagnosis'], dtrain_predprob)\n                    \n    feat_imp = pd.Series(alg.booster().get_fscore()).sort_values(ascending=False)\n    feat_imp.plot(kind='bar', title='Feature Importance', color='g')\n    plt.ylabel('Feature Importance Score')"
        }, 
        {
            "source": "### 4.3 Evaluating the default model<a class=\"anchor\" id=\"bullet-20\"></a>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# a list of features to be used for training the model\nfeatures = [x for x in X_train.columns if x not in ['id']]\n\n# evaluating the first model\nevaluate_model(xgb0, X_train, y_train, features)"
        }, 
        {
            "source": "### 4.4 Interpreting evaluation output<a class=\"anchor\" id=\"bullet-21\"></a>\n\n* The first set of readings shows you the cross-validation scores at each boosting round. So at line [0], one tree has been fit to the data. Train-auc gives you the average AUC score for the five training sets during cross-validation, and test-auc gives you the same metric for the five testing sets during cross validation. Naturally, the test score is lower, as the model is predicting on data that was not used to train it. You can see that the test-auc catches up with more boosting rounds, as more trees are used to minimize the error.\n<br/><br/>\n* The model stops training after ten rounds, which means that the cross-validation scores are not significantly improving with more than 10 rounds. For a learning rate of 0.1, you can take the number of estimators to be 10 to save computation time.\n<br/><br/>\n* Overall, the accuracy is 0.9824, which tells you how many cases were correctly classified, both benign and malignant.\n<br/><br/>\n* Overall, the AUC score is 0.99877, which indicates that the model rarely lets malignant cases slip through its screening process.\n<br/><br/>\n* The graph displays the features ranked by their importance to the model. Feature importance refers to the number of times it is used as a splitting feature in the decision trees. This function is especially useful in large datasets, where many variables are just noise and <a href=\"https://en.wikipedia.org/wiki/Feature_engineering\" target=\"_blank\" rel=\"noopener no referrer\">feature engineering</a> may be required.\n\n### 4.5 Grid search<a class=\"anchor\" id=\"bullet-22\"></a>\nTo tune the remaining XGBoost parameters, use grid search with cross-validation.\n\n1. Define set of values for the parameter in question. These values of interest again depend on the nature of the data and your experience.\n2. The grid search function systematically produces a model for each unique combination of these values, and evaluates it using cross-validation.\n3. The optimal parameter values are found in the model with the highest cv score.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "from sklearn.model_selection import GridSearchCV"
        }, 
        {
            "source": "### 4.6 Tuning max depth and child weight<a class=\"anchor\" id=\"bullet-23\"></a>\n\n**max depth**: The size of each new decision tree. Smaller trees = less complexity. <br/>\n**min child weight**: The minimum number of observations that must be in the children after a split. Smaller weight = more conservative.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# updating our default model with the optimal number of estimators\nxgb1 = XGBClassifier(\n objective = 'binary:logistic',\n learning_rate =0.1,\n n_estimators=10)\n\n# array of values for max_depth and min_child_weight parameters\nparam_test1 = {'max_depth':range(3,7,1),'min_child_weight':range(1,3,1)}\n\n# grid search with cross-validation using the updated model and parameter value array \ngsearch1 = GridSearchCV(estimator = xgb1, param_grid = param_test1, scoring='roc_auc',iid=False, cv=5)\ngsearch1.fit(X_train[features],y_train['diagnosis'])\ngsearch1.cv_results_['params'], gsearch1.best_params_, gsearch1.best_score_"
        }, 
        {
            "source": "The grid search found that the model works best with a max depth of 5, and a minimum child weight of 2. You can update our model accordingly and continue tuning.\n\n### 4.7 Tuning gamma<a class=\"anchor\" id=\"bullet-24\"></a>\nIncreasing gamma makes the algorithm more conservative (less prone to overfitting).", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# updating our current model with the max_depth and min_child weight parameter values found in last grid search\nxgb2 = XGBClassifier(\n objective='binary:logistic',\n learning_rate =0.1,\n n_estimators=10,\n max_depth=5,\n min_child_weight=2,\n gamma=0)\n\n# array of values for the gamma parameter\ngamma_test = {'gamma':[i/100.0 for i in range(0,6)]}\n\n# grid search with cross-validation using the updated model and gamma value array \ngsearch2 = GridSearchCV(estimator = xgb2, param_grid = gamma_test, scoring='roc_auc',iid=False, cv=5)\ngsearch2.fit(X_train[features],y_train['diagnosis'])\ngsearch2.cv_results_['params'], gsearch2.best_params_, gsearch2.best_score_"
        }, 
        {
            "source": "The grid search found that the model works best when gamma is 0.03.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 4.8 Evaluating updated model<a class=\"anchor\" id=\"bullet-25\"></a>\n\nRevisit your evaluation function to see if you need to update the number of boosting rounds. Use the parameter values found for max_depth, min_child_weight, and gamma, and set the number of estimators as 30 to see when it stops running.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "xgb_check = XGBClassifier(\n objective='binary:logistic',\n learning_rate =0.1,\n n_estimators=30,\n max_depth=5,\n min_child_weight=2,\n gamma=0.03)\n\nevaluate_model(xgb_check, X_train, y_train, features)"
        }, 
        {
            "source": "### 4.9 Tuning sampling parameters<a class=\"anchor\" id=\"bullet-26\"></a>\n\n**subsample**: The ratio of data that is randomly selected for growing trees<br/>\n**colsample_bytree**: The ratio of the subsample from which the algorithm selects splitting features\n\n**Note:** You have reduced your data set several times now; originally, you divided it into 80/20 train and test sets. You input only the training set to XGBoost, which are segmented twice more by subsample and colsample. Perhaps less than 50% of the original data will be used to train the model, reducing the chance of overfitting and improving the model's ability to generalize. It is one of the reasons why you see a better test score for XGBoost compared to the single decision tree model. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# updating our current model with the most recent n_estimators\nxgb3 = XGBClassifier(\n objective='binary:logistic',\n learning_rate =0.1,\n n_estimators=23,\n max_depth=5,\n min_child_weight=2,\n gamma=0.03)\n\n# array of values for subsample and colsample_bytree parameters\nsample_test = {\n 'subsample':[i/10.0 for i in range(5,9)],\n 'colsample_bytree':[i/10.0 for i in range(5,9)]\n}\n\n# grid search with cross validation for sampling parameters\ngsearch3 = GridSearchCV(estimator = xgb3, param_grid = sample_test, scoring='roc_auc',iid=False, cv=5)\ngsearch3.fit(X_train[features],y_train['diagnosis'])\ngsearch3.cv_results_['params'], gsearch3.best_params_, gsearch3.best_score_"
        }, 
        {
            "source": "The grid search found that the model works best with a colsample_bytree of 0.5, and a subsample of 0.7.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 4.10 Tuning lambda and alpha<a class=\"anchor\" id=\"bullet-27\"></a>\n\nLambda and alpha are both regularization parameters, which mathematically reduce the impact of features that might be too dominant in the model. The difference between the two is in how they apply these penalties.\n\n**lambda**: Applies L2 regularization, which shrinks the weights of all selected features equally. The default value is 0. <br/>\n**alpha**: Applies L1 regularization, which can shrink the weights down to 0 - essentially discarding features that have little impact on the model. The default is 1.\n\nThese parameters have more significant effects on large models (with more features). Tune them to see how it impacts your model.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# updating our current model with the sampling parameters found in the last grid search\nxgb4 = XGBClassifier(\n objective='binary:logistic',\n learning_rate =0.1,\n n_estimators=23,\n max_depth=5,\n min_child_weight=2,\n gamma=0.03, \n subsample=0.7,\n colsample_bytree=0.5)\n\n# array of values for subsample and colsample_bytree parameters\nreg_test = {'reg_alpha':[0, 0.01, 0.1], 'reg_lambda':[1, 1.1, 1.2, 1.3]}\n\n# grid search with cross validation for regularization parameters\ngsearch4 = GridSearchCV(estimator = xgb4, param_grid = reg_test, scoring='roc_auc',iid=False, cv=5)\ngsearch4.fit(X_train[features],y_train['diagnosis'])\ngsearch4.cv_results_['params'], gsearch4.best_params_, gsearch4.best_score_"
        }, 
        {
            "source": "The grid search found that the model works best when the reg_alpha regualization is 0.01 and the reg_lambda regulization is 1.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### 4.11 Evaluating final model<a class=\"anchor\" id=\"bullet-28\"></a>\nRun the current model through your evaluation function a final time. Compared to the model with default parameters, tuning has improved accuracy from 0.9824 to 0.9912, and the AUC score from 0.998772 to 0.999195. You can see why XGBoost is most popular for competition use, where those fourth decimal places really count!", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {}, 
            "outputs": [], 
            "source": "# updated model with alpha value found with last grid search. (lambda does not need to be set, since default is 1)\nxgb_final = XGBClassifier(\n objective='binary:logistic',\n learning_rate =0.1,\n n_estimators=30,\n max_depth=5,\n min_child_weight=2,\n gamma=0.03,\n reg_alpha=0.01)\n\nevaluate_model(xgb_final, X_train, y_train, features)"
        }, 
        {
            "source": "### 4.12 Predict on the test set<a class=\"anchor\" id=\"bullet-29\"></a>\nFinally, predict on the test data you reserved at the beginning. The XGBoost ensemble gives you approximately a 3% increase in accuracy and a 5% increase in AUC score over a single decision tree.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "execution_count": null, 
            "cell_type": "code", 
            "metadata": {
                "scrolled": true
            }, 
            "outputs": [], 
            "source": "test_features = [x for x in X_test.columns if x not in ['id']]\n\n#Predict test set:\ntest_predictions = xgb_final.predict(X_test[test_features])\ntest_predprob = xgb_final.predict_proba(X_test[test_features])[:,1]\n        \n#Print model report:\nprint \"Accuracy : %.4g\" % metrics.accuracy_score(y_test['diagnosis'].values, test_predictions)\nprint \"AUC Score (Test): %f\" % metrics.roc_auc_score(y_test['diagnosis'], test_predprob)"
        }, 
        {
            "source": "If you want to increase the accuracy and AUC score even further, you can restart the parameter tuning process with a smaller learning rate. The next value to try is 0.01. ", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "## Summary\nYou've found out how to use a decision tree as a machine learning model to classify cancer data, and you've seen how you how tools like XGBoost can be used to tune parameters to improve the accuracy of your model.\n\n### Learn more:\n* <p id=\"footnote-1\"><sup>[1]</sup><a id=\"fn1\" href=\"https://www.analyticsvidhya.com/blog/2016/03/complete-guide-parameter-tuning-xgboost-with-codes-python\" target=\"_blank\" rel=\"noopener noreferrer\">Jain, A. (March 1, 2016) Complete Guide to Parameter Tuning in XGBoost(with codes in Python)</a>\n* <a href=\"https://en.wikipedia.org/wiki/Feature_engineering\" target=\"_blank\" rel=\"noopener no referrer\">Feature engineering</a>\n* <a href=\"https://en.wikipedia.org/wiki/Decision_tree_learning#Metrics\" target=\"_blank\" rel=\"noopener noreferrer\">Decision tree learning metrics</a>\n* <a href=\"https://www.quora.com/Machine-Learning-What-is-an-intuitive-explanation-of-AUC\" rel=\"noopener noreferrer\">An intuitive explanation of AUC</a>\n* <a href=\"http://www.wikihow.com/Play-20-Questions\" target=\"_blank\" rel=\"noopener noreferrer\">How to play 20 questions</a>\n", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Author\nNatalie Ho studies statistics at the University of Waterloo.", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "### Data citations\n\n<p id=\"footnote-2\"> Lichman, M. (2013). [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml). Irvine, CA: University of California, School of Information and Computer Science. <a id=\"fn2\"></a>", 
            "cell_type": "markdown", 
            "metadata": {}
        }, 
        {
            "source": "Copyright \u00a9 IBM Corp. 2017. This notebook and its source code are released under the terms of the MIT License.", 
            "cell_type": "markdown", 
            "metadata": {}
        }
    ], 
    "metadata": {
        "kernelspec": {
            "display_name": "Python 2 with Spark 2.0 (Deprecated)", 
            "name": "python2-spark20", 
            "language": "python"
        }, 
        "language_info": {
            "mimetype": "text/x-python", 
            "nbconvert_exporter": "python", 
            "version": "2.7.11", 
            "name": "python", 
            "file_extension": ".py", 
            "pygments_lexer": "ipython2", 
            "codemirror_mode": {
                "version": 2, 
                "name": "ipython"
            }
        }
    }, 
    "nbformat": 4
}