BIDS_09_ClassificationRegression_3.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "55cac4d8",
   "metadata": {},
   "source": [
    "Aim to complete as much of this tutorial on your own *before* coming to the practical session.\n",
    "\n",
    "Use the practical session to get help for any aspect you do not understand or were unable to complete."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "85b9ad73-d175-426e-b127-4143dbec9d2e",
   "metadata": {},
   "source": [
    "# Classification and Regression 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e973373-614e-488f-9d55-d074847c548d",
   "metadata": {},
   "source": [
    "Learning objectives\n",
    "1. Apply RFs and GBDTs in a multi-group classification setting using the popular python libraries [sklearn](https://scikit-learn.org/stable/) and [xgboost](https://xgboost.readthedocs.io/en/stable/python/python_intro.html)\n",
    "2. Explore differences in model performance based on changing model parameters\n",
    "3. Visualise the important variables \n",
    "4. Apply RFs and GBDTs in a regression setting and explore the results"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "434f88d8",
   "metadata": {},
   "source": [
    "## Ensure you have installed xgboost _before_ starting the notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "20b6f9c4",
   "metadata": {},
   "source": [
    "xgboost is implemented in the `xgboost` package.\n",
    "\n",
    "Install it via ```conda install -c conda-forge xgboost``` or ```pip install xgboost``` in the terminal.\n",
    "\n",
    "Also ensure you have installed graphviz and python-graphviz:\n",
    "\n",
    "```conda install graphviz``` <br>\n",
    "```conda install python-graphviz```"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e39532a6",
   "metadata": {},
   "source": [
    "## Import specific packages and functions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2dfdb807-bc8d-469d-a338-e541449ee443",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.ensemble import RandomForestClassifier, RandomForestRegressor, GradientBoostingClassifier\n",
    "from sklearn.model_selection import GridSearchCV, train_test_split\n",
    "from sklearn.feature_selection import RFECV\n",
    "from sklearn.preprocessing import StandardScaler\n",
    "from sklearn.inspection import permutation_importance\n",
    "from sklearn.manifold import MDS \n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.metrics import roc_curve, confusion_matrix, roc_auc_score, f1_score, accuracy_score, balanced_accuracy_score, classification_report\n",
    "import xgboost as xgb\n",
    "import sklearn"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5447a4e1-c88a-4d86-9d88-fbdf00a1b988",
   "metadata": {},
   "source": [
    "## Load in dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d687795-16cb-4bab-a682-16d45da57269",
   "metadata": {},
   "source": [
    "We will follow the steps from the Classification and Regression 2 (BIDS 8) tutorial for loading, scaling, and splitting the data in preparation for using these as input to the models. Here we use the cancer microbiome dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "53438422-b94b-4757-8045-914b458e9e46",
   "metadata": {},
   "outputs": [],
   "source": [
    "cancer_microbiome_genus = pd.read_excel('../Data-main/cancer_microbiome_genus.xlsx')\n",
    "cancer_microbiome_genus.cancer.value_counts()\n",
    "cancer_microbiome_genus.adenomas_or_cancer.value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e61f4dc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# enter your CID here, or date of birth, or another number of your choosing to use as random state\n",
    "CID = 0\n",
    "\n",
    "# remember to check the documentation of each algorithm if setting the random_state is needed\n",
    "# for this tutorial and all upcoming tutorials..."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cc6e86a",
   "metadata": {},
   "source": [
    "We will use [sklearn](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) functions [train_test_split()](https://scikit-learn.org/stable/modules/generated/sklearn.model_selection.train_test_split.html) and [StandardScaler()](https://scikit-learn.org/stable/modules/generated/sklearn.preprocessing.StandardScaler.html) as before to split and scale the data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6b1da441-aba1-4954-8401-91cddc6afd67",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the data into train and test sets\n",
    "X_train_unscaled, X_test_unscaled, y_train, y_test = train_test_split(cancer_microbiome_genus.iloc[:, 4:], cancer_microbiome_genus.adenomas_or_cancer, test_size=0.3, random_state=CID)\n",
    "\n",
    "# Scale the data with standard scaling (0 mean and unit variance)\n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train_unscaled)\n",
    "X_test = scaler.transform(X_test_unscaled)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6cf74628-e98f-41ee-a087-c993a830ce1e",
   "metadata": {},
   "source": [
    "## Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffba4b76-e8b3-4ef4-a9dc-5fed754a1398",
   "metadata": {},
   "source": [
    "A random forest is a meta estimator that fits a number of decision tree classifiers on various sub-samples of the dataset and uses averaging to improve the predictive accuracy and control over-fitting. The sub-sample size is controlled with the `max_samples` parameter if `bootstrap=True` (default), otherwise the whole dataset is used to build each tree. Some other parameters, specific to Random Forests, that you can set to optimise the [RandomForestClassifier()](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestClassifier.html) are:\n",
    "- `n_estimators`: the number of trees in the forest\n",
    "- `criterion`: the function to measure the quality of the split, can be Gini inpurity, log_loss and entropy for the Shannon information gain, look at the specific formulas [here](https://scikit-learn.org/stable/modules/tree.html#tree-mathematical-formulation)\n",
    "- `max_depth`: the maximum depth of the tree, ie how many nodes to be expanded until all leaves are pure or contain less that `min_samples_split`\n",
    "- `min_samples_split`: the minimum number of samples required to split an internal node\n",
    "- `min_samples_leaf`: the minimum number of samples required to be a leaf (terminal) node\n",
    "- `oob_score`: whether to use out-of-bag samples to estimate the generalization score\n",
    "\n",
    "Make sure to look at the [RandomForestClassifier()](https://scikit-learn.org/stable/modules/generate/sklearn.ensemble.RandomForestClassifier.html) object page for the rest of the parameters available. You can read more about Random Forests in the user guide [documentation](https://scikit-learn.org/stable/modules/ensemble.html#forest).\n",
    "\n",
    "Here, we will use the [classification_report()](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.classification_report.html) function to generate a summary of different metrics we used before in BIDS 8."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ca727e6d-f15e-4abf-b56e-87e142116921",
   "metadata": {},
   "outputs": [],
   "source": [
    "# define the classifier with default parameters\n",
    "\n",
    "rf_default = RandomForestClassifier()\n",
    "y_pred_default = rf_default.fit(X_train, y_train).predict(X_test)\n",
    "print(classification_report(y_test, y_pred_default))\n",
    "print('accuracy:', accuracy_score(y_test, y_pred_default))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8768b17-1c1a-4a66-9912-6de1cdbce5e9",
   "metadata": {},
   "outputs": [],
   "source": [
    "cm_default = confusion_matrix(y_test, y_pred_default)\n",
    "\n",
    "def conf_matrix_heatmap(cm):\n",
    "    ax = plt.subplot()\n",
    "    sns.heatmap(cm, annot=True, ax=ax, cmap='Greens'); #annot=True to annotate cells\n",
    "    ax.set_xlabel('Predicted status')\n",
    "    ax.set_ylabel('True status')\n",
    "    ax.set_title('Confusion Matrix')\n",
    "\n",
    "conf_matrix_heatmap(cm_default)\n",
    "display(y_test.value_counts()) # to see the class breakdown in the test-set "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d654355-347f-4c2f-8a22-143d8962d644",
   "metadata": {},
   "source": [
    "Given what you see at the [confusion matrix](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html) results on true and predicted status for each of the 4 classes, can you explain the zero division warning error we get when trying to calculate the [precision](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.precision_score.html) and [F1 score](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) above?\n",
    "\n",
    "Now let's change the model parameters and see if we get a higher accuracy score..."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c072cdf0-7077-473f-8f11-44284c5525f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf = RandomForestClassifier(\n",
    "    n_estimators=1000, \n",
    "    criterion='entropy', \n",
    "    oob_score=True, \n",
    "    class_weight='balanced'\n",
    ")\n",
    "y_pred = rf.fit(X_train, y_train).predict(X_test)\n",
    "print(classification_report(y_test, y_pred))\n",
    "print('accuracy:', accuracy_score(y_test, y_pred)) # if you want to look at extra decimals\n",
    "print('oob score:', rf.oob_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a5e23a0-fc09-46fe-b9eb-d8d3306320b5",
   "metadata": {},
   "source": [
    "Some questions to consider: \n",
    "- Why did we use a balanced class weight?\n",
    "- What does the oob (out-of-bag) score tell us about the training set?\n",
    "- Are more estimators (number of trees) always a good idea or is there a threshold?\n",
    "\n",
    "Have a go at tweaking the different parameters yourself!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "646aedcf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add your code here to change some parameters and comparing the output with the results above\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "58671f01-197a-4d2c-b69b-b64b7cd0ed9a",
   "metadata": {},
   "source": [
    "### Feature Importance"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b5e9087-2084-46d0-9c87-34f72fdb19ab",
   "metadata": {},
   "source": [
    "Feature importances are provided by the fitted attribute `feature_importances_` and they are computed as the mean and standard deviation of accumulation of the impurity decrease within each tree. Impurity-based feature importances can be misleading for high cardinality features (many unique values). For this reason we can use [permutation importance()](https://scikit-learn.org/stable/modules/permutation_importance.html).\n",
    "\n",
    "We will use the same cancer dataset but change our Y target to the cancer column (0 - no cancer, 1 - cancer)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ba4d738a-6e75-445a-9192-26677175f27d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Split the data into train and test sets\n",
    "X_train_unscaled, X_test_unscaled, y_train, y_test = train_test_split(cancer_microbiome_genus.iloc[:, 4:], cancer_microbiome_genus.cancer, test_size=0.3, random_state=8)\n",
    "\n",
    "# scale \n",
    "scaler = StandardScaler()\n",
    "X_train = scaler.fit_transform(X_train_unscaled)\n",
    "X_test = scaler.transform(X_test_unscaled)\n",
    "\n",
    "# double check that the y is correct (since we are using same name variables)\n",
    "display(y_train.value_counts())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f7231cf8-67a7-4349-9f9d-22fc40c18daf",
   "metadata": {},
   "outputs": [],
   "source": [
    "# fit a new model \n",
    "rf = RandomForestClassifier(\n",
    "    n_estimators=100, \n",
    "    criterion='gini', \n",
    "    oob_score=True\n",
    ")\n",
    "y_pred = rf.fit(X_train, y_train).predict(X_test)\n",
    "print(classification_report(y_test, y_pred))\n",
    "print('oob score:', rf.oob_score_)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8a753b52-e591-49a4-98ab-64fc442eeb77",
   "metadata": {},
   "source": [
    "Below we will create a function that looks at the samples on the terminal nodes (leafs) and creates a proximity matrix. This matrix holds information on which samples have ended up together in the same leaf, for example when a pair of samples reach the same leaf in a tree their proximity value increases by one. We then normalise by dividing by the number of trees. We can then use multidimensional scaling, [MDS()](https://scikit-learn.org/stable/modules/generated/sklearn.manifold.MDS.html) to visualise the sample separation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a4623b1-eebf-4c2a-a13e-b904517b0eaa",
   "metadata": {},
   "outputs": [],
   "source": [
    "def proximity_matrix(rf, X, normalise=True):      \n",
    "    leafs = rf.apply(X)\n",
    "    n_trees = leafs.shape[1]\n",
    "    val = leafs[:,0]\n",
    "    matrix = 1*np.equal.outer(val, val)\n",
    "    for i in range(1, n_trees):\n",
    "        val = leafs[:, i]\n",
    "        matrix += 1*np.equal.outer(val, val)\n",
    "    if normalise:\n",
    "        matrix = matrix / n_trees\n",
    "    return matrix"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5868c400-acad-40ed-a55f-9a06c622ceed",
   "metadata": {},
   "outputs": [],
   "source": [
    "pm = proximity_matrix(rf, X_train, normalise=True)\n",
    "\n",
    "df = pd.DataFrame(pm)\n",
    "mds = MDS(n_components=2)\n",
    "df_mds = MDS().fit_transform(X=df)\n",
    "df_mds = pd.DataFrame(df_mds)\n",
    "df_mds['y'] = y_train.tolist()\n",
    "\n",
    "sns.scatterplot(df_mds[0], df_mds[1], data=df_mds, hue='y')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "aaee940b-9cd2-4531-8c48-60c43abfc9fe",
   "metadata": {},
   "outputs": [],
   "source": [
    "# look at feature importances \n",
    "feat_impo = rf.feature_importances_\n",
    "features = cancer_microbiome_genus.columns[4:]\n",
    "\n",
    "feature_dict = dict(zip(features, feat_impo))\n",
    "bardf = pd.DataFrame(\n",
    "        {'features': feature_dict.keys(),\n",
    "         'values': feature_dict.values(),\n",
    "        })\n",
    "bardf = bardf.sort_values('values') # order values \n",
    "bardf = bardf[bardf['values'] != 0] # drop 0 value importance features\n",
    "\n",
    "plt.figure(figsize=(10,15))\n",
    "\n",
    "sns.barplot(bardf['values'], bardf['features'], data=bardf, palette = 'flare', order=bardf['features'], orient='h')\n",
    "\n",
    "plt.tight_layout()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "7775faa2-0847-4af3-be22-4f617261c905",
   "metadata": {},
   "outputs": [],
   "source": [
    "# we will use the same classifier for the permutation importance \n",
    "rf = RandomForestClassifier(\n",
    "    n_estimators=100, \n",
    "    criterion='gini', \n",
    "    oob_score=True\n",
    ").fit(X_train, y_train)\n",
    "%time feat_impo_perm = permutation_importance(rf, X_train, y_train, n_repeats=10, random_state=0, scoring='accuracy')\n",
    "\n",
    "# visualise the new importance values similarly to how we did it above\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "63a8c122-9a98-4322-bd62-de2c051a8100",
   "metadata": {},
   "source": [
    "## Gradient Boosted Decision Trees"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "16356514-3568-4f41-b3b5-e4003a5f2e3c",
   "metadata": {},
   "source": [
    "GB builds an additive model in a forward stage-wise fashion; it allows for the optimization of arbitrary differentiable loss functions. In each stage `n_classes_` regression trees are fit on the negative gradient of the loss function, e.g. binary or multiclass log loss. Binary classification is a special case where only a single regression tree is induced. To explore GBDTs we will use the [GradientBoostingClassifier()](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) object and the sklearn-supported [XGBClassifier()](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.XGBClassifier) object from the [xgboost](https://xgboost.readthedocs.io/en/stable/python/python_intro.html) package. To learn more about boosted trees take a look at the [XGBoost documentation](https://xgboost.readthedocs.io/en/stable/tutorials/model.html).\n",
    "\n",
    "For the `xgboost.XGBClassifier()` object there are a number of parameters to similar to the Random Forest parameters we encountered before (n_estimators, max_depth, etc). You can set one of the various boosting algorithms to use with the parameter `tree_method` and you can learn more about the different options [here](https://xgboost.readthedocs.io/en/stable/treemethod.html)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "64c4ce22-5c1d-479c-8db0-c2af8c5012fc",
   "metadata": {},
   "outputs": [],
   "source": [
    "# lets fit the model\n",
    "clf = xgb.XGBClassifier(\n",
    "    n_estimators=1000,\n",
    "    tree_method=\"hist\", \n",
    "    enable_categorical=False, \n",
    "    use_label_encoder=False, \n",
    "    predictor='cpu_predictor'\n",
    ")\n",
    "clf.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e8ce8b94-8bd7-4545-b1d7-402c9c06d3a1",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# here you can visualise a tree and plot it (this can also be done for the sklearn functions too)\n",
    "# Get a graph\n",
    "graph = xgb.to_graphviz(clf, num_trees=1)\n",
    "\n",
    "# Or get a matplotlib axis\n",
    "ax = xgb.plot_tree(clf, num_trees=1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2e3743f2-6617-4f3f-9f4b-766198040ec7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# now let's predict a model with out testing set, and also look at feature importances \n",
    "y_pred = clf.predict(X_test)\n",
    "display(accuracy_score(y_test, y_pred))\n",
    "\n",
    "# and let's explore some of the attributes, feel free to check more!\n",
    "clf.feature_importances_\n",
    "\n",
    "clf.get_booster"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "31de3d84-4fb0-4e3c-8527-3e651ac1d018",
   "metadata": {},
   "source": [
    "Now use the [sklearn.ensemble.GradientBoostingClassifier()](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) function to look at different regularisation parameters and how they affect the model. Example code is given below to create graphs of the loss function deviance in relation to the number of boosting iterations. Make sure to read the different parameters required for the object, and change the combinations given below to try and find what maximises accuracy. Maybe try a different dataset as well...?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "347d73ea-f0b0-4afd-87e0-6b612379e396",
   "metadata": {},
   "outputs": [],
   "source": [
    "constant_params = {\n",
    "    \"n_estimators\": 500,\n",
    "    \"max_leaf_nodes\": None,\n",
    "    \"max_depth\": None,\n",
    "    \"random_state\": 2,\n",
    "    \"min_samples_split\": 5,\n",
    "} \n",
    "\n",
    "plt.figure()\n",
    "for label, color, param_setting in [\n",
    "    (\"No shrinkage\", \"orange\", {\"learning_rate\": 1.0, \"subsample\": 1.0}),\n",
    "    (\"learning_rate=0.2\", \"turquoise\", {\"learning_rate\": 0.2, \"subsample\": 1.0}),\n",
    "    (\"subsample=0.5\", \"blue\", {\"learning_rate\": 1.0, \"subsample\": 0.5}),\n",
    "    (\"learning_rate=0.2, subsample=0.5\",\"gray\",{\"learning_rate\": 0.2, \"subsample\": 0.5}),\n",
    "    (\"learning_rate=0.2, max_features=2\", \"magenta\",{\"learning_rate\": 0.2, \"max_features\": 2})\n",
    "]:\n",
    "    params = dict(constant_params)\n",
    "    params.update(param_setting)\n",
    "\n",
    "    clf = GradientBoostingClassifier(**params)\n",
    "    clf.fit(X_train, y_train)\n",
    "\n",
    "    # compute test set deviance\n",
    "    test_deviance = np.zeros((params[\"n_estimators\"],), dtype=np.float64)\n",
    "\n",
    "    for i, y_pred in enumerate(clf.staged_decision_function(X_test)):\n",
    "        # clf.loss_ assumes that y_test[i] in {0, 1}\n",
    "        test_deviance[i] = clf.loss_(y_test, y_pred)\n",
    "\n",
    "    sns.lineplot(\n",
    "        x=(np.arange(test_deviance.shape[0]) + 1)[::5],\n",
    "        y=test_deviance[::5],\n",
    "#         \"-\",\n",
    "        color=color,\n",
    "        label=label,\n",
    "    )\n",
    "\n",
    "plt.legend(loc=\"upper left\")\n",
    "plt.xlabel(\"Boosting Iterations\")\n",
    "plt.ylabel(\"Test Set Deviance\")\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff149b22",
   "metadata": {},
   "source": [
    "How do the two gradient boosting algorithms ([GradientBoostingClassifier()](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingClassifier.html) and [XGBClassifier()](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.XGBClassifier)) compare in terms of accuracy?\n",
    "\n",
    "Do they find the same important features?\n",
    "\n",
    "How does this change for other datasets?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cfbdbc5e",
   "metadata": {},
   "source": [
    "## Optimise hyperparameters for RF and GBDTs using a 'searchCV'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2cc6f7b0",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Reuse code from a prior BIDS tutorial to find the optimal set of hyperparameters\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "06794f6b-4785-416e-aad7-f8a98ae661ac",
   "metadata": {},
   "source": [
    "# Your turn... Random Forest Regression!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c17ae676-75ca-4869-ae4f-6f3246f91b80",
   "metadata": {},
   "source": [
    "Now that you have learned about random forests and decision trees in a classification setting, try to repeat some or all of the steps for regression models ([RandomForestRegressor()](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html), [GradientBoostingRegressor()](https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.GradientBoostingRegressor.html) and [XGBRegressor()](https://xgboost.readthedocs.io/en/stable/python/python_api.html#xgboost.XGBRegressor)).\n",
    "\n",
    "Make sure to select the right metrics to evaluate (see BIDS 7).\n",
    "\n",
    "Compare your results to those obtained with the regression methods from BIDS 7 ([Ridge()](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Ridge.html), [Lasso()](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.Lasso.html), [ElasticNet()](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.ElasticNet.html) and [PLSRegression()](https://scikit-learn.org/stable/modules/generated/sklearn.cross_decomposition.PLSRegression.html)) and BIDS 8 ([SVR()](https://scikit-learn.org/stable/modules/generated/sklearn.svm.SVR.html)).\n",
    "\n",
    "You can call on the help of code-LLMs if you are stuck, but considering you have done this a few times now, can you do it without help...?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "232d9fe6-9cc7-431e-883a-4660db8cf5d2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Add your regression code here, which algorithm(s) performs best?\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}