Gini-Decision_Tree and Linear_Regression.ipynb

{
 "cells": [
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "09ad1c84",
   "metadata": {},
   "source": [
    "# Gini-Decision Tree and Linear Regression"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "4142726d",
   "metadata": {},
   "source": [
    "# 1) Gini-Decision Tree"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "6f37fac1",
   "metadata": {},
   "source": [
    "    A decision tree is a popular machine learning algorithm used for both classification and regression tasks. One of the most commonly used criteria for constructing decision trees is the Gini index, which measures the impurity of a set of data.\n",
    "\n",
    "    The Gini index is a statistical measure of inequality that is often used to evaluate the distribution of wealth in a society. In the context of decision trees, the Gini index is used to evaluate the impurity of a set of data based on the distribution of class labels. The Gini index ranges from 0 to 1, where 0 indicates a completely pure dataset (all examples belong to the same class), and 1 indicates a completely impure dataset (an equal number of examples from each class).\n",
    "\n",
    "    To construct a decision tree using the Gini index, the algorithm starts by calculating the Gini index for the entire dataset. It then splits the data into two subsets based on a chosen feature and evaluates the Gini index of each subset. The feature that results in the lowest Gini index (i.e., the most pure subsets) is selected as the splitting criterion. This process is repeated recursively for each subset until a stopping criterion is met, such as reaching a maximum tree depth or a minimum number of examples in a leaf node.\n",
    "\n",
    "    One of the advantages of using the Gini index for constructing decision trees is its simplicity and speed. The Gini index can be calculated quickly, even for datasets with a large number of features and examples. Additionally, decision trees constructed using the Gini index are often easy to interpret, making them useful for explaining the decision-making process to non-technical stakeholders.\n",
    "\n",
    "    However, a potential disadvantage of using the Gini index is that it may not always result in the most accurate decision tree. In some cases, other splitting criteria, such as information gain or gain ratio, may result in a more accurate model. Therefore, it is important to experiment with different splitting criteria and evaluate the performance of the resulting decision tree using appropriate evaluation metrics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "e62492cd",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn import datasets\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.tree import DecisionTreeClassifier\n",
    "from sklearn.metrics import accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "b29a11fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "((105, 4), (45, 4))"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "iris = datasets.load_iris()\n",
    "X = iris.data\n",
    "y = iris.target\n",
    "\n",
    "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.30, random_state = 42)\n",
    "X_train.shape, X_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "3b8fda7a",
   "metadata": {},
   "outputs": [],
   "source": [
    "clf_gini = DecisionTreeClassifier(criterion='gini', max_depth=3, random_state=0)\n",
    "clf_gini.fit(X_train, y_train)\n",
    "y_pred_gini = clf_gini.predict(X_test)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "bcc58552",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1.0\n"
     ]
    }
   ],
   "source": [
    "print(accuracy_score(y_test, y_pred_gini))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4d6ccecc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([1, 1, 2, 1, 2, 1, 2, 1, 0, 2, 1, 0, 0, 0, 1, 2, 0, 0, 0, 1, 0, 1,\n",
       "       2, 0, 1, 2, 0, 2, 2, 1, 1, 2, 1, 0, 1, 2, 0, 0, 1, 1, 0, 2, 0, 0,\n",
       "       2, 1, 2, 1, 1, 2, 1, 0, 0, 1, 2, 0, 0, 0, 1, 2, 0, 2, 2, 0, 1, 1,\n",
       "       2, 1, 2, 0, 2, 1, 2, 1, 1, 1, 0, 1, 1, 0, 1, 2, 2, 0, 1, 2, 1, 0,\n",
       "       2, 0, 1, 2, 2, 1, 2, 1, 1, 2, 2, 0, 1, 2, 0, 1, 2])"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_pred_train_gini = clf_gini.predict(X_train)\n",
    "y_pred_train_gini"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "7a11e952",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0.9523809523809523\n"
     ]
    }
   ],
   "source": [
    "print(accuracy_score(y_train, y_pred_train_gini))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f98bc074",
   "metadata": {},
   "source": [
    "# 2) Linear Regression"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "bba82065",
   "metadata": {},
   "source": [
    "    Linear regression is a commonly used statistical technique for modeling the relationship between a dependent variable and one or more independent variables. The goal of linear regression is to find the best linear relationship between the dependent variable and the independent variables, which can then be used to predict the values of the dependent variable for new observations.\n",
    "\n",
    "    In simple linear regression, there is only one independent variable, and the relationship between the independent variable and the dependent variable is modeled using a straight line. The equation of the line is given by:\n",
    "\n",
    "    y = β0 + β1x + ε\n",
    "\n",
    "    where y is the dependent variable, x is the independent variable, β0 and β1 are the intercept and slope of the line, respectively, and ε is the error term.\n",
    "\n",
    "    The intercept β0 represents the value of y when x is equal to zero, and the slope β1 represents the change in y for a one-unit change in x. The error term ε represents the random variation in the data that is not accounted for by the model.\n",
    "\n",
    "    In multiple linear regression, there are multiple independent variables, and the relationship between the independent variables and the dependent variable is modeled using a plane or hyperplane. The equation of the plane or hyperplane is given by:\n",
    "\n",
    "    y = β0 + β1x1 + β2x2 + ... + βpxp + ε\n",
    "\n",
    "    where p is the number of independent variables.\n",
    "\n",
    "    The coefficients β1, β2, ..., βp represent the change in y for a one-unit change in the corresponding independent variable, holding all other independent variables constant. The intercept β0 represents the value of y when all independent variables are equal to zero. The error term ε represents the random variation in the data that is not accounted for by the model.\n",
    "\n",
    "    Linear regression is often used in machine learning and data analysis applications, such as predictive modeling, forecasting, and trend analysis. The performance of a linear regression model can be evaluated using metrics such as the mean squared error (MSE) or the coefficient of determination (R-squared)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "2580c55a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "coefficient of determination: 0.7406426641094095 \n",
      "\n",
      "intercept: 36.459488385089884 \n",
      "\n",
      "slope: [-1.08011358e-01  4.64204584e-02  2.05586264e-02  2.68673382e+00\n",
      " -1.77666112e+01  3.80986521e+00  6.92224640e-04 -1.47556685e+00\n",
      "  3.06049479e-01 -1.23345939e-02 -9.52747232e-01  9.31168327e-03\n",
      " -5.24758378e-01] \n",
      "\n",
      "predicted response:\n",
      "[30.00384338 25.02556238 30.56759672 28.60703649 27.94352423 25.25628446\n",
      " 23.00180827 19.53598843 11.52363685 18.92026211 18.99949651 21.58679568\n",
      " 20.90652153 19.55290281 19.28348205 19.29748321 20.52750979 16.91140135\n",
      " 16.17801106 18.40613603 12.52385753 17.67103669 15.83288129 13.80628535\n",
      " 15.67833832 13.38668561 15.46397655 14.70847428 19.54737285 20.8764282\n",
      " 11.45511759 18.05923295  8.81105736 14.28275814 13.70675891 23.81463526\n",
      " 22.34193708 23.10891142 22.91502612 31.35762569 34.21510225 28.02056414\n",
      " 25.20386628 24.60979273 22.94149176 22.09669817 20.42320032 18.03655088\n",
      "  9.10655377 17.20607751 21.28152535 23.97222285 27.6558508  24.04901809\n",
      " 15.3618477  31.15264947 24.85686978 33.10919806 21.77537987 21.08493555\n",
      " 17.8725804  18.51110208 23.98742856 22.55408869 23.37308644 30.36148358\n",
      " 25.53056512 21.11338564 17.42153786 20.78483633 25.20148859 21.7426577\n",
      " 24.55744957 24.04295712 25.50499716 23.9669302  22.94545403 23.35699818\n",
      " 21.26198266 22.42817373 28.40576968 26.99486086 26.03576297 25.05873482\n",
      " 24.78456674 27.79049195 22.16853423 25.89276415 30.67461827 30.83110623\n",
      " 27.1190194  27.41266734 28.94122762 29.08105546 27.03977365 28.62459949\n",
      " 24.72744978 35.78159518 35.11454587 32.25102801 24.58022019 25.59413475\n",
      " 19.79013684 20.31167129 21.43482591 18.53994008 17.18755992 20.75049026\n",
      " 22.64829115 19.7720367  20.64965864 26.52586744 20.77323638 20.71548315\n",
      " 25.17208881 20.43025591 23.37724626 23.69043261 20.33578364 20.79180873\n",
      " 21.91632071 22.47107777 20.55738556 16.36661977 20.56099819 22.48178446\n",
      " 14.61706633 15.17876684 18.93868592 14.05573285 20.03527399 19.41013402\n",
      " 20.06191566 15.75807673 13.25645238 17.26277735 15.87841883 19.36163954\n",
      " 13.81483897 16.44881475 13.57141932  3.98885508 14.59495478 12.1488148\n",
      "  8.72822362 12.03585343 15.82082058  8.5149902   9.71844139 14.80451374\n",
      " 20.83858153 18.30101169 20.12282558 17.28601894 22.36600228 20.10375923\n",
      " 13.62125891 33.25982697 29.03017268 25.56752769 32.70827666 36.77467015\n",
      " 40.55765844 41.84728168 24.78867379 25.37889238 37.20347455 23.08748747\n",
      " 26.40273955 26.65382114 22.5551466  24.29082812 22.97657219 29.07194308\n",
      " 26.5219434  30.72209056 25.61669307 29.13740979 31.43571968 32.92231568\n",
      " 34.72440464 27.76552111 33.88787321 30.99238036 22.71820008 24.7664781\n",
      " 35.88497226 33.42476722 32.41199147 34.51509949 30.76109485 30.28934141\n",
      " 32.91918714 32.11260771 31.55871004 40.84555721 36.12770079 32.6692081\n",
      " 34.70469116 30.09345162 30.64393906 29.28719501 37.07148392 42.03193124\n",
      " 43.18949844 22.69034796 23.68284712 17.85447214 23.49428992 17.00587718\n",
      " 22.39251096 17.06042754 22.73892921 25.21942554 11.11916737 24.51049148\n",
      " 26.60334775 28.35518713 24.91525464 29.68652768 33.18419746 23.77456656\n",
      " 32.14051958 29.7458199  38.37102453 39.81461867 37.58605755 32.3995325\n",
      " 35.45665242 31.23411512 24.48449227 33.28837292 38.0481048  37.16328631\n",
      " 31.71383523 25.26705571 30.10010745 32.71987156 28.42717057 28.42940678\n",
      " 27.29375938 23.74262478 24.12007891 27.40208414 16.3285756  13.39891261\n",
      " 20.01638775 19.86184428 21.2883131  24.0798915  24.20633547 25.04215821\n",
      " 24.91964007 29.94563374 23.97228316 21.69580887 37.51109239 43.30239043\n",
      " 36.48361421 34.98988594 34.81211508 37.16631331 40.98928501 34.44634089\n",
      " 35.83397547 28.245743   31.22673593 40.8395575  39.31792393 25.70817905\n",
      " 22.30295533 27.20340972 28.51169472 35.47676598 36.10639164 33.79668274\n",
      " 35.61085858 34.83993382 30.35192656 35.30980701 38.79756966 34.33123186\n",
      " 40.33963075 44.67308339 31.59689086 27.3565923  20.10174154 27.04206674\n",
      " 27.2136458  26.91395839 33.43563311 34.40349633 31.8333982  25.81783237\n",
      " 24.42982348 28.45764337 27.36266999 19.53928758 29.11309844 31.91054611\n",
      " 30.77159449 28.94275871 28.88191022 32.79887232 33.20905456 30.76831792\n",
      " 35.56226857 32.70905124 28.64244237 23.58965827 18.54266897 26.87889843\n",
      " 23.28133979 25.54580246 25.48120057 20.53909901 17.61572573 18.37581686\n",
      " 24.29070277 21.32529039 24.88682244 24.86937282 22.86952447 19.45123791\n",
      " 25.11783401 24.66786913 23.68076177 19.34089616 21.17418105 24.25249073\n",
      " 21.59260894 19.98446605 23.33888    22.14060692 21.55509929 20.61872907\n",
      " 20.16097176 19.28490387 22.1667232  21.24965774 21.42939305 30.32788796\n",
      " 22.04734975 27.70647912 28.54794117 16.54501121 14.78359641 25.27380082\n",
      " 27.54205117 22.14837562 20.45944095 20.54605423 16.88063827 25.40253506\n",
      " 14.32486632 16.59488462 19.63704691 22.71806607 22.20218887 19.20548057\n",
      " 22.66616105 18.93192618 18.22846804 20.23150811 37.4944739  14.28190734\n",
      " 15.54286248 10.83162324 23.80072902 32.6440736  34.60684042 24.94331333\n",
      " 25.9998091   6.126325    0.77779806 25.30713064 17.74061065 20.23274414\n",
      " 15.83331301 16.83512587 14.36994825 18.47682833 13.4276828  13.06177512\n",
      "  3.27918116  8.06022171  6.12842196  5.6186481   6.4519857  14.20764735\n",
      " 17.21225183 17.29887265  9.89116643 20.22124193 17.94181175 20.30445783\n",
      " 19.29559075 16.33632779  6.55162319 10.89016778 11.88145871 17.81174507\n",
      " 18.26126587 12.97948781  7.37816361  8.21115861  8.06626193 19.98294786\n",
      " 13.70756369 19.85268454 15.22308298 16.96071981  1.71851807 11.80578387\n",
      " -4.28131071  9.58376737 13.36660811  6.89562363  6.14779852 14.60661794\n",
      " 19.6000267  18.12427476 18.52177132 13.1752861  14.62617624  9.92374976\n",
      " 16.34590647 14.07519426 14.25756243 13.04234787 18.15955693 18.69554354\n",
      " 21.527283   17.03141861 15.96090435 13.36141611 14.52079384  8.81976005\n",
      "  4.86751102 13.06591313 12.70609699 17.29558059 18.740485   18.05901029\n",
      " 11.51474683 11.97400359 17.68344618 18.12695239 17.5183465  17.22742507\n",
      " 16.52271631 19.41291095 18.58215236 22.48944791 15.28000133 15.82089335\n",
      " 12.68725581 12.8763379  17.18668531 18.51247609 19.04860533 20.17208927\n",
      " 19.7740732  22.42940768 20.31911854 17.88616253 14.37478523 16.94776851\n",
      " 16.98405762 18.58838397 20.16719441 22.97718032 22.45580726 25.57824627\n",
      " 16.39147632 16.1114628  20.534816   11.54272738 19.20496304 21.86276391\n",
      " 23.46878866 27.09887315 28.56994302 21.08398783 19.45516196 22.22225914\n",
      " 19.65591961 21.32536104 11.85583717  8.22386687  3.66399672 13.75908538\n",
      " 15.93118545 20.62662054 20.61249414 16.88541964 14.01320787 19.10854144\n",
      " 21.29805174 18.45498841 20.46870847 23.53334055 22.37571892 27.6274261\n",
      " 26.12796681 22.34421229]\n"
     ]
    }
   ],
   "source": [
    "import pandas as pd\n",
    "import numpy as np\n",
    "from sklearn.linear_model import LinearRegression\n",
    "\n",
    "data_url = \"http://lib.stat.cmu.edu/datasets/boston\"\n",
    "raw_df = pd.read_csv(data_url, sep=\"\\s+\", skiprows=22, header=None) \n",
    "data = np.hstack([raw_df.values[::2, :], raw_df.values[1::2, :2]]) \n",
    "target = raw_df.values[1::2, 2]\n",
    "\n",
    "model = LinearRegression()\n",
    "model.fit(data, target)\n",
    "model = LinearRegression().fit(data, target)\n",
    "\n",
    "r_sq = model.score(data, target)\n",
    "print()\n",
    "print(f\"coefficient of determination: {r_sq}\", \"\\n\")\n",
    "\n",
    "print(f\"intercept: {model.intercept_}\", \"\\n\")\n",
    "print(f\"slope: {model.coef_}\", \"\\n\")\n",
    "\n",
    "y_pred = model.predict(data)\n",
    "print(f\"predicted response:\\n{y_pred}\")\n"
   ]
  },
  {
   "attachments": {},
   "cell_type": "markdown",
   "id": "58d9e37a",
   "metadata": {},
   "source": [
    "    This code will print the coefficient of determination (R-squared) value, which represents the proportion of the variance in the dependent variable that is explained by the independent variables. The closer the R-squared value is to 1, the better the model fits the data.\n",
    "\n",
    "    You can customize the print statements to include additional information about the accuracy of the model, such as the mean squared error or the root mean squared error. These metrics provide information about the average difference between the predicted values and the actual values, and can be useful for evaluating the performance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "1071bc0f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}