Ridge_Reg-Random_Forest_EnergyEfficiencyDataset.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "28360f50",
   "metadata": {},
   "source": [
    "The link of dataset : http://archive.ics.uci.edu/ml/datasets/energy+efficiency.\n",
    "The dataset comprises 768samples and 8 features. The features are relative compactness, surface area, wall area, roof area,overall height, orientation, glazing area, glazing area distribution. \n",
    "The two output variables are heating load (HL) and cooling load (CL), of residential buildings.\n",
    "\n",
    "Ridge Regression and Random Forest are used to predict heading and cooling loads for the aferomentioned features of a building. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "208a8c8d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.metrics import mean_squared_error,mean_absolute_error\n",
    "from sklearn.model_selection import cross_validate, RepeatedKFold\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.ensemble import RandomForestRegressor"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dbf3a083",
   "metadata": {},
   "source": [
    "### Ridge Regression "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7916fa0",
   "metadata": {},
   "source": [
    "Ridge regression is a linear regression technique that is used to deal with multicollinearity in the data. Multicollinearity is a phenomenon that occurs when the predictor variables in a regression model are highly correlated with each other, which can lead to unstable and unreliable estimates of the regression coefficients.\n",
    "\n",
    "Ridge regression adds a penalty term to the least squares estimation of the regression coefficients. The penalty term is proportional to the sum of the squares of the coefficients, and a tuning parameter called the regularization parameter controls the amount of shrinkage applied to the coefficients.\n",
    "\n",
    "The effect of the penalty term is to constrain the size of the regression coefficients, which can help to reduce their variance and improve the stability of the estimates.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "0b981340",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>X1</th>\n",
       "      <th>X2</th>\n",
       "      <th>X3</th>\n",
       "      <th>X4</th>\n",
       "      <th>X5</th>\n",
       "      <th>X6</th>\n",
       "      <th>X7</th>\n",
       "      <th>X8</th>\n",
       "      <th>Y1</th>\n",
       "      <th>Y2</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0.98</td>\n",
       "      <td>514.5</td>\n",
       "      <td>294.0</td>\n",
       "      <td>110.25</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>15.55</td>\n",
       "      <td>21.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0.98</td>\n",
       "      <td>514.5</td>\n",
       "      <td>294.0</td>\n",
       "      <td>110.25</td>\n",
       "      <td>7.0</td>\n",
       "      <td>3</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>15.55</td>\n",
       "      <td>21.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0.98</td>\n",
       "      <td>514.5</td>\n",
       "      <td>294.0</td>\n",
       "      <td>110.25</td>\n",
       "      <td>7.0</td>\n",
       "      <td>4</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>15.55</td>\n",
       "      <td>21.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0.98</td>\n",
       "      <td>514.5</td>\n",
       "      <td>294.0</td>\n",
       "      <td>110.25</td>\n",
       "      <td>7.0</td>\n",
       "      <td>5</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>15.55</td>\n",
       "      <td>21.33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0.90</td>\n",
       "      <td>563.5</td>\n",
       "      <td>318.5</td>\n",
       "      <td>122.50</td>\n",
       "      <td>7.0</td>\n",
       "      <td>2</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0</td>\n",
       "      <td>20.84</td>\n",
       "      <td>28.28</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "     X1     X2     X3      X4   X5  X6   X7  X8     Y1     Y2\n",
       "0  0.98  514.5  294.0  110.25  7.0   2  0.0   0  15.55  21.33\n",
       "1  0.98  514.5  294.0  110.25  7.0   3  0.0   0  15.55  21.33\n",
       "2  0.98  514.5  294.0  110.25  7.0   4  0.0   0  15.55  21.33\n",
       "3  0.98  514.5  294.0  110.25  7.0   5  0.0   0  15.55  21.33\n",
       "4  0.90  563.5  318.5  122.50  7.0   2  0.0   0  20.84  28.28"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "\n",
    "# Load the data into a Pandas DataFrame\n",
    "data = pd.read_excel('ENB2012_data.xlsx')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "de8dd6ec",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.model_selection import train_test_split\n",
    "\n",
    "# Split the data into training and testing sets\n",
    "X_train, X_test, y1_train, y1_test, y2_train, y2_test = train_test_split(\n",
    "    data.iloc[:, :-2],  # Features\n",
    "    data.iloc[:, -2],   # Target variable 1 (heating load)\n",
    "    data.iloc[:, -1],   # Target variable 2 (cooling load)\n",
    "    test_size=0.2,      # 20% of data as test set\n",
    "    random_state=42)    # Fix random seed for reproducibility"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "9454cf6c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Alpha: 0.001, MSE for Y1: 9.1562, MAE for Y1: 2.1828\n",
      "Alpha: 0.01, MSE for Y1: 9.1862, MAE for Y1: 2.1901\n",
      "Alpha: 0.1, MSE for Y1: 9.4154, MAE for Y1: 2.2474\n",
      "Alpha: 1.0, MSE for Y1: 9.6535, MAE for Y1: 2.3136\n",
      "Alpha: 10.0, MSE for Y1: 10.8190, MAE for Y1: 2.4438\n",
      "\n",
      "Alpha: 0.001, MSE for Y2: 9.8932, MAE for Y2: 2.1948\n",
      "Alpha: 0.01, MSE for Y2: 9.9003, MAE for Y2: 2.1925\n",
      "Alpha: 0.1, MSE for Y2: 10.0780, MAE for Y2: 2.2242\n",
      "Alpha: 1.0, MSE for Y2: 10.3453, MAE for Y2: 2.2924\n",
      "Alpha: 10.0, MSE for Y2: 11.1643, MAE for Y2: 2.4535\n"
     ]
    }
   ],
   "source": [
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.metrics import mean_squared_error\n",
    "\n",
    "alphas = [0.001, 0.01, 0.1, 1.0, 10.0]  # Different alpha values to test\n",
    "\n",
    "# Train and test the Ridge Regression model for heating load (y1)\n",
    "for alpha in alphas:\n",
    "    model1 = Ridge(alpha=alpha)\n",
    "    model1.fit(X_train, y1_train)\n",
    "    y1_pred = model1.predict(X_test)\n",
    "    mse = mean_squared_error(y1_test, y1_pred)\n",
    "    mae = mean_absolute_error(y1_test, y1_pred)\n",
    "    print(f\"Alpha: {alpha}, MSE for Y1: {mse:.4f}, MAE for Y1: {mae:.4f}\")\n",
    "    \n",
    "print()    \n",
    "# Train and test the Ridge Regression model for cooling load (y2)\n",
    "for alpha in alphas:\n",
    "    model2 = Ridge(alpha=alpha)\n",
    "    model2.fit(X_train, y2_train)\n",
    "    y2_pred = model2.predict(X_test)\n",
    "    mse = mean_squared_error(y2_test, y2_pred)\n",
    "    mae = mean_absolute_error(y2_test, y2_pred)\n",
    "    print(f\"Alpha: {alpha}, MSE for Y2: {mse:.4f}, MAE for Y2: {mae:.4f}\")\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "e26c8946",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "from sklearn.linear_model import Ridge\n",
    "from sklearn.model_selection import cross_validate, RepeatedKFold\n",
    "\n",
    "\n",
    "# The cross_validate_ridge function uses repeated k-fold cross-validation with 10 folds and 10 repeats \n",
    "# to evaluate the performance of the ridge regression model\n",
    "\n",
    "def cross_validate_ridge(X, y, alpha):\n",
    "    kf = RepeatedKFold(n_splits=10, n_repeats=10, random_state=42)  # 10-fold and 10 repeats cross-validation\n",
    "    mse_scores = []\n",
    "    \n",
    "    for train_index, test_index in kf.split(X):\n",
    "        X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "        y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n",
    "        \n",
    "        model = Ridge(alpha=alpha)\n",
    "        model.fit(X_train, y_train)\n",
    "        y_pred = model.predict(X_test)\n",
    "        \n",
    "        mae_scores.append(mean_absolute_error(y_test, y_pred))\n",
    "        mse_scores.append(mean_squared_error(y_test, y_pred))\n",
    "    \n",
    "    mean_mae = np.mean(mae_scores)\n",
    "    std_mae = np.std(mae_scores)\n",
    "    mean_mse = np.mean(mse_scores)\n",
    "    std_mse = np.std(mse_scores)\n",
    "    \n",
    "    return mean_mae, std_mae, mean_mse, std_mse\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "408eeca1",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal alpha for Y1 (Heating Load): 0.001\n",
      "Optimal alpha for Y2 (Cooling Load): 0.001\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Ridge(alpha=0.001)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.model_selection import GridSearchCV\n",
    "from sklearn.linear_model import Ridge\n",
    "\n",
    "# Define the Ridge Regression model\n",
    "ridge = Ridge()\n",
    "\n",
    "# Define the range of alpha values to test\n",
    "alphas = [0.001, 0.01, 0.1, 1.0, 10.0]\n",
    "\n",
    "# Define the grid search parameters\n",
    "param_grid = {'alpha': alphas}\n",
    "\n",
    "# Perform grid search with 10-fold cross-validation\n",
    "grid_search = GridSearchCV(ridge, param_grid=param_grid, cv=10, scoring='neg_mean_absolute_error')\n",
    "grid_search.fit(X_train, y1_train)\n",
    "\n",
    "# Print the optimal alpha for Y1\n",
    "print(f\"Optimal alpha for Y1 (Heating Load): {grid_search.best_params_['alpha']}\")\n",
    "\n",
    "# Rebuild the model with the optimal alpha for Y1\n",
    "ridge_y1 = Ridge(alpha=grid_search.best_params_['alpha'])\n",
    "ridge_y1.fit(X_train, y1_train)\n",
    "\n",
    "# Perform grid search for Y2\n",
    "grid_search.fit(X_train, y2_train)\n",
    "\n",
    "# Print the optimal alpha for Y2\n",
    "print(f\"Optimal alpha for Y2 (Cooling Load): {grid_search.best_params_['alpha']}\")\n",
    "\n",
    "# Rebuild the model with the optimal alpha for Y2\n",
    "ridge_y2 = Ridge(alpha=grid_search.best_params_['alpha'])\n",
    "ridge_y2.fit(X_train, y2_train)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "dc43ceb9",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Y1 (Heating Load):\n",
      "MAE: 2.1119 +/- 0.2397\n",
      "MSE: 8.7708 +/- 1.7933\n",
      "\n",
      "Y2 (Cooling Load):\n",
      "MAE: 2.3284 +/- 0.2515\n",
      "MSE: 10.7225 +/- 2.4964\n"
     ]
    }
   ],
   "source": [
    "alpha = 00.1  # Optimal alpha \n",
    "\n",
    "# Perform 10-fold and 10 repeat cross-validation for Y1 (heating load)\n",
    "mean_mae1, std_mae1, mean_mse1, std_mse1 = cross_validate_ridge(X_train, y1_train, alpha)\n",
    "\n",
    "print(f\"Y1 (Heating Load):\")\n",
    "print(f\"MAE: {mean_mae1:.4f} +/- {std_mae1:.4f}\")\n",
    "print(f\"MSE: {mean_mse1:.4f} +/- {std_mse1:.4f}\\n\")\n",
    "\n",
    "# Perform 10-fold and 10 repeat cross-validation for Y2 (cooling load)\n",
    "\n",
    "mean_mae2, std_mae2, mean_mse2, std_mse2 = cross_validate_ridge(X_train, y2_train, alpha)\n",
    "\n",
    "print(f\"Y2 (Cooling Load):\")\n",
    "print(f\"MAE: {mean_mae2:.4f} +/- {std_mae2:.4f}\")\n",
    "print(f\"MSE: {mean_mse2:.4f} +/- {std_mse2:.4f}\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3cc90a93",
   "metadata": {},
   "source": [
    "### Another way\n",
    "\n",
    "\n",
    "##### Split into input (X) and output (y) variables\n",
    "X = data.iloc[:, :-2].values\n",
    "y = data.iloc[:, -2:].values\n",
    "##### Define the alpha parameters to test\n",
    "alphas = [0.001, 0.01, 0.1, 1.0, 10.0]\n",
    "\n",
    "##### Find the optimal alpha parameter using RidgeCV\n",
    "ridge_cv = RidgeCV(alphas=alphas, cv=10).fit(X_train, y_train)\n",
    "alpha_opt = ridge_cv.alpha_\n",
    "print(\"Optimal alpha parameter:\", alpha_opt)\n",
    "##### Train the Ridge model using the optimal alpha parameter\n",
    "ridge = Ridge(alpha=alpha_opt).fit(X_train, y_train)\n",
    "##### perform cross-validation with 10-fold 10-repetition strategy for Y1\n",
    "ridge_y1 = cross_validate(ridge, X, y1, cv=10, n_jobs=-1, scoring=('neg_mean_absolute_error', 'neg_mean_squared_error'), return_train_score=True)\n",
    "\n",
    "##### perform cross-validation with 10-fold 10-repetition strategy for Y2\n",
    "ridge_y2 = cross_validate(ridge, X, y2, cv=10, n_jobs=-1, scoring=('neg_mean_absolute_error', 'neg_mean_squared_error'), return_train_score=True)\n",
    "\n",
    "##### calculate the mean and standard deviation of the MAE and MSE scores for both Y1 and Y2\n",
    "mae_ridge_y1 = -ridge_y1['test_neg_mean_absolute_error']\n",
    "\n",
    "mse_ridge_y1 = -ridge_y1['test_neg_mean_squared_error']\n",
    "\n",
    "mae_ridge_y2 = -ridge_y2['test_neg_mean_absolute_error']\n",
    "\n",
    "mse_ridge_y2 = -ridge_y2['test_neg_mean_squared_error']\n",
    "\n",
    "print('RIDGE REGRESSION')\n",
    "\n",
    "print('Y1 MAE Mean:', mean(mae_ridge_y1), 'Std:', std(mae_ridge_y1))\n",
    "\n",
    "print('Y1 MSE Mean:', mean(mse_ridge_y1), 'Std:', std(mse_ridge_y1))\n",
    "\n",
    "print('Y2 MAE Mean:', mean(mae_ridge_y2), 'Std:', std(mae_ridge_y2))\n",
    "\n",
    "print('Y2 MSE Mean:', mean(mse_ridge_y2), 'Std:', std(mse_ridge_y2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "157f6f98",
   "metadata": {},
   "source": [
    "### Random Forest"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e1716b4f",
   "metadata": {},
   "source": [
    "Random forest is a popular machine learning algorithm that is used for both classification and regression tasks. It is an ensemble method that combines multiple decision trees to make predictions.\n",
    "\n",
    "In a random forest, a large number of decision trees are constructed using different subsets of the training data and a random subset of the predictor variables at each split. The trees are constructed independently of each other\n",
    "\n",
    "During prediction, the random forest algorithm aggregates the predictions from all the decision trees to make a final prediction.In regression tasks, the final prediction is the mean or median of the predictions from the individual trees.\n",
    "\n",
    "Random forests have several advantages over individual decision trees, including improved accuracy, reduced overfitting, and better generalization to new data. The use of multiple trees in a random forest enable us smooth the noise and variability in the data, which can improve the robustness and accuracy of the predictions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "f10b7d66",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Optimal parameters for Y1 (Heating Load):\n",
      "{'max_depth': 50, 'min_samples_leaf': 1, 'min_samples_split': 2, 'n_estimators': 100}\n",
      "Optimal parameters for Y2 (Cooling Load):\n",
      "{'max_depth': 50, 'min_samples_leaf': 1, 'min_samples_split': 2, 'n_estimators': 250}\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "RandomForestRegressor(max_depth=50, n_estimators=250, random_state=42)"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "from sklearn.ensemble import RandomForestRegressor\n",
    "from sklearn.model_selection import GridSearchCV\n",
    "\n",
    "# Define the RandomForestRegressor model\n",
    "rf = RandomForestRegressor(random_state=42)\n",
    "\n",
    "# Define the grid search parameters\n",
    "param_grid = {'n_estimators': [10, 50, 100, 250, 500],\n",
    "              'max_depth': [50, 150, 250],\n",
    "              'min_samples_split': [2, 3],\n",
    "              'min_samples_leaf': [1, 2, 3]}\n",
    "\n",
    "# Perform grid search with 10-fold cross-validation\n",
    "grid_search = GridSearchCV(rf, param_grid=param_grid, cv=10, scoring='neg_mean_absolute_error', n_jobs=-1)\n",
    "grid_search.fit(X_train, y1_train)\n",
    "\n",
    "# Print the optimal parameters for Y1\n",
    "print(\"Optimal parameters for Y1 (Heating Load):\")\n",
    "print(grid_search.best_params_)\n",
    "\n",
    "# Rebuild the model with the optimal parameters for Y1\n",
    "rf_y1 = RandomForestRegressor(random_state=42, **grid_search.best_params_)\n",
    "rf_y1.fit(X_train, y1_train)\n",
    "\n",
    "# Perform grid search for Y2\n",
    "grid_search.fit(X_train, y2_train)\n",
    "\n",
    "# Print the optimal parameters for Y2\n",
    "print(\"Optimal parameters for Y2 (Cooling Load):\")\n",
    "print(grid_search.best_params_)\n",
    "\n",
    "# Rebuild the model with the optimal parameters for Y2\n",
    "rf_y2 = RandomForestRegressor(random_state=42, **grid_search.best_params_)\n",
    "rf_y2.fit(X_train, y2_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c96b01f5",
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "# The cross_validate_ridge function uses repeated k-fold cross-validation with 10 folds and 10 repeats \n",
    "# to evaluate the performance of the random forest model\n",
    "\n",
    "def cross_validate_RandomForest(X, y, max_depth,min_samples_leaf,min_samples_split,n_estimators):\n",
    "    kf = RepeatedKFold(n_splits=10, n_repeats=10, random_state=42)   # 10-fold cross-validation with random shuffle\n",
    "    mae_scores = []\n",
    "    mse_scores = []\n",
    "    \n",
    "    for train_index, test_index in kf.split(X):\n",
    "        X_train, X_test = X.iloc[train_index], X.iloc[test_index]\n",
    "        y_train, y_test = y.iloc[train_index], y.iloc[test_index]\n",
    "        \n",
    "        model = RandomForestRegressor(random_state=42, max_depth=max_depth, min_samples_leaf=min_samples_leaf, min_samples_split=min_samples_split, n_estimators=n_estimators)\n",
    "        model.fit(X_train, y_train)\n",
    "        y_pred = model.predict(X_test)\n",
    "        \n",
    "        mae_scores.append(mean_absolute_error(y_test, y_pred))\n",
    "        mse_scores.append(mean_squared_error(y_test, y_pred))\n",
    "    \n",
    "    mean_mae = np.mean(mae_scores)\n",
    "    std_mae = np.std(mae_scores)\n",
    "    mean_mse = np.mean(mse_scores)\n",
    "    std_mse = np.std(mse_scores)\n",
    "    \n",
    "    return mean_mae, std_mae, mean_mse, std_mse"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "4d4b3515",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Y1 (Heating Load):\n",
      "MAE: 0.3216 +/- 0.0472\n",
      "MSE: 0.2380 +/- 0.0874\n",
      "\n",
      "Y2 (Cooling Load):\n",
      "MAE: 0.9887 +/- 0.1531\n",
      "MSE: 2.7092 +/- 0.7935\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# Perform 10-fold cross-validation for Y1 (heating load)\n",
    "mean_mae1, std_mae1, mean_mse1, std_mse1 = cross_validate_RandomForest(X_train, y1_train, 50,1,2,250)\n",
    "\n",
    "print(f\"Y1 (Heating Load):\")\n",
    "print(f\"MAE: {mean_mae1:.4f} +/- {std_mae1:.4f}\")\n",
    "print(f\"MSE: {mean_mse1:.4f} +/- {std_mse1:.4f}\\n\")\n",
    "\n",
    "# Perform 10-fold cross-validation for Y2 (cooling load)\n",
    "\n",
    "mean_mae2, std_mae2, mean_mse2, std_mse2 = cross_validate_RandomForest(X_train, y2_train, 50,1,2,250)\n",
    "\n",
    "print(f\"Y2 (Cooling Load):\")\n",
    "print(f\"MAE: {mean_mae2:.4f} +/- {std_mae2:.4f}\")\n",
    "print(f\"MSE: {mean_mse2:.4f} +/- {std_mse2:.4f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "35a3763c",
   "metadata": {},
   "source": [
    "### Another way\n",
    "##### define the RandomForestRegressor model with the optimal hyperparameters\n",
    "rf_model = RandomForestRegressor(n_estimators=100, max_depth=50, min_samples_split=2, min_samples_leaf=3, random_state=42)\n",
    "\n",
    "##### perform cross-validation with 10-fold 10-repetition strategy for Y1\n",
    "rf_y1 = cross_validate(rf_model, X, y1, cv=10, n_jobs=-1, scoring=('neg_mean_absolute_error', 'neg_mean_squared_error'), return_train_score=True)\n",
    "\n",
    "##### perform cross-validation with 10-fold 10-repetition strategy for Y2\n",
    "rf_y2 = cross_validate(rf_model, X, y2, cv=10, n_jobs=-1, scoring=('neg_mean_absolute_error', 'neg_mean_squared_error'), return_train_score=True)\n",
    "\n",
    "##### calculate the mean and standard deviation of the MAE and MSE scores for both Y1 and Y2\n",
    "mae_rf_y1 = -rf_y1['test_neg_mean_absolute_error']\n",
    "\n",
    "mse_rf_y1 = -rf_y1['test_neg_mean_squared_error']\n",
    "\n",
    "mae_rf_y2 = -rf_y2['test_neg_mean_absolute_error']\n",
    "\n",
    "mse_rf_y2 = -rf_y2['test_neg_mean_squared_error']\n",
    "\n",
    "print('RANDOM FOREST')\n",
    "\n",
    "print('Y1 MAE Mean:', mean(mae_rf_y1), 'Std:', std(mae_rf_y1))\n",
    "\n",
    "print('Y1 MSE Mean:', mean(mse_rf_y1), 'Std:', std(mse_rf_y1))\n",
    "\n",
    "print('Y2 MAE Mean:', mean(mae_rf_y2), 'Std:', std(mae_rf_y2))\n",
    "\n",
    "print('Y2 MSE Mean:', mean(mse_rf_y2), 'Std:', std(mse_rf_y2))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4190977d",
   "metadata": {},
   "source": [
    "### COMPARISON\n",
    "\n",
    "For Y1, the Random Forest model has an MAE of 0.3216 +/- 0.0472 and an MSE of 0.2380 +/- 0.0874, compared to the Ridge Regression model's MAE of 2.1119 +/- 0.2397 and MSE of 8.7708 +/- 1.7933.\n",
    "\n",
    "For Y2, the Random Forest model has an MAE of 0.9887 +/- 0.1531 and an MSE of 2.7092 +/- 0.7935, compared to the Ridge Regression model's MAE of 2.3284 +/- 0.2515 and MSE of 10.7225 +/- 2.4964.\n",
    "\n",
    "To sum up, the Random Forest model appears to be a much better model for predicting both Y1 and Y2, as it has significantly lower MAE and MSE values than the Ridge Regression model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "cdd1e1bd",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}