mrozov_project_ENG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "95a4387e-610f-438c-9edc-322d768f23df",
   "metadata": {},
   "source": [
    "# Course project on topic \"Megafon videocourse\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "75f308c8-9f80-4fa9-9d63-f60b01556f7f",
   "metadata": {},
   "source": [
    "### Task description"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d7974cb0-6b27-4b31-8283-7f006f7b58be",
   "metadata": {},
   "source": [
    "In this project we have binary classification task.  \n",
    "We need to predict whether a customer would accept a service, if we  \n",
    "propose it to them.  \n",
    "By the task condition, the method of evaluating the quality of the model  \n",
    "is F1-score macro metric. Steps made will be described in detail in this  \n",
    "notebook as well as some thoughts about metrics and model selection.  \n",
    "After the main classification task is finished we will tune the model  \n",
    "depending on business and write a function that allows to use the model."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e55d43e5-d14e-486c-9138-361437dc6972",
   "metadata": {},
   "source": [
    "### Import modules"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "a990ce6d-0301-4a08-a273-03d72b918c7b",
   "metadata": {},
   "outputs": [],
   "source": [
    "%matplotlib inline\n",
    "from catboost import CatBoostClassifier\n",
    "# from category_encoders import MEstimateEncoder\n",
    "import datetime\n",
    "from itertools import product\n",
    "import matplotlib.pyplot as plt\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import pickle\n",
    "import re\n",
    "from sklearn.base import BaseEstimator, TransformerMixin\n",
    "from sklearn.ensemble import RandomForestClassifier\n",
    "from sklearn.linear_model import LogisticRegression\n",
    "from sklearn.metrics import classification_report, f1_score, precision_recall_curve, precision_score, recall_score\n",
    "from sklearn.model_selection import train_test_split\n",
    "from sklearn.pipeline import FeatureUnion, Pipeline\n",
    "from sklearn.preprocessing import OneHotEncoder, StandardScaler\n",
    "from sklearn.svm import LinearSVC\n",
    "from xgboost import XGBClassifier"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "761947ef-91f8-4d38-a4bf-aab5362b0c1b",
   "metadata": {},
   "source": [
    "## 1. Data loading"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ee3ca3fb-f165-4370-a440-180e263631cf",
   "metadata": {},
   "source": [
    "Column *id* will be cast into string data type because, by default,  \n",
    "pandas load it into integer type and if some feature values have '0'  \n",
    "at their beginning, they will lose this zero."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "777f47aa-edfd-40f3-b44b-b066f68cef96",
   "metadata": {},
   "source": [
    "#### 1.1 Features dataset "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4eeb97c3-604c-4a2b-b836-18945c1c2029",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 11min 36s\n",
      "Wall time: 11min 36s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "df = pd.read_csv('features.csv', delim_whitespace=True, dtype={'id': str})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "75c4c4e6-3421-42d2-b562-0417cccf6ba0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(4512528, 255)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b46afbdd-12da-4ab7-b218-a97833a448fa",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>buy_time</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>...</th>\n",
       "      <th>243</th>\n",
       "      <th>244</th>\n",
       "      <th>245</th>\n",
       "      <th>246</th>\n",
       "      <th>247</th>\n",
       "      <th>248</th>\n",
       "      <th>249</th>\n",
       "      <th>250</th>\n",
       "      <th>251</th>\n",
       "      <th>252</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2013026</td>\n",
       "      <td>1531688400</td>\n",
       "      <td>18.910029</td>\n",
       "      <td>46.980888</td>\n",
       "      <td>4.969214</td>\n",
       "      <td>-1.386798</td>\n",
       "      <td>3.791754</td>\n",
       "      <td>-14.01179</td>\n",
       "      <td>-16.08618</td>\n",
       "      <td>-65.076097</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-301.747724</td>\n",
       "      <td>-25.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014722</td>\n",
       "      <td>1539550800</td>\n",
       "      <td>36.690029</td>\n",
       "      <td>152.400888</td>\n",
       "      <td>448.069214</td>\n",
       "      <td>563.833202</td>\n",
       "      <td>463.841754</td>\n",
       "      <td>568.99821</td>\n",
       "      <td>-16.08618</td>\n",
       "      <td>-53.216097</td>\n",
       "      <td>...</td>\n",
       "      <td>-891.373846</td>\n",
       "      <td>-544.770792</td>\n",
       "      <td>-20.996269</td>\n",
       "      <td>48.369552</td>\n",
       "      <td>80.252276</td>\n",
       "      <td>-13.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-1.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2015199</td>\n",
       "      <td>1545598800</td>\n",
       "      <td>-67.019971</td>\n",
       "      <td>157.050888</td>\n",
       "      <td>-63.180786</td>\n",
       "      <td>178.103202</td>\n",
       "      <td>-68.598246</td>\n",
       "      <td>156.99821</td>\n",
       "      <td>3.51382</td>\n",
       "      <td>25.183903</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-12.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>10829.252276</td>\n",
       "      <td>-25.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2021765</td>\n",
       "      <td>1534107600</td>\n",
       "      <td>7.010029</td>\n",
       "      <td>150.200888</td>\n",
       "      <td>-6.930786</td>\n",
       "      <td>216.213202</td>\n",
       "      <td>76.621754</td>\n",
       "      <td>351.84821</td>\n",
       "      <td>-16.08618</td>\n",
       "      <td>-65.076097</td>\n",
       "      <td>...</td>\n",
       "      <td>-973.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-23.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-205.747724</td>\n",
       "      <td>-24.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-11.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2027465</td>\n",
       "      <td>1533502800</td>\n",
       "      <td>-90.439971</td>\n",
       "      <td>134.220888</td>\n",
       "      <td>-104.380786</td>\n",
       "      <td>153.643202</td>\n",
       "      <td>-109.798246</td>\n",
       "      <td>132.53821</td>\n",
       "      <td>-16.08618</td>\n",
       "      <td>-65.076097</td>\n",
       "      <td>...</td>\n",
       "      <td>1643.626154</td>\n",
       "      <td>2007.229208</td>\n",
       "      <td>206.003731</td>\n",
       "      <td>-21.630448</td>\n",
       "      <td>6667.252276</td>\n",
       "      <td>92.167111</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>49.824067</td>\n",
       "      <td>47.54386</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 255 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id    buy_time          0           1           2           3  \\\n",
       "0  2013026  1531688400  18.910029   46.980888    4.969214   -1.386798   \n",
       "1  2014722  1539550800  36.690029  152.400888  448.069214  563.833202   \n",
       "2  2015199  1545598800 -67.019971  157.050888  -63.180786  178.103202   \n",
       "3  2021765  1534107600   7.010029  150.200888   -6.930786  216.213202   \n",
       "4  2027465  1533502800 -90.439971  134.220888 -104.380786  153.643202   \n",
       "\n",
       "            4          5         6          7  ...          243          244  \\\n",
       "0    3.791754  -14.01179 -16.08618 -65.076097  ...  -977.373846  -613.770792   \n",
       "1  463.841754  568.99821 -16.08618 -53.216097  ...  -891.373846  -544.770792   \n",
       "2  -68.598246  156.99821   3.51382  25.183903  ...  -977.373846  -613.770792   \n",
       "3   76.621754  351.84821 -16.08618 -65.076097  ...  -973.373846  -613.770792   \n",
       "4 -109.798246  132.53821 -16.08618 -65.076097  ...  1643.626154  2007.229208   \n",
       "\n",
       "          245        246           247        248       249        250  \\\n",
       "0  -25.996269 -37.630448   -301.747724 -25.832889 -0.694428 -12.175933   \n",
       "1  -20.996269  48.369552     80.252276 -13.832889 -0.694428  -1.175933   \n",
       "2  -12.996269 -37.630448  10829.252276 -25.832889 -0.694428 -12.175933   \n",
       "3  -23.996269 -37.630448   -205.747724 -24.832889 -0.694428 -11.175933   \n",
       "4  206.003731 -21.630448   6667.252276  92.167111 -0.694428  49.824067   \n",
       "\n",
       "        251  252  \n",
       "0  -0.45614  0.0  \n",
       "1  -0.45614  0.0  \n",
       "2  -0.45614  0.0  \n",
       "3  -0.45614  1.0  \n",
       "4  47.54386  0.0  \n",
       "\n",
       "[5 rows x 255 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f403aba3-c054-43ac-9eb5-d2a986a907a2",
   "metadata": {},
   "source": [
    "We have the following attributes:  \n",
    "- **id** - unique customer number.  \n",
    "- **buy_time** - customer's profile date of creation.  \n",
    "- **other attributes** - anonymized set of attributes."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1e9378a1-ec68-4638-b4e7-3cd4c056b0ce",
   "metadata": {},
   "source": [
    "#### 1.2 Training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "a31c4a7c-a66b-4e88-b988-1fd018045cf5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(831653, 4)"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train = pd.read_csv('data_train.csv', index_col='Unnamed: 0', dtype={'id': str})\n",
    "df_train.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "f98474f1-c884-4a44-b967-645106e56274",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>vas_id</th>\n",
       "      <th>buy_time</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>540968</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1537131600</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1454121</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1531688400</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2458816</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1534107600</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3535012</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1535922000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1693214</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1535922000</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  vas_id    buy_time  target\n",
       "0   540968     8.0  1537131600     0.0\n",
       "1  1454121     4.0  1531688400     0.0\n",
       "2  2458816     1.0  1534107600     0.0\n",
       "3  3535012     5.0  1535922000     0.0\n",
       "4  1693214     1.0  1535922000     0.0"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08878f27-942c-468f-a69f-651ab1d63afd",
   "metadata": {},
   "source": [
    "Attributes:  \n",
    "- **id** - same as in feature dataset.  \n",
    "- **vas_id** - proposed service unique number.  \n",
    "- **buy_time** - service proposing date.  \n",
    "- **target** - label, where:  \n",
    "1 - customer accepted the service proposal;  \n",
    "0 - customer didn't accept the service proposal."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1191a0ab-9851-48ae-90ca-d9a9c85cc8dd",
   "metadata": {},
   "source": [
    "#### 1.3 Test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "75b25850-cf24-415f-80a1-5cda7f685908",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(71231, 3)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test = pd.read_csv('data_test.csv', index_col='Unnamed: 0', dtype={'id': str})\n",
    "df_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "35bc45f8-a1c7-4ee2-aaee-07a31cfce646",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>vas_id</th>\n",
       "      <th>buy_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3130519</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1548018000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000860</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1548018000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1099444</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1546808400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1343255</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1547413200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1277040</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1546808400</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  vas_id    buy_time\n",
       "0  3130519     2.0  1548018000\n",
       "1  2000860     4.0  1548018000\n",
       "2  1099444     2.0  1546808400\n",
       "3  1343255     5.0  1547413200\n",
       "4  1277040     2.0  1546808400"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61434dc0-65e9-43af-b816-080d7f51c935",
   "metadata": {},
   "source": [
    "Attributes are the same as in the training dataset,  \n",
    "except for the *target*, which we need to predict."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "69168902-723b-4db5-b4f8-995599e8e7d9",
   "metadata": {},
   "source": [
    "## 2. EDA <a class='anchor' id='edas'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5bdc647e-1c34-4399-a2fc-a7239cf8ae22",
   "metadata": {},
   "source": [
    "#### 2.1 Features dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "9215149b-529b-463e-8a83-53b310b86544",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "float64    253\n",
       "object       1\n",
       "int64        1\n",
       "dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.dtypes.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93b1bede-3289-4a72-bad6-01975c1e7f8e",
   "metadata": {},
   "source": [
    "All anonymized attributes are continuous."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "4cf83107-3267-4cee-9ac6-a0bd9e52aae9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "149789"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.shape[0] - df['id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38e77c07-bbc5-48ce-a313-403fd4e00356",
   "metadata": {},
   "source": [
    "Duplicated unique id values are present."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "c95203e3-1d42-4d31-83ec-297ca162599c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(0, 255)"
      ]
     },
     "execution_count": 11,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.duplicated(['id', 'buy_time'], keep=False)].shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03180b68-15fb-4fbf-ac32-866dcbe9a261",
   "metadata": {},
   "source": [
    "There is no duplicate where we have the same unique id and  \n",
    "profile creation date. This tells us that a customer's profile  \n",
    "was being updated during time. Let's have a look at these customers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "615e1ed1-f563-4c57-873a-c98b308ec6e5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>buy_time</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>5</th>\n",
       "      <th>6</th>\n",
       "      <th>7</th>\n",
       "      <th>...</th>\n",
       "      <th>243</th>\n",
       "      <th>244</th>\n",
       "      <th>245</th>\n",
       "      <th>246</th>\n",
       "      <th>247</th>\n",
       "      <th>248</th>\n",
       "      <th>249</th>\n",
       "      <th>250</th>\n",
       "      <th>251</th>\n",
       "      <th>252</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>2709826</th>\n",
       "      <td>1000</td>\n",
       "      <td>1544389200</td>\n",
       "      <td>-64.179971</td>\n",
       "      <td>-261.519112</td>\n",
       "      <td>-66.290786</td>\n",
       "      <td>-266.806798</td>\n",
       "      <td>-71.708246</td>\n",
       "      <td>-287.91179</td>\n",
       "      <td>16.53382</td>\n",
       "      <td>65.403903</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-22.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-283.747724</td>\n",
       "      <td>-25.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1561535</th>\n",
       "      <td>1000</td>\n",
       "      <td>1546203600</td>\n",
       "      <td>-60.909971</td>\n",
       "      <td>-208.119112</td>\n",
       "      <td>-63.140786</td>\n",
       "      <td>-213.526798</td>\n",
       "      <td>-68.558246</td>\n",
       "      <td>-234.63179</td>\n",
       "      <td>16.05382</td>\n",
       "      <td>64.923903</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-288.747724</td>\n",
       "      <td>-24.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>794594</th>\n",
       "      <td>1000009</td>\n",
       "      <td>1542574800</td>\n",
       "      <td>-89.159971</td>\n",
       "      <td>-356.269112</td>\n",
       "      <td>-103.100786</td>\n",
       "      <td>-404.636798</td>\n",
       "      <td>-108.518246</td>\n",
       "      <td>-425.74179</td>\n",
       "      <td>-16.08618</td>\n",
       "      <td>-65.076097</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>11.369552</td>\n",
       "      <td>-288.747724</td>\n",
       "      <td>-25.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>500359</th>\n",
       "      <td>1000009</td>\n",
       "      <td>1534712400</td>\n",
       "      <td>-14.989971</td>\n",
       "      <td>32.140888</td>\n",
       "      <td>-16.220786</td>\n",
       "      <td>-3.516798</td>\n",
       "      <td>-21.638246</td>\n",
       "      <td>-24.62179</td>\n",
       "      <td>-16.08618</td>\n",
       "      <td>-23.556097</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-303.747724</td>\n",
       "      <td>-25.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2236836</th>\n",
       "      <td>1000016</td>\n",
       "      <td>1537736400</td>\n",
       "      <td>51.510029</td>\n",
       "      <td>36.750888</td>\n",
       "      <td>37.569214</td>\n",
       "      <td>70.583202</td>\n",
       "      <td>32.151754</td>\n",
       "      <td>49.47821</td>\n",
       "      <td>-16.08618</td>\n",
       "      <td>-65.076097</td>\n",
       "      <td>...</td>\n",
       "      <td>-976.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-220.747724</td>\n",
       "      <td>-2.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>10.824067</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>817063</th>\n",
       "      <td>999945</td>\n",
       "      <td>1542574800</td>\n",
       "      <td>-92.219971</td>\n",
       "      <td>-106.989112</td>\n",
       "      <td>-106.160786</td>\n",
       "      <td>-129.936798</td>\n",
       "      <td>-111.578246</td>\n",
       "      <td>-151.04179</td>\n",
       "      <td>-16.08618</td>\n",
       "      <td>-65.076097</td>\n",
       "      <td>...</td>\n",
       "      <td>-975.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-3.630448</td>\n",
       "      <td>-179.747724</td>\n",
       "      <td>-14.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183494</th>\n",
       "      <td>999956</td>\n",
       "      <td>1543179600</td>\n",
       "      <td>-51.379971</td>\n",
       "      <td>-233.959112</td>\n",
       "      <td>-65.320786</td>\n",
       "      <td>-286.566798</td>\n",
       "      <td>-70.738246</td>\n",
       "      <td>-307.67179</td>\n",
       "      <td>21.19382</td>\n",
       "      <td>79.383903</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-32.630448</td>\n",
       "      <td>-298.747724</td>\n",
       "      <td>-25.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1717302</th>\n",
       "      <td>999956</td>\n",
       "      <td>1537736400</td>\n",
       "      <td>-68.839971</td>\n",
       "      <td>-247.119112</td>\n",
       "      <td>-82.780786</td>\n",
       "      <td>-283.616798</td>\n",
       "      <td>-88.198246</td>\n",
       "      <td>-304.72179</td>\n",
       "      <td>11.87382</td>\n",
       "      <td>60.763903</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-290.747724</td>\n",
       "      <td>-25.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1087142</th>\n",
       "      <td>99999</td>\n",
       "      <td>1539550800</td>\n",
       "      <td>25.350029</td>\n",
       "      <td>91.430888</td>\n",
       "      <td>11.409214</td>\n",
       "      <td>43.063202</td>\n",
       "      <td>5.991754</td>\n",
       "      <td>21.95821</td>\n",
       "      <td>7.64382</td>\n",
       "      <td>29.843903</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-306.747724</td>\n",
       "      <td>-25.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2551434</th>\n",
       "      <td>99999</td>\n",
       "      <td>1541365200</td>\n",
       "      <td>-78.489971</td>\n",
       "      <td>40.950888</td>\n",
       "      <td>-92.430786</td>\n",
       "      <td>-11.656798</td>\n",
       "      <td>-97.848246</td>\n",
       "      <td>-32.76179</td>\n",
       "      <td>-16.08618</td>\n",
       "      <td>-21.006097</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-306.747724</td>\n",
       "      <td>-25.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>1.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>299578 rows × 255 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "              id    buy_time          0           1           2           3  \\\n",
       "2709826     1000  1544389200 -64.179971 -261.519112  -66.290786 -266.806798   \n",
       "1561535     1000  1546203600 -60.909971 -208.119112  -63.140786 -213.526798   \n",
       "794594   1000009  1542574800 -89.159971 -356.269112 -103.100786 -404.636798   \n",
       "500359   1000009  1534712400 -14.989971   32.140888  -16.220786   -3.516798   \n",
       "2236836  1000016  1537736400  51.510029   36.750888   37.569214   70.583202   \n",
       "...          ...         ...        ...         ...         ...         ...   \n",
       "817063    999945  1542574800 -92.219971 -106.989112 -106.160786 -129.936798   \n",
       "183494    999956  1543179600 -51.379971 -233.959112  -65.320786 -286.566798   \n",
       "1717302   999956  1537736400 -68.839971 -247.119112  -82.780786 -283.616798   \n",
       "1087142    99999  1539550800  25.350029   91.430888   11.409214   43.063202   \n",
       "2551434    99999  1541365200 -78.489971   40.950888  -92.430786  -11.656798   \n",
       "\n",
       "                  4          5         6          7  ...         243  \\\n",
       "2709826  -71.708246 -287.91179  16.53382  65.403903  ... -977.373846   \n",
       "1561535  -68.558246 -234.63179  16.05382  64.923903  ... -977.373846   \n",
       "794594  -108.518246 -425.74179 -16.08618 -65.076097  ... -977.373846   \n",
       "500359   -21.638246  -24.62179 -16.08618 -23.556097  ... -977.373846   \n",
       "2236836   32.151754   49.47821 -16.08618 -65.076097  ... -976.373846   \n",
       "...             ...        ...       ...        ...  ...         ...   \n",
       "817063  -111.578246 -151.04179 -16.08618 -65.076097  ... -975.373846   \n",
       "183494   -70.738246 -307.67179  21.19382  79.383903  ... -977.373846   \n",
       "1717302  -88.198246 -304.72179  11.87382  60.763903  ... -977.373846   \n",
       "1087142    5.991754   21.95821   7.64382  29.843903  ... -977.373846   \n",
       "2551434  -97.848246  -32.76179 -16.08618 -21.006097  ... -977.373846   \n",
       "\n",
       "                244        245        246         247        248       249  \\\n",
       "2709826 -613.770792 -22.996269 -37.630448 -283.747724 -25.832889 -0.694428   \n",
       "1561535 -613.770792 -25.996269 -37.630448 -288.747724 -24.832889 -0.694428   \n",
       "794594  -613.770792 -25.996269  11.369552 -288.747724 -25.832889 -0.694428   \n",
       "500359  -613.770792 -25.996269 -37.630448 -303.747724 -25.832889 -0.694428   \n",
       "2236836 -613.770792 -25.996269 -37.630448 -220.747724  -2.832889 -0.694428   \n",
       "...             ...        ...        ...         ...        ...       ...   \n",
       "817063  -613.770792 -25.996269  -3.630448 -179.747724 -14.832889 -0.694428   \n",
       "183494  -613.770792 -25.996269 -32.630448 -298.747724 -25.832889 -0.694428   \n",
       "1717302 -613.770792 -25.996269 -37.630448 -290.747724 -25.832889 -0.694428   \n",
       "1087142 -613.770792 -25.996269 -37.630448 -306.747724 -25.832889 -0.694428   \n",
       "2551434 -613.770792 -25.996269 -37.630448 -306.747724 -25.832889 -0.694428   \n",
       "\n",
       "               250      251  252  \n",
       "2709826 -12.175933 -0.45614  0.0  \n",
       "1561535 -12.175933 -0.45614  0.0  \n",
       "794594  -12.175933 -0.45614  0.0  \n",
       "500359  -12.175933 -0.45614  0.0  \n",
       "2236836  10.824067 -0.45614  1.0  \n",
       "...            ...      ...  ...  \n",
       "817063  -12.175933 -0.45614  0.0  \n",
       "183494  -12.175933 -0.45614  0.0  \n",
       "1717302 -12.175933 -0.45614  1.0  \n",
       "1087142 -12.175933 -0.45614  1.0  \n",
       "2551434 -12.175933 -0.45614  1.0  \n",
       "\n",
       "[299578 rows x 255 columns]"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[df.duplicated(['id'], keep=False)].sort_values('id')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f4ea312c-d541-4d54-85d4-706a809c1c23",
   "metadata": {},
   "source": [
    "Indeed, some features of certain customers were being\n",
    "updated during time.  \n",
    "Let's have a look if we have missing values in the dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "89f34186-11cf-4dee-8be0-b6c0c6e44ddd",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.isna().sum().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0d072183-de42-4643-b0d9-172afa5ecd88",
   "metadata": {},
   "source": [
    "There are no missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4fb86150-2091-4480-8f4a-e09b48921c50",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count                          4512528\n",
       "mean     2018-10-15 12:03:06.373934080\n",
       "min                2018-07-09 00:00:00\n",
       "25%                2018-08-20 00:00:00\n",
       "50%                2018-10-15 00:00:00\n",
       "75%                2018-12-17 00:00:00\n",
       "max                2019-01-21 00:00:00\n",
       "Name: buy_time, dtype: object"
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df['buy_time'].apply(lambda x: datetime.datetime.fromtimestamp(x)).describe(datetime_is_numeric=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0b339e1-e93e-4e97-bafa-09c4894e3cb8",
   "metadata": {},
   "source": [
    "We have customers' profiles creation dates from July 2018  \n",
    "to January 2019."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aa59de78-bc39-42a2-a521-45ea24db0338",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 2.2 Training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "fa154f34-a7ee-4e69-9647-36d6166f2559",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0    0.927631\n",
       "1.0    0.072369\n",
       "Name: target, dtype: float64"
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train['target'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c74fc283-cc60-4a9c-8e59-8ac31167fdf2",
   "metadata": {},
   "source": [
    "The dataset has an enormous disbalance of classes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "c597fab4-fa2f-4e37-b3ae-ee544dc2aad8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "25040"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.shape[0] - df_train['id'].nunique()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cef1031-6187-4c6d-9e96-19d0cbba1677",
   "metadata": {},
   "source": [
    "There are repetitions among customers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "0c52dc31-3013-4d01-a62e-eb76abd5de1a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "12498"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train[df_train.duplicated(['id', 'vas_id'], keep=False)].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "86ca769e-aeb6-41b0-b545-adac4fd9be83",
   "metadata": {},
   "source": [
    "Some customers got proposals of the same service more than once."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "9dcc3018-1611-4324-946a-4c59c1ee666c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train[df_train.duplicated(['id', 'vas_id', 'buy_time'], keep=False)].shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42fb994d-7854-4ebc-ad8a-2d3c07fd274e",
   "metadata": {},
   "source": [
    "The dataset doesn't have full duplicates."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "30272043-2b5b-40c2-8261-457ba371a24e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.isna().sum().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ffb1ef5d-cfb5-47b1-b697-ab8262f8aeeb",
   "metadata": {},
   "source": [
    "There are no missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "169fd1c3-ce6b-4b55-9f6d-aa25bcbc89f1",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1.0    310175\n",
       "2.0    249505\n",
       "5.0     94085\n",
       "4.0     85756\n",
       "6.0     57878\n",
       "7.0     15432\n",
       "8.0     13350\n",
       "9.0      5472\n",
       "Name: vas_id, dtype: int64"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train['vas_id'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ec2e20d-4dd6-4b14-b0ef-7a7598eeed41",
   "metadata": {},
   "source": [
    "We have 8 different proposable services in total."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "fc13bddb-fdc6-42f0-b496-1e4bc43783ed",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count                           831653\n",
       "mean     2018-10-07 21:23:38.779707904\n",
       "min                2018-07-09 00:00:00\n",
       "25%                2018-08-13 00:00:00\n",
       "50%                2018-10-01 00:00:00\n",
       "75%                2018-12-10 00:00:00\n",
       "max                2018-12-31 00:00:00\n",
       "Name: buy_time, dtype: object"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train['buy_time'].apply(lambda x: datetime.datetime.fromtimestamp(x)).describe(datetime_is_numeric=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "42cac644-3070-4e17-9e43-1aaa6c3f0fb3",
   "metadata": {},
   "source": [
    "In the training dataset the services were being proposed  \n",
    "to the customers starting from July 2018 till December 2018."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "649da0b8-a4f8-446d-86e3-093bd516ff05",
   "metadata": {
    "tags": []
   },
   "source": [
    "#### 2.3 Test dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "be690b1b-b225-44cc-87e9-ac1c557e512b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1079"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test.shape[0] - df_test['id'].nunique()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "7ec9d11a-8784-4d08-82d6-b615483893fe",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "184"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test[df_test.duplicated(['id', 'vas_id'], keep=False)].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "eba4aadb-22c2-48a8-bd46-86e73c7eda8b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test[df_test.duplicated(['id', 'vas_id', 'buy_time'], keep=False)].shape[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "e0858106-e880-4ad3-a017-6cc72a4c5f1d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test.isna().sum().sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "50892969-f374-4f1b-a96c-beb6b98b3d3f",
   "metadata": {},
   "source": [
    "It's the same as in the training dataset as far as duplicates  \n",
    "and missing values."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "94504c5a-50de-4745-94a3-369119567569",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "count                            71231\n",
       "mean     2019-01-12 20:42:25.790456320\n",
       "min                2019-01-07 00:00:00\n",
       "25%                2019-01-07 00:00:00\n",
       "50%                2019-01-14 00:00:00\n",
       "75%                2019-01-14 00:00:00\n",
       "max                2019-01-21 00:00:00\n",
       "Name: buy_time, dtype: object"
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test['buy_time'].apply(lambda x: datetime.datetime.fromtimestamp(x)).describe(datetime_is_numeric=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a62ae3b-72f6-4bcf-86cc-363fb52d3743",
   "metadata": {},
   "source": [
    "In the test dataset the services were being proposed to the customers  \n",
    "only in January 2019."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76e9d4f7-16a4-4556-9bb2-6facd8029434",
   "metadata": {},
   "source": [
    "From the preliminary acquaintance with the datasets we may conclude  \n",
    "that we don't need every row of the features dataset to merge it with  \n",
    "the train and test datasets, because the number of customers there  \n",
    "is considerably fewer. Thus, there is potential for decreasing the  \n",
    "size of the initial large dataset in case, e.g., lack of memory to store it.  \n",
    "In our case we will leave the dataset as is, because we will need the  \n",
    "additional rows for more precise definition of the features' distributions  \n",
    "in order to handle outliers and because the size of the initial dataset  \n",
    "is not actually a problem."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db45ddec-540d-40c7-902a-a7e00d5795ae",
   "metadata": {},
   "source": [
    "## 3. Baseline model training <a class='anchor' id='baseline'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "844a607a-7397-4143-8c33-6e0530194f2c",
   "metadata": {},
   "source": [
    "To prepare a complete training dataset let's merge our training  \n",
    "dataset with the features dataset by unique customer number:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "9a9ca8e1-9759-4705-a756-f7592c784795",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df_train.merge(df, how='left', on='id')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "21ddcd66-1fdc-422b-96ea-a0c5299e586b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>vas_id</th>\n",
       "      <th>buy_time_x</th>\n",
       "      <th>target</th>\n",
       "      <th>buy_time_y</th>\n",
       "      <th>0</th>\n",
       "      <th>1</th>\n",
       "      <th>2</th>\n",
       "      <th>3</th>\n",
       "      <th>4</th>\n",
       "      <th>...</th>\n",
       "      <th>243</th>\n",
       "      <th>244</th>\n",
       "      <th>245</th>\n",
       "      <th>246</th>\n",
       "      <th>247</th>\n",
       "      <th>248</th>\n",
       "      <th>249</th>\n",
       "      <th>250</th>\n",
       "      <th>251</th>\n",
       "      <th>252</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>540968</td>\n",
       "      <td>8.0</td>\n",
       "      <td>1537131600</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1541970000</td>\n",
       "      <td>-31.559971</td>\n",
       "      <td>327.360888</td>\n",
       "      <td>-45.500786</td>\n",
       "      <td>274.753202</td>\n",
       "      <td>-50.918246</td>\n",
       "      <td>...</td>\n",
       "      <td>-845.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-20.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-28.747724</td>\n",
       "      <td>4.167111</td>\n",
       "      <td>7.305572</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>21.54386</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1454121</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1531688400</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1531083600</td>\n",
       "      <td>547.270029</td>\n",
       "      <td>238.430888</td>\n",
       "      <td>533.329214</td>\n",
       "      <td>274.803202</td>\n",
       "      <td>527.911754</td>\n",
       "      <td>...</td>\n",
       "      <td>-972.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-19.630448</td>\n",
       "      <td>-278.747724</td>\n",
       "      <td>-24.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-11.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2458816</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1534107600</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1543179600</td>\n",
       "      <td>-92.139971</td>\n",
       "      <td>-95.469112</td>\n",
       "      <td>-106.080786</td>\n",
       "      <td>-139.596798</td>\n",
       "      <td>-111.498246</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-37.630448</td>\n",
       "      <td>-304.747724</td>\n",
       "      <td>-25.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>3535012</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1535922000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1533502800</td>\n",
       "      <td>54.880029</td>\n",
       "      <td>12.970888</td>\n",
       "      <td>54.079214</td>\n",
       "      <td>-9.116798</td>\n",
       "      <td>48.661754</td>\n",
       "      <td>...</td>\n",
       "      <td>-977.373846</td>\n",
       "      <td>-613.770792</td>\n",
       "      <td>-25.996269</td>\n",
       "      <td>-18.630448</td>\n",
       "      <td>-133.747724</td>\n",
       "      <td>-14.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-1.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1693214</td>\n",
       "      <td>1.0</td>\n",
       "      <td>1535922000</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1543179600</td>\n",
       "      <td>45.160029</td>\n",
       "      <td>295.240888</td>\n",
       "      <td>64.679214</td>\n",
       "      <td>344.283202</td>\n",
       "      <td>59.261754</td>\n",
       "      <td>...</td>\n",
       "      <td>-965.373846</td>\n",
       "      <td>-612.770792</td>\n",
       "      <td>-22.996269</td>\n",
       "      <td>-32.630448</td>\n",
       "      <td>-127.747724</td>\n",
       "      <td>-4.832889</td>\n",
       "      <td>-0.694428</td>\n",
       "      <td>-12.175933</td>\n",
       "      <td>-0.45614</td>\n",
       "      <td>0.0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 258 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  vas_id  buy_time_x  target  buy_time_y           0           1  \\\n",
       "0   540968     8.0  1537131600     0.0  1541970000  -31.559971  327.360888   \n",
       "1  1454121     4.0  1531688400     0.0  1531083600  547.270029  238.430888   \n",
       "2  2458816     1.0  1534107600     0.0  1543179600  -92.139971  -95.469112   \n",
       "3  3535012     5.0  1535922000     0.0  1533502800   54.880029   12.970888   \n",
       "4  1693214     1.0  1535922000     0.0  1543179600   45.160029  295.240888   \n",
       "\n",
       "            2           3           4  ...         243         244        245  \\\n",
       "0  -45.500786  274.753202  -50.918246  ... -845.373846 -613.770792 -20.996269   \n",
       "1  533.329214  274.803202  527.911754  ... -972.373846 -613.770792 -25.996269   \n",
       "2 -106.080786 -139.596798 -111.498246  ... -977.373846 -613.770792 -25.996269   \n",
       "3   54.079214   -9.116798   48.661754  ... -977.373846 -613.770792 -25.996269   \n",
       "4   64.679214  344.283202   59.261754  ... -965.373846 -612.770792 -22.996269   \n",
       "\n",
       "         246         247        248       249        250       251  252  \n",
       "0 -37.630448  -28.747724   4.167111  7.305572 -12.175933  21.54386  0.0  \n",
       "1 -19.630448 -278.747724 -24.832889 -0.694428 -11.175933  -0.45614  0.0  \n",
       "2 -37.630448 -304.747724 -25.832889 -0.694428 -12.175933  -0.45614  0.0  \n",
       "3 -18.630448 -133.747724 -14.832889 -0.694428  -1.175933  -0.45614  0.0  \n",
       "4 -32.630448 -127.747724  -4.832889 -0.694428 -12.175933  -0.45614  0.0  \n",
       "\n",
       "[5 rows x 258 columns]"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "6e4c5e18-a8ba-4aee-a101-26941e8df0f8",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(860052, 258)"
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_train.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9309f576-3ae4-45f3-9014-d3afde5f2f7f",
   "metadata": {},
   "source": [
    "The size of the dataset has increased due to the duplicated customers' ids  \n",
    "from the features dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94b009d0-d18f-4f81-b724-6b0230562d7f",
   "metadata": {},
   "source": [
    "Let's set *random_state* value for reproducibility of the results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "f5927577-1ccd-4778-86d2-9908b47ab266",
   "metadata": {},
   "outputs": [],
   "source": [
    "RANDOM_STATE = 29"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "117c29e4-969b-4314-a329-e9884d6fc85f",
   "metadata": {},
   "source": [
    "Split for train and test:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "b4cce273-18fa-42d1-b80e-5c96183a033f",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train, X_test, y_train, y_test = train_test_split(df_train.drop(['target', 'id'], axis=1), df_train['target'],\n",
    "                                                    test_size=0.3, random_state=RANDOM_STATE)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c889ae67-a092-4556-a691-612609b86ef0",
   "metadata": {},
   "source": [
    "Resting assured the samples are well stratified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "e38e1aa9-0dde-4c9d-8c0c-58c727e676df",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0    0.927474\n",
       "1.0    0.072526\n",
       "Name: target, dtype: float64"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_train.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "87a765b2-c60c-4c9b-9b9c-130db584264d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.0    0.928128\n",
       "1.0    0.071872\n",
       "Name: target, dtype: float64"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "y_test.value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "03dc1251-f500-4311-ab67-3a261b547946",
   "metadata": {},
   "source": [
    "We'll choose logistic regression for the baseline model due to its simplicity.  \n",
    "We'll scale the data before training the model because linear models may  \n",
    "incorrectly determine patterns in data if it's not scaled."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "9431c1f0-8744-4c55-bb30-b4044e2f2e00",
   "metadata": {},
   "outputs": [],
   "source": [
    "baseline_model = Pipeline([\n",
    "    ('scaler', StandardScaler()),\n",
    "    ('classifier', LogisticRegression(max_iter=500))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "14458e16-66c7-48fc-9f0c-9d01de25d8bb",
   "metadata": {},
   "source": [
    "Training the model and predicting the probability of belonging to class 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "e936aecd-bf27-4288-ba77-9c6dfecffae0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Pipeline(steps=[('scaler', StandardScaler()),\n",
       "                ('classifier', LogisticRegression(max_iter=500))])"
      ]
     },
     "execution_count": 35,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "baseline_model.fit(X_train, y_train)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "ae4a58d4-5670-4391-baba-1a0a22ba8e01",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = baseline_model.predict_proba(X_test)[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cae8b023-743c-4fe4-b56b-684935718418",
   "metadata": {},
   "source": [
    "Let's write a function that searches for an optimal classification threshold.  \n",
    "The search will be performed taking maximization of F1-score of the class 1  \n",
    "into account. With such classes' disbalance it's F1-score of the class 1  \n",
    "metric that will actively react to changing the classification threshold,  \n",
    "which, in turn, will influence F1-score macro:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "82ae4dec-8ca6-47b8-adc2-717dd8a24f96",
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_best_threshold(y_test, y_pred, verbose=True):\n",
    "\n",
    "    # Finding component values of F-score metric and thresholds,\n",
    "    # that give us these values\n",
    "    precision, recall, thresholds = precision_recall_curve(y_test, y_pred)\n",
    "\n",
    "    # Counting F1-score class 1 for every threshold\n",
    "    fscore = (2 * precision * recall) / np.clip((precision + recall), a_min=1e-4, a_max=np.inf)\n",
    "    \n",
    "    # Finding index, which gives us maximum F1-score\n",
    "    idx = np.argmax(fscore)\n",
    "              \n",
    "    if verbose:\n",
    "        print(f'Best Threshold={thresholds[idx]}')\n",
    "    \n",
    "    # Returning threshold value, where F1-score is maximum\n",
    "    return thresholds[idx]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "48078b86-596e-4439-8e14-0ff11e802971",
   "metadata": {},
   "source": [
    "Finding the classification threshold for the baseline model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "141f620d-72ce-4ce6-a52a-43f6e0a8fc40",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Threshold=0.2009788465918036\n"
     ]
    }
   ],
   "source": [
    "baseline_threshold = find_best_threshold(y_test, y_pred)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "703e9f4d-b8c0-4dfd-9411-a7e6ba3addbf",
   "metadata": {},
   "source": [
    "Printing basic metrics values of the baseline model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "b10d3f0a-e860-416e-a565-76b335c72827",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.95      0.93      0.94    239472\n",
      "         1.0       0.29      0.38      0.33     18544\n",
      "\n",
      "    accuracy                           0.89    258016\n",
      "   macro avg       0.62      0.65      0.63    258016\n",
      "weighted avg       0.90      0.89      0.90    258016\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, y_pred >= baseline_threshold))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a955d95-a7ea-49ed-989e-84566819161c",
   "metadata": {},
   "source": [
    "Let's have a look at the attributes, which the baseline model   \n",
    "considered the most important:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "c3035361-3152-4999-aa28-6f79f743a424",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coefficient</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>vas_id</th>\n",
       "      <td>0.768880</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>buy_time_x</th>\n",
       "      <td>0.410939</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>45</th>\n",
       "      <td>0.316268</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>162</th>\n",
       "      <td>0.235124</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209</th>\n",
       "      <td>0.176331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>43</th>\n",
       "      <td>0.174638</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>123</th>\n",
       "      <td>0.143851</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14</th>\n",
       "      <td>0.112211</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50</th>\n",
       "      <td>0.096607</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>116</th>\n",
       "      <td>0.073154</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            coefficient\n",
       "vas_id         0.768880\n",
       "buy_time_x     0.410939\n",
       "45             0.316268\n",
       "162            0.235124\n",
       "209            0.176331\n",
       "43             0.174638\n",
       "123            0.143851\n",
       "14             0.112211\n",
       "50             0.096607\n",
       "116            0.073154"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = pd.DataFrame(baseline_model['classifier'].coef_.reshape(-1, 1), \n",
    "                        X_train.columns, \n",
    "                        columns=['coefficient'])\n",
    "\n",
    "features.sort_values('coefficient', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fb21a769-5f72-4ba4-8f27-be5193646f04",
   "metadata": {},
   "source": [
    "The most important attribute for the model - service, proposed to a customer."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2c1704bf-b884-4fbb-b392-8a4564dc76d0",
   "metadata": {},
   "source": [
    "## 4. Handling outliers <a class='anchor' id='outliers'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10cc488d-3355-4e90-b773-19a732f25044",
   "metadata": {},
   "source": [
    "In order to avoid possible outliers in a set of anonymized continuous features, we will  \n",
    "filter out values which exceed 1.5 interquartile ranges. Let's write a function for this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "4878c494-c45b-435d-aba5-9c553d20cee2",
   "metadata": {},
   "outputs": [],
   "source": [
    "def process_outliers(df, column):\n",
    "    \n",
    "    # Finding 1st and 3rd quartiles:\n",
    "    q25, q75 = np.quantile(df[column], 0.25), np.quantile(df[column], 0.75)\n",
    "    \n",
    "    # Interquartile range is a distance between them\n",
    "    iqr = q75 - q25\n",
    "    \n",
    "    # Calculating upper and lower bounds, values beyond them we'll\n",
    "    # consider as outliers\n",
    "    lower_bound = q25 - 1.5 * iqr\n",
    "    upper_bound = q75 + 1.5 * iqr\n",
    "    \n",
    "    # We'll replace the outliers with mode because many features have\n",
    "    # an overwhelming majority of one certain value\n",
    "    df.loc[(df[column] < lower_bound) | (df[column] > upper_bound), column] = df[column].value_counts().index[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0b478d3-c30c-4fbf-80a5-de80b8435bb0",
   "metadata": {},
   "source": [
    "Let's randomly take 3 features and make sure we were right about the majority  \n",
    "of just one value in the features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "1e61706c-e7d2-4027-8e8c-808cea1bd696",
   "metadata": {},
   "outputs": [],
   "source": [
    "np.random.seed(0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "4383c2c1-46f9-4d3a-92d1-15608246cb37",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-30.821719    1083377\n",
       "-30.571719       5022\n",
       "-30.588386       4777\n",
       "-30.521719       4770\n",
       "-30.655052       4671\n",
       "Name: 172, dtype: int64"
      ]
     },
     "execution_count": 43,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[f'{np.random.randint(0, 250)}'].value_counts().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "a8e237b1-93be-4f99-9da7-afe46496c4e7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-7.997875    2364084\n",
       "-7.397875         98\n",
       "-6.797875         74\n",
       "-7.697875         66\n",
       "-7.797875         64\n",
       "Name: 47, dtype: int64"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[f'{np.random.randint(0, 250)}'].value_counts().head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "acf4af96-978e-4c93-85a7-44f0f2e9de6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "-2298.725139    1471676\n",
       "-2298.724162       6003\n",
       "-2298.722209       1817\n",
       "-2298.723185       1794\n",
       "-2298.723186       1741\n",
       "Name: 117, dtype: int64"
      ]
     },
     "execution_count": 45,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[f'{np.random.randint(0, 250)}'].value_counts().head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ca690a4f-c3a3-4957-aab2-324d440155bd",
   "metadata": {},
   "source": [
    "Processing continuous features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 46,
   "id": "4b0969d8-ce28-4e3c-948a-80c1782332be",
   "metadata": {},
   "outputs": [],
   "source": [
    "for column in df.columns:\n",
    "    if column not in ['id', 'buy_time']:\n",
    "        process_outliers(df, column)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4803fa3f-119d-4225-84d1-e5f642d502c7",
   "metadata": {},
   "source": [
    "## 5. Feature types determination <a class='anchor' id='types'>"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "43169f88-7c4a-4936-b8b4-65cbe58f3c35",
   "metadata": {},
   "source": [
    "To clarify the feature types let's find the quantity of their unique values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "id": "277283e3-79e9-453d-aac2-08427808e319",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_nunique = df.drop(['id', 'buy_time'], axis=1).apply(lambda x: x.nunique())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "30f73929-ea1e-4c82-850c-1d22ba8dd2b0",
   "metadata": {},
   "source": [
    "Drawing distribution plot:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "id": "1bd8f1cc-e133-452e-aa1b-0882985517ad",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAlYAAAFMCAYAAAAJAPJ/AAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjUuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/YYfK9AAAACXBIWXMAAAsTAAALEwEAmpwYAAAdi0lEQVR4nO3de7zldV3v8dcbQTLGuDi4w2FyOoqZSt7mINWphkMZKIWdYxw6pIDQaHmpk1p415KyjlSKmY1pYKCAeWGOUkrk9lLhBVMQOCTZKDPc5DYwiJeBT3/8fhvW7Fl79po93733mpnX8/GYx6z1+/4u39/v+/vt9V7f72+tlapCkiRJO26Pxa6AJEnSrsJgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJQyVZnWTfJMuS/Mpi10faGRistKCSrEtyT5JNSW5KclaSJYtdL0lD7QVcA1wKfHeR6yLtFOIXhGohJVkHnFpV/5BkGfAx4CNVddri1kySpB1nj5UWTVVtAP4OeAJAkpOTXJ3kriRfS/L8wfmTHJvkS0nuTPLvSY7qp08m+XbfC7ap7xFbN7DcuiSvSHJVktuT/HWS7xsoP6Zf7x1J/jnJj03b7jlJvjuw7vUDZXsneXOSb/Q9cO9I8pCB8hVJaqBu9yY5tS/bI8lp/b7cmuSCJAdMW27PafV4ff941bR6HNfPf+rAtOf1x/P2JB9L8shh7ZDko0lePG3a5Ul+qX9cSR49UPbGJGcNq2eSw/rnb5yhnn+c5JNTx39g/+/q2+eXptVjVZL7Bo7ffUl+ti87a2o7Q/bp9UnOGXi+Z1+vFdtaNsmpSSb7xz+R5JYky/vnT+yP5WOHLLdPkiv6c/PWJGsGjslsdXlmkn/tl71uqo37spOSfGYbzx+b5OIktyW5JslxA2Vb7GOSRyepgeeT087FK6a11bqBY72kP7/v3/a0/Z9+nm9K8r3p52uSV/bHdF2SE7ZR14umnVfn9sd+Y5KpN2VbnV/9tM8kOal//Kgk/9i3yS39evabYR8PS3JDksP653sn+bMk1/f//izJ3gPbnTov70ryuSRPGHZstPsxWGnR9C9YzwD+tZ90M3AM8APAycCfJnlKP+9hwHuAlwP7AT8NrBtY3YuqaklVLQF+YcjmTgB+HngU8Bjg1f16nwy8G3g+8DDgL4G1U39Ap6oKnN6v++hp631Tv74nAY8GlgGvHSifusb27Zf/9EDZi4FnAT8DPAK4HfjzIXXfpiR7Ab8P3DAw7VjglcD/AA7st/u+GVZxNvCrA8s+sd+Pj25vXYD/C2yYoZ6/C/ws8AtV9e1+8r8DPwXsC7wBOCfJQQOL7QFsGGjbb8yhTnNSVf9Mdz6cnS4snwO8pqr+/5DZvwMcT3duPhb4cbY+V2ZyN/DcftlnAr+e5Fl92X3M8Hc6yT7AxcB7gYf32397kseNuN1BJwL7b6P85cD3RljPfgNtdf60sh8EltKdWycCa5L8yPQVJDkC+LFpk/8AmKDbzw3Ab49QF+iu3T+ku75+FFgOvH7INn8U+BDwq1X1uX7yq4DD6a7tJwKH0f/d6F3f7+d+wJeHrVe7J4OVFsOHk9wBfAb4JN0fTarqo1X179X5JPBxuhddgFOAd1fVxVV1X1VtmOEFbiZvq6rrquo24HRg6kbc1cBfVtVnq+reqjqb7kXy8IFlH8KQ+0uSpF/+/1TVbVV1V78vxw/M9mDgvqq6d0idXgC8qqrWV9V36P4wPzsDvVQjej7wWeDfpq37D6vq6qra3NfrSRnea7UWeEySQ/rnzwHOr6rtuqcmyTF0L2T/MKTsVOBlwFFVdefU9Kp6f1Vd37fp+cBX6V7ApjyYxb235/V0oe9zdC/oQ4NvVW2uqiur6j66Y3A3W7bHjKpqsqqu6I/B5XQB+Gf64m8AP5rk4CGLHgOsq6q/7rf/r8AHgF8effcgXe/ha+nC+bDyH6S7/v5ke9Y7g9dU1Xf66/ujwHGDhf019cds+eaE/th+l+7YwgNvxrapqq7t/2Z8p6q+SbcPPzNttkfS/a15dVVdMjD9BOD3qurmftk30F0b0+0BPAi4dZQ6addnsNJieFZV7VdVj6yq36iqewCSHJ3k0n5Y4w663qyl/TLL6Xo35uq6gcdfp3sHC90f1ZemGwa8o9/u8oFy6N5pf3PIOg8Evh+4bGDZv++nTzmAridqmEcCHxpY9mrgXrp35lNuGSg/bvoKkjwU+B3gNUPW/ZaBZW+je1FaNn0dfe/R+cCvJtmDLnT+zbTZvjiwrpcN2ZcH0fUM/M6QsgP7+n2L7t3/YP2fmweGYe+gGxZeOjDLto4fwMv6ZW9K8sEkDxsoO25gvbds57IAVNX3gLP6ep1Rs9yU2m/rJmA9cOModUnytCSfSPLNJBvpQvHUMfgkXdt8uV/27QOLPhJ42rRz9wS683X6Pt4BfHGGav8m3Xl7zQzlrwPOpDuHdsTtVXX3wPPB63DKcXTH5x+nL5zkI8BdwJOBywaKHjHtGBw+sMxEkvOSbEhyJ12v4+D5Bd2+rQd+btr0R/R1nKm+j+i3dxdd7+SZW+2xdksGK42FfujtA8CbgYmq2g+4iAfeoV5HN4w3V8sHHv8QcP3Aek/vg97Uv++vqvf19dqL7kX1y0PWeQtwD/D4gWWnhvymPIaZey6uA46etu3v6+89m7J0qgy4YMg6Xg5cUFVfnzb9OuD509b9kH54a5iz6V6UjwS+VVX/Mq38KQP1ePOQ5U8ErqmqS4eU3Uv3wrOabvjnoQB979k7gRcBD+vX/RUeaHPY9vEDeHO/3H+hC7kvHyi7YKDO019MZ1uWvo7L6ILFXwNnTBsi3kq/vgPohodeOWJd3kvXa7i8qvYF3kF/DPre2xdU1dTx+Y2B5a4DPjmtjZdU1a9P38d+2acMqfIBdMf/DTPs0mPohtDfsq39HtH+/fDllMHrELpPIP4+8LvDFq6qY4B96Hq6zhooun7wGNB9gnHKHwAFHFpVP0A35D14fkE3fH0kcFiSXxxcL114nam+1/fbewhwGt3fL8lgpbHxYGBvup6hzUmOBp4+UP4u4OQkR6a70XZZhtxEvA0vTHJwupvDX8UD93+8E3hB32uQdDchP3PqxZ/uXq8bgS9MX2E/7PNOunvBHg7dC3GSn+8fL6frDfjwDHV6B3D61PBckgP7e6NG9dC+fqfPsO5XJHl8v+59k8w4RNQHqfuAM9i6t2oUrwJeMUPZbVV1VVV9DLiEbqgHuhfJou8NTHIy/QcZ+uePA57HzMdv0LfpesTm8jdt6LL9sNRZdOfeKXT3sM00XHbgwL1he9KFhHtG3P5D6Y7Rt/t7Cf/3iMt9hG4I9zlJ9ur//df+fqFR/Rbwrqq6cYbyV9MNh317hvLt9YYkD07yU3RDme8fKHsO8M/9cOj9+uv98X177EH3d2J7ju0mYGMfkrcKz8Cnq+pbdG389jxwc/v7gFf3bbuUbnjynOkL972Y9zI8vGs3ZLDSWOjvT3oJXa/M7XQvLmsHyj9Hf0M7sJFuiGTop9xm8F66+yi+Rjek+MZ+vV8Afg14W7/da4GTANJ9aukvgR8G7kqyie5TjI9I8o5+vb/bL3NpP9TwD8DUDbkfAyb7Og/zln4fP57kLrp32k/bjn36AeCtVbXVUFlVfQj4I+C8vl5fYfabqd8DHMqQF48RfKSqvjrCfL8NHJNkVVVdRRfk/oVu+OxQ4J/g/huzP053/9uwnropL0n3qbBvAN/H8N60uS77ErqbpV/Tv3ieTBfuf4qtHQx8sj9HrgT+g64nZBS/Afxefw68luE9k1vpr5mn093Tdz3dG4A/ogseo3oQ2z5mt9CdFy3cSHeNXQ+cC7ygtrxPcn+2HtKequPZdNf9jXTnyQtG3OYb6HrqNtL1dH1wphn7+74+zAPX6xvp3lBdDlxBN5Q6+EnSR6T/VCDdG4vnjVgn7eL8Hivt8jLw3VnbudxJwIqqev206QcDb6yqkxpVcSwkeS6wuqr+22LXRbuWJKuAc6pq2E340i7FHitpZncDdw6Zvpkdv5F3rCT5frqekzWLXRdJ2plt78e6pd1GVb1/huk3Mvr36Iy9/p6wD9INY753kasjSTs1hwIlSZIacShQkiSpEYOVJElSI2Nxj9XSpUtrxYoV876du+++m3322Wf2GbXgbJvxZvuMN9tnvNk+420u7XPZZZfdUlUHDisbi2C1YsUKvvCFrb5/sbnJyUlWrVo179vR9rNtxpvtM95sn/Fm+4y3ubRPkum/dnE/hwIlSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJamQsfitwoVyxYSMnnfbRLaate9MzF6k2kiRpV2OPlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqZFZg1WS5Uk+keSqJFcm+c1++gFJLk7y1f7//fvpSfLWJNcmuTzJU+Z7JyRJksbBKD1Wm4GXVtXjgMOBFyZ5HHAacElVHQJc0j8HOBo4pP+3GviL5rWWJEkaQ7MGq6q6oaq+2D++C7gaWAYcC5zdz3Y28Kz+8bHAe6pzKbBfkoNaV1ySJGncbNc9VklWAE8GPgtMVNUNfdGNwET/eBlw3cBi6/tpkiRJu7Q9R50xyRLgA8BvVdWdSe4vq6pKUtuz4SSr6YYKmZiYYHJycnsWn5OJh8BLD928xbSF2K5mt2nTJttijNk+4832GW+2z3hr3T4jBaske9GFqnOr6oP95JuSHFRVN/RDfTf30zcAywcWP7iftoWqWgOsAVi5cmWtWrVqbnuwHc4890LOuGLLXV53wvxvV7ObnJxkIc4BzY3tM95sn/Fm+4y31u0zyqcCA7wLuLqq/mSgaC1wYv/4RODCgenP7T8deDiwcWDIUJIkaZc1So/VTwLPAa5I8qV+2iuBNwEXJDkF+DpwXF92EfAM4FrgW8DJLSssSZI0rmYNVlX1GSAzFB85ZP4CXriD9ZIkSdrp+M3rkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRmYNVkneneTmJF8ZmPb6JBuSfKn/94yBslckuTbJNUl+fr4qLkmSNG5G6bE6CzhqyPQ/raon9f8uAkjyOOB44PH9Mm9P8qBWlZUkSRpnswarqvoUcNuI6zsWOK+qvlNV/wFcCxy2A/WTJEnaaezIPVYvSnJ5P1S4fz9tGXDdwDzr+2mSJEm7vFTV7DMlK4CPVNUT+ucTwC1AAb8PHFRVz0vyNuDSqjqnn+9dwN9V1d8OWedqYDXAxMTEU88777w2e7QNN9+2kZvu2XLaocv2nfftanabNm1iyZIli10NzcD2GW+2z3izfcbbXNrniCOOuKyqVg4r23Mulaiqm6YeJ3kn8JH+6QZg+cCsB/fThq1jDbAGYOXKlbVq1aq5VGW7nHnuhZxxxZa7vO6E+d+uZjc5OclCnAOaG9tnvNk+4832GW+t22dOQ4FJDhp4+kvA1CcG1wLHJ9k7yQ8DhwCf27EqSpIk7Rxm7bFK8j5gFbA0yXrgdcCqJE+iGwpcBzwfoKquTHIBcBWwGXhhVd07LzWXJEkaM7MGq6r6lSGT37WN+U8HTt+RSkmSJO2M/OZ1SZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRmYNVkneneTmJF8ZmHZAkouTfLX/f/9+epK8Ncm1SS5P8pT5rLwkSdI4GaXH6izgqGnTTgMuqapDgEv65wBHA4f0/1YDf9GmmpIkSeNv1mBVVZ8Cbps2+Vjg7P7x2cCzBqa/pzqXAvslOahRXSVJksbaXO+xmqiqG/rHNwIT/eNlwHUD863vp0mSJO3y9tzRFVRVJantXS7JarrhQiYmJpicnNzRqsxq4iHw0kM3bzFtIbar2W3atMm2GGO2z3izfcab7TPeWrfPXIPVTUkOqqob+qG+m/vpG4DlA/Md3E/bSlWtAdYArFy5slatWjXHqozuzHMv5IwrttzldSfM/3Y1u8nJSRbiHNDc2D7jzfYZb7bPeGvdPnMdClwLnNg/PhG4cGD6c/tPBx4ObBwYMpQkSdqlzdpjleR9wCpgaZL1wOuANwEXJDkF+DpwXD/7RcAzgGuBbwEnz0OdJUmSxtKswaqqfmWGoiOHzFvAC3e0UpIkSTsjv3ldkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUyJ47snCSdcBdwL3A5qpameQA4HxgBbAOOK6qbt+xakqSJI2/Fj1WR1TVk6pqZf/8NOCSqjoEuKR/LkmStMubj6HAY4Gz+8dnA8+ah21IkiSNnR0NVgV8PMllSVb30yaq6ob+8Y3AxA5uQ5IkaaeQqpr7wsmyqtqQ5OHAxcCLgbVVtd/APLdX1f5Dll0NrAaYmJh46nnnnTfneozq5ts2ctM9W047dNm+875dzW7Tpk0sWbJksauhGdg+4832GW+2z3ibS/scccQRlw3cArWFHbp5vao29P/fnORDwGHATUkOqqobkhwE3DzDsmuANQArV66sVatW7UhVRnLmuRdyxhVb7vK6E+Z/u5rd5OQkC3EOaG5sn/Fm+4w322e8tW6fOQ8FJtknyUOnHgNPB74CrAVO7Gc7EbhwRyspSZK0M9iRHqsJ4ENJptbz3qr6+ySfBy5IcgrwdeC4Ha+mJEnS+JtzsKqqrwFPHDL9VuDIHamUJEnSzshvXpckSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhoxWEmSJDVisJIkSWrEYCVJktSIwUqSJKkRg5UkSVIjBitJkqRGDFaSJEmNGKwkSZIaMVhJkiQ1YrCSJElqxGAlSZLUiMFKkiSpEYOVJElSIwYrSZKkRgxWkiRJjRisJEmSGjFYSZIkNbLnYldgsa047aNbTVv3pmcuQk0kSdLOzh4rSZKkRgxWkiRJjRisJEmSGjFYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiPz9gWhSY4C3gI8CPirqnrTfG1rsfjlopIkadC8BKskDwL+HPg5YD3w+SRrq+qq+dheawYmSZI0F/M1FHgYcG1Vfa2qvgucBxw7T9uSJEkaC/M1FLgMuG7g+XrgafO0rZ3SQvSKDdvGfGxH28ce0c5iXgPzvd3F3rZ2/etsZ9y/+XhNGsfjkKpqv9Lk2cBRVXVq//w5wNOq6kUD86wGVvdPfwS4pnlFtrYUuGUBtqPtZ9uMN9tnvNk+4832GW9zaZ9HVtWBwwrmq8dqA7B84PnB/bT7VdUaYM08bX+oJF+oqpULuU2NxrYZb7bPeLN9xpvtM95at8983WP1eeCQJD+c5MHA8cDaedqWJEnSWJiXHquq2pzkRcDH6L5u4d1VdeV8bEuSJGlczNv3WFXVRcBF87X+OVrQoUdtF9tmvNk+4832GW+2z3hr2j7zcvO6JEnS7siftJEkSWpklwtWSY5Kck2Sa5OcNqR87yTn9+WfTbJiEaq52xqhfU5K8s0kX+r/nboY9dwdJXl3kpuTfGWG8iR5a992lyd5ykLXcXc2QvusSrJx4Np57ULXcXeWZHmSTyS5KsmVSX5zyDxeQ4tkxPZpcg3N2z1Wi2HEn9I5Bbi9qh6d5Hjgj4D/tfC13f1sx08dnT/4nWdaMGcBbwPeM0P50cAh/b+nAX+BX/y7kM5i2+0D8OmqOmZhqqNpNgMvraovJnkocFmSi6f9ffMaWjyjtA80uIZ2tR6rUX5K51jg7P7x3wJHJskC1nF35k8djbGq+hRw2zZmORZ4T3UuBfZLctDC1E4jtI8WUVXdUFVf7B/fBVxN9yskg7yGFsmI7dPErhashv2UzvQDd/88VbUZ2Ag8bEFqp1HaB+B/9t3kf5tk+ZByLY5R20+L58eTfDnJ3yV5/GJXZnfV32LyZOCz04q8hsbANtoHGlxDu1qw0s7v/wErqurHgIt5oHdR0rZ9ke5nNp4InAl8eHGrs3tKsgT4APBbVXXnYtdHW5qlfZpcQ7tasJr1p3QG50myJ7AvcOuC1E6j/NTRrVX1nf7pXwFPXaC6aXajXF9aJFV1Z1Vt6h9fBOyVZOkiV2u3kmQvuhftc6vqg0Nm8RpaRLO1T6traFcLVqP8lM5a4MT+8bOBfyy/zGuhzNo+0+43+EW6cXCNh7XAc/tPNh0ObKyqGxa7Uuok+cGp+0WTHEb39903jQukP/bvAq6uqj+ZYTavoUUySvu0uoZ2qU8FzvRTOkl+D/hCVa2lO7B/k+RauhtBj1+8Gu9eRmyflyT5RbpPcNwGnLRoFd7NJHkfsApYmmQ98DpgL4CqegfdLyk8A7gW+BZw8uLUdPc0Qvs8G/j1JJuBe4DjfdO4oH4SeA5wRZIv9dNeCfwQeA2NgVHap8k15DevS5IkNbKrDQVKkiQtGoOVJElSIwYrSZKkRgxWkiRJjRisJEnSbmG2HzMfMv9xAz/c/N6RlvFTgZIkaXeQ5KeBTXS/2fiEWeY9BLgA+O9VdXuSh1fVzbNtwx4rSZK0Wxj2Y+ZJHpXk75NcluTTSR7bF/0a8OdVdXu/7KyhCgxWkiRp97YGeHFVPRV4GfD2fvpjgMck+acklyY5apSV7VLfvC5JkjSq/keZfwJ4f/9rNgB79//vCRxC94sHBwOfSnJoVd2xrXUarCRJ0u5qD+COqnrSkLL1wGer6nvAfyT5N7qg9fnZVihJkrTbqao76ULTL0P3Y81JntgXf5iut4okS+mGBr822zoNVpIkabfQ/5j5vwA/kmR9klOAE4BTknwZuBI4tp/9Y8CtSa4CPgG8vKpunXUbft2CJElSG/ZYSZIkNWKwkiRJasRgJUmS1IjBSpIkqRGDlSRJUiMGK0mSpEYMVpIkSY0YrCRJkhr5TwfZHHlG27GZAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 720x360 with 1 Axes>"
      ]
     },
     "metadata": {
      "needs_background": "light"
     },
     "output_type": "display_data"
    }
   ],
   "source": [
    "plt.title(\"Features' unique values distribution\")\n",
    "X_nunique.hist(bins=100, figsize=(10, 5))\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0212c2fe-c5a9-4179-b1be-56670660019d",
   "metadata": {},
   "source": [
    "From the plot we may conclude that the vast majority of features have relatively  \n",
    "small amount of unique values. In fact, we already had an opportunity to see this  \n",
    "when we looked at the value distributions of randomly picked features.  \n",
    "  \n",
    "We'll split the features into categories for future individual work with them."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c879360-e9ae-4b69-a2c2-858025cce630",
   "metadata": {},
   "source": [
    "Total of numeric features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 49,
   "id": "8b1473e9-60ca-499e-8bc6-97acf6280b9a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "253"
      ]
     },
     "execution_count": 49,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f_all = set(X_nunique.index.tolist())\n",
    "len(f_all)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c32b1d9-8e50-4139-8d0c-04162587f617",
   "metadata": {},
   "source": [
    "#### 5.1 Constant features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "f9882b46-8899-429b-a201-8a2257a5c9c9",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "146"
      ]
     },
     "execution_count": 50,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f_const = set(X_nunique[X_nunique == 1].index.tolist())\n",
    "len(f_const)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c3d460cb-a658-408b-9240-54d476cbf0e6",
   "metadata": {},
   "source": [
    "More than a half of the features have just one value after the processing.  \n",
    "We will not use them in the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "c739d60a-6ba6-4cfe-a792-e51318966b9a",
   "metadata": {},
   "outputs": [],
   "source": [
    "f_other = f_all - f_const"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c36046b1-ce77-429c-ae3e-71823c786186",
   "metadata": {},
   "source": [
    "#### 5.2 Binary features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "5d820a4a-f39e-43aa-9f45-dc9f5603f124",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "6"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f_binary = set(X_nunique[X_nunique == 2].index.tolist())\n",
    "len(f_binary)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "91eea6b9-31b3-44da-9916-dee2cfac67af",
   "metadata": {},
   "outputs": [],
   "source": [
    "f_other = f_other - f_binary"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fd7e787d-6393-4a78-9bd2-0091054829e4",
   "metadata": {},
   "source": [
    "#### 5.3 Categorical features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c1d129f6-d054-478e-a93c-c5049f156d19",
   "metadata": {},
   "source": [
    "For categorical features we'll count features with no more than 20 unique  \n",
    "values. And the *vas_id* attribute will fit here as well:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 54,
   "id": "86a3f742-a72c-496e-a545-0fb6b7e1fcec",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "23"
      ]
     },
     "execution_count": 54,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "f_categorical = set(X_nunique.loc[list(f_other)][X_nunique[list(f_other)] <= 20].index.tolist())\n",
    "f_categorical.add('vas_id')\n",
    "len(f_categorical)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b0829c-2bbf-4a60-bbed-e7becb6e504e",
   "metadata": {},
   "source": [
    "#### 5.4 Continuous features"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9cfb5877-e465-4f2f-a47d-c34153be5e3d",
   "metadata": {},
   "source": [
    "All other features we'll consider continuous. We will not include customer's  \n",
    "profile creation and service proposal dates here, they'll only be used for splitting  \n",
    "and merging the datasets."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 55,
   "id": "672ab791-e5a0-436f-b31e-3f4448393a3a",
   "metadata": {},
   "outputs": [],
   "source": [
    "f_numeric = f_other - f_categorical"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d28c82c-fb1d-4e46-88f2-467b78cb6ff1",
   "metadata": {},
   "source": [
    "We have the following picture:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "id": "0df9f05b-6900-49e2-9353-2867e7b0767b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "f_const:\t146\n",
      "f_binary:\t6\n",
      "f_categorical:\t23\n",
      "f_numeric:\t79\n"
     ]
    }
   ],
   "source": [
    "print(f'f_const:\\t{len(f_const)}\\n'\n",
    "      f'f_binary:\\t{len(f_binary)}\\n'\n",
    "      f'f_categorical:\\t{len(f_categorical)}\\n'\n",
    "      f'f_numeric:\\t{len(f_numeric)}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e7a67836-a630-4dc5-b089-07b54c6ba326",
   "metadata": {},
   "source": [
    "The number of features that will be used in training:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 57,
   "id": "c6cf527f-d45b-4d39-bd31-b3335bfd9a45",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "108"
      ]
     },
     "execution_count": 57,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(f_binary) + len(f_categorical) + len(f_numeric)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ef3b4d03-af54-45e8-abb3-aaf36c4ff52f",
   "metadata": {},
   "source": [
    "## 6. Model selection"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c717d728-22ce-435c-a97d-a063d17994a4",
   "metadata": {},
   "source": [
    "Let's write a function for datasets merging. When we encounter duplicates,  \n",
    "we will take rows by the closest customer's profile creation date to service  \n",
    "proposal date in the features dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 58,
   "id": "6f4579e7-dd6e-43d2-a704-9102a55f4709",
   "metadata": {},
   "outputs": [],
   "source": [
    "def merge_datasets(df_features, df_labels):\n",
    "    df_labels = df_labels.merge(df_features, how='left', on='id')\n",
    "    df_labels['diff'] = np.abs(df_labels['buy_time_x'] - df_labels['buy_time_y'])\n",
    "    return df_labels.sort_values('diff').drop_duplicates(['id', 'vas_id', 'buy_time_x'])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4f8d4a5-89b8-405e-89ad-221cbcdeb77b",
   "metadata": {},
   "source": [
    "We'll write a custom class for the pipelines, which selects attributes according  \n",
    "to our categories. At first, an attempt to conduct target-encoding of *vas_id*  \n",
    "feature was made, because it's categorical. However, after processing it like  \n",
    "other categorical features (one-hot-encoding) some of its values got pretty low  \n",
    "importance from the models and it ended up being less effective by target metric.  \n",
    "In the end, it was decided to leave one-hot-encoding. Code that was used initially  \n",
    "is commented:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "9474700a-4208-425f-a340-b92310523117",
   "metadata": {},
   "outputs": [],
   "source": [
    "class ColumnSelector(BaseEstimator, TransformerMixin):\n",
    "    def __init__(self, columns):\n",
    "        self.columns = columns\n",
    "    \n",
    "    def fit(self, X, y=None):\n",
    "        return self\n",
    "    \n",
    "    def transform(self, X):\n",
    "        return X[self.columns]\n",
    "\n",
    "# class TargetEncoder(BaseEstimator, TransformerMixin):\n",
    "#     def __init__(self):\n",
    "#         self.encoder = None\n",
    "        \n",
    "#     def fit(self, X, y):\n",
    "          \n",
    "#         # Taking subsample of size 20% of training dataset to decrease\n",
    "#         # potential overfitting\n",
    "#         X_encode = X.sample(frac=0.2, random_state=RANDOM_STATE)\n",
    "#         y_encode = y[X_encode.index]\n",
    "#         self.encoder = MEstimateEncoder(cols=X.columns)\n",
    "#         self.encoder.fit(X_encode, y_encode)\n",
    "#         return self\n",
    "    \n",
    "#     def transform(self, X):\n",
    "#         return self.encoder.transform(X)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d28c3e43-3360-4634-9702-ad2dac1c32fe",
   "metadata": {},
   "source": [
    "Declaring pipelines:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "574cf085-ef7b-48eb-83f8-d0131fe50988",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Scale binary features, because not all of them have values 0 and 1\n",
    "binary_features = Pipeline([\n",
    "    ('selector', ColumnSelector(columns=list(f_binary))),\n",
    "    ('scaler', StandardScaler())\n",
    "])\n",
    "\n",
    "# One-hot-encode categorical features\n",
    "cat_features = Pipeline([\n",
    "    ('selector', ColumnSelector(columns=list(f_categorical))),\n",
    "    ('encoder', OneHotEncoder(handle_unknown='ignore'))\n",
    "])\n",
    "\n",
    "# Scale continuous features\n",
    "numeric_features = Pipeline([\n",
    "    ('selector', ColumnSelector(columns=list(f_numeric))),\n",
    "    ('scaler', StandardScaler())\n",
    "])\n",
    "\n",
    "# # Target-encode vas_id feature\n",
    "# vas_id_feature = Pipeline([\n",
    "#     ('selector', ColumnSelector(columns=['vas_id'])),\n",
    "#     ('encoder', TargetEncoder())\n",
    "# ])\n",
    "\n",
    "# Collect all features with Feature Union\n",
    "feats = FeatureUnion([\n",
    "    ('binary', binary_features),\n",
    "    ('categorical', cat_features),\n",
    "    ('numeric', numeric_features),\n",
    "    # ('vas_id', vas_id_feature)\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "612339ed-31cd-470e-b7ac-d28011aad567",
   "metadata": {},
   "source": [
    "Loading training set once again because earlier we merged it with features,  \n",
    "not yet cleaned from outliers:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 61,
   "id": "ec14bcf1-f252-40c8-9bd2-4388631d5663",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = pd.read_csv('data_train.csv', index_col='Unnamed: 0', dtype={'id': str})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "78251bac-7c48-4a1e-be3f-85c3762e6590",
   "metadata": {},
   "source": [
    "Counting class disbalance to use it while training models, which need it to be  \n",
    "clearly specified:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "a7f61b78-06f0-45af-aad3-51c274c9dfb6",
   "metadata": {},
   "outputs": [],
   "source": [
    "disbalance = (df_train['target'] == 0).sum() / (df_train['target'] == 1).sum()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7dd1e08b-baae-4b01-8579-e20e7da5bf8a",
   "metadata": {},
   "source": [
    "Declaring models. We'll use default sets of parameters, the main goal now is  \n",
    "to choose an optimal model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "79097577-8a9f-4e2f-9ee5-94f3de9aa397",
   "metadata": {},
   "outputs": [],
   "source": [
    "rf_model = Pipeline([\n",
    "    ('features', feats),\n",
    "    ('classifier', RandomForestClassifier(n_jobs=-1, random_state=RANDOM_STATE))\n",
    "])\n",
    "\n",
    "cb_model = Pipeline([\n",
    "    ('features', feats),\n",
    "    ('classifier', CatBoostClassifier(iterations=200, silent=True,\n",
    "                                      eta=0.1, task_type='GPU',\n",
    "                                      auto_class_weights='Balanced', random_state=RANDOM_STATE))\n",
    "])\n",
    "\n",
    "xgb_model = Pipeline([\n",
    "    ('features', feats),\n",
    "    ('classifier', XGBClassifier(n_estimators=200,\n",
    "                                 eta=0.1, tree_method='gpu_hist',\n",
    "                                 scale_pos_weight=disbalance,\n",
    "                                 predictor='gpu_predictor',\n",
    "                                 random_state=RANDOM_STATE))\n",
    "])\n",
    "\n",
    "lr_model = Pipeline([\n",
    "    ('features', feats),\n",
    "    ('classifier', LogisticRegression(max_iter=500, C=10, class_weight='balanced'))\n",
    "])\n",
    "\n",
    "svc_model = Pipeline([\n",
    "    ('features', feats),\n",
    "    ('classifier', LinearSVC(max_iter=500, C=10, class_weight='balanced', dual=False))\n",
    "])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "bd1ed732-c4e0-49b6-9287-5eda1ad74c61",
   "metadata": {},
   "source": [
    "Merging training dataset with features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 64,
   "id": "be13c491-fdaa-454a-94e8-53a9d92ffe54",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = merge_datasets(df, df_train)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7bd52c10-0ab8-42d5-9657-820e456b25cf",
   "metadata": {},
   "source": [
    "Writing a function, which helps to collect models' metrics into one table  \n",
    "for more convenient comparison:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "id": "4ca88f41-1cf2-4369-9ed7-8452147f40f0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_metrics(model, X_train, y_train, X_test, y_test, verbose=False, new_data=False):\n",
    "    \n",
    "    # In case of repeated training, we will load random forest  \n",
    "    # model from a file, because it takes time to train it.\n",
    "    # File with the model won't be added to the project because it \n",
    "    # occupies almost 500 Mb\n",
    "    if (isinstance(model['classifier'], RandomForestClassifier)) and (not new_data):\n",
    "        with open('./models/rforest.pkl', 'rb') as f:\n",
    "            model = pickle.load(f)\n",
    "    else:                \n",
    "        model.fit(X_train, y_train)\n",
    "    \n",
    "    if isinstance(model['classifier'], RandomForestClassifier) and new_data:\n",
    "        with open('./models/rforest.pkl', 'wb') as f:\n",
    "            pickle.dump(model, f)\n",
    "    \n",
    "    # Handling situation when SVC model doesn't have\n",
    "    # predict_proba method\n",
    "    try:\n",
    "        y_pred = model.predict_proba(X_test)[:, 1]\n",
    "    except AttributeError:\n",
    "        y_pred = model.decision_function(X_test)\n",
    "    \n",
    "    # Finding classification threshold with the function, \n",
    "    # written earlier\n",
    "    threshold = find_best_threshold(y_test, y_pred, verbose=False)\n",
    "    \n",
    "    # Printing metrics if needed\n",
    "    if verbose:\n",
    "        print(classification_report(y_test, y_pred >= threshold))\n",
    "        \n",
    "    return pd.DataFrame(index=re.findall('(\\w*?)[\\(\\s]', str(model['classifier']))[:1],\n",
    "                        columns=('Precision', 'Recall', 'F1-score', 'F1-macro'),\n",
    "                        data=np.round(np.array([precision_score(y_test, y_pred >= threshold),\n",
    "                                                recall_score(y_test, y_pred >= threshold),\n",
    "                                                f1_score(y_test, y_pred >= threshold),\n",
    "                                                f1_score(y_test, y_pred >= threshold, average='macro')]).reshape(1, -1), 3))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b0039bbd-fad5-45fd-96d9-6f9705804251",
   "metadata": {},
   "source": [
    "We'll make a split of the training dataset on training and validation parts.  \n",
    "Because, as we saw before, the test dataset has only January 2019 as service  \n",
    "proposal date, the split will be made getting the validation dataset as close  \n",
    "as possible to the test dataset as far as dates, which means December 2018.  \n",
    "Other months will go to the training."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "9cb555f8-1954-4fb5-8ec1-1d0f80aac94a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train['month'] = df_train['buy_time_x'].apply(lambda x: datetime.datetime.fromtimestamp(x)).apply(lambda x: x.month)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "5a5d7036-ddd2-4afe-81a2-dc59610f074b",
   "metadata": {},
   "outputs": [],
   "source": [
    "X_train = df_train.loc[df_train['month'] != 12]\n",
    "y_train = X_train['target']\n",
    "\n",
    "X_test = df_train.loc[df_train['month'] == 12]\n",
    "y_test = X_test['target']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "dce39904-89a1-4421-8da7-cb1c40f2ab8e",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.29769627476844307"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "X_test.shape[0] / df_train.shape[0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "699481b3-172b-4f82-81c7-ff5bc86a9846",
   "metadata": {},
   "source": [
    "We got a nice samples' sizes ratio."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6c893d76-4485-4ed5-abfa-50456483edf2",
   "metadata": {},
   "source": [
    "Comparing models' metrics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "7e265545-a014-4fc5-a4f8-aa1d1f3325ae",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 7h 32min 52s\n",
      "Wall time: 42min 10s\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1-score</th>\n",
       "      <th>F1-macro</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>LogisticRegression</th>\n",
       "      <td>0.424</td>\n",
       "      <td>0.881</td>\n",
       "      <td>0.573</td>\n",
       "      <td>0.752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>CatBoostClassifier</th>\n",
       "      <td>0.424</td>\n",
       "      <td>0.884</td>\n",
       "      <td>0.573</td>\n",
       "      <td>0.752</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>LinearSVC</th>\n",
       "      <td>0.416</td>\n",
       "      <td>0.900</td>\n",
       "      <td>0.569</td>\n",
       "      <td>0.749</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>RandomForestClassifier</th>\n",
       "      <td>0.405</td>\n",
       "      <td>0.910</td>\n",
       "      <td>0.561</td>\n",
       "      <td>0.743</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>XGBClassifier</th>\n",
       "      <td>0.396</td>\n",
       "      <td>0.959</td>\n",
       "      <td>0.560</td>\n",
       "      <td>0.740</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                        Precision  Recall  F1-score  F1-macro\n",
       "LogisticRegression          0.424   0.881     0.573     0.752\n",
       "CatBoostClassifier          0.424   0.884     0.573     0.752\n",
       "LinearSVC                   0.416   0.900     0.569     0.749\n",
       "RandomForestClassifier      0.405   0.910     0.561     0.743\n",
       "XGBClassifier               0.396   0.959     0.560     0.740"
      ]
     },
     "execution_count": 69,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "pd.concat((get_metrics(lr_model, X_train, y_train, X_test, y_test),\n",
    "           get_metrics(cb_model, X_train, y_train, X_test, y_test),\n",
    "           get_metrics(xgb_model, X_train, y_train, X_test, y_test),\n",
    "           get_metrics(svc_model, X_train, y_train, X_test, y_test),\n",
    "           get_metrics(rf_model, X_train, y_train, X_test, y_test, new_data=True)),\n",
    "           axis=0).sort_values('F1-macro', ascending=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "279c1d9d-5f14-46a7-80b7-55442c98b3a0",
   "metadata": {},
   "source": [
    "The best models are CatBoost and both linear models. Comparing training  \n",
    "time of these models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "050ba2a7-2dbe-4cfa-a78a-af3004020a6c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 20.8 s\n",
      "Wall time: 8.63 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "cb_model.fit(X_train, y_train)\n",
    "print(end='')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "d2bcfbac-81f6-474d-95b5-e174219e2b21",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 1min\n",
      "Wall time: 59.9 s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "lr_model.fit(X_train, y_train)\n",
    "print(end='')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "83185b69-b38c-4092-99c9-ae5b7a37e5a8",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "CPU times: total: 1min 3s\n",
      "Wall time: 1min 3s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "svc_model.fit(X_train, y_train)\n",
    "print(end='')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c62662ff-9fd3-4032-9105-926192eb2c1a",
   "metadata": {},
   "source": [
    "Because we are to search for model's hyperparameters, we'll choose CatBoost  \n",
    "for much lower training time."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dcf48183-51fc-4362-ad3a-0a56c4f6663e",
   "metadata": {},
   "source": [
    "## 7. Tuning model hyperparameters"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9c8fe8fe-cc70-4e41-937e-0d44a105782e",
   "metadata": {},
   "source": [
    "Let's write a function for model hyperparameters tuning. It was decided to reject  \n",
    "existing solution with cross-validation, because the notebook kernel froze trying  \n",
    "to process it. Probably, it's because of the training dataset size."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c5c4ed49-a9a9-4b3a-af3a-bc3a10c2c30b",
   "metadata": {},
   "source": [
    "Because CatBoost allows to change model hyperparameters only when it's not fitted,  \n",
    "we'll initialize it again and save the untrained model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "fd46f2e7-8749-48fc-b5ae-18ca4bc1e056",
   "metadata": {},
   "outputs": [],
   "source": [
    "cb_model = Pipeline([\n",
    "    ('features', feats),\n",
    "    ('classifier', CatBoostClassifier(iterations=200, silent=True,\n",
    "                                      eta=0.1, task_type='GPU',\n",
    "                                      auto_class_weights='Balanced', random_state=RANDOM_STATE))\n",
    "])\n",
    "\n",
    "with open('./models/cb_model.pkl', 'wb') as f:\n",
    "    pickle.dump(cb_model, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4f0d1df8-dc49-48d0-8555-d31d7d12a1fa",
   "metadata": {},
   "source": [
    "Writing its own function for hyperparameters changing:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "cfcf89d5-7394-4e65-bdf7-3149b1d33127",
   "metadata": {},
   "outputs": [],
   "source": [
    "def cb_set_params(params):\n",
    "    with open('./models/cb_model.pkl', 'rb') as f:\n",
    "        model = pickle.load(f)\n",
    "    model['classifier'].set_params(**params)\n",
    "    return model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "345eadc4-97d3-4645-a295-c1b54404372c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def find_best_params(model, params, X_train, X_test, y_train, y_test, verbose=False):\n",
    "    \n",
    "    parameters = list(params.keys())\n",
    "    \n",
    "    # Using itertools module to create all possible combinations\n",
    "    # of hyperparameters\n",
    "    values = list(product(*params.values()))\n",
    "    \n",
    "    best_score = 0\n",
    "    scores = []\n",
    "    for params_set in values:\n",
    "        \n",
    "        # Transforming set of hyperparameters into a dictionary with\n",
    "        # names and values\n",
    "        current_params = {param[0]: param[1] for param in zip(parameters, params_set)}\n",
    "        \n",
    "        # Setting model hyperparameters, training it, finding an optimal\n",
    "        # threshold and saving the metrics\n",
    "        if isinstance(model['classifier'], CatBoostClassifier):\n",
    "            model = cb_set_params(current_params)\n",
    "        else:\n",
    "            model['classifier'].set_params(**current_params)\n",
    "        model.fit(X_train, y_train)\n",
    "        y_pred = model.predict_proba(X_test)[:, 1]\n",
    "        threshold = find_best_threshold(y_test, y_pred, verbose=False)\n",
    "        score = f1_score(y_test, y_pred >= threshold, average='macro')\n",
    "        scores.append((current_params, score))\n",
    "        \n",
    "        # Possibility to print function progress with hyperparameters\n",
    "        # and their metrics\n",
    "        if verbose:\n",
    "            print(f'{current_params}\\t\\t{score:.3f}')\n",
    "        \n",
    "        # Saving the best metrics and hyperparameters\n",
    "        if score > best_score:\n",
    "            best_score = score\n",
    "            best_params = current_params\n",
    "        \n",
    "        # We'll save all sets of hyperparameters which are close to the best score.\n",
    "        # We'll test them on stability later\n",
    "        top_params = [top_score[0] for top_score in scores if np.isclose(top_score[1], best_score, atol=1e-3)]\n",
    "        \n",
    "    return best_score, best_params, top_params"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0728a399-1e3c-44b9-a9c5-039b9fea8be0",
   "metadata": {},
   "source": [
    "Writing a function to check model's stability, i.e. we'll train it with  \n",
    "different dataset splits and look at the value and the spread of its metric:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "138ef85f-568a-4a13-b643-d4bedf4062c4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def check_stability(model, df):\n",
    "    seeds = (0, 29, 42, 77, 100)\n",
    "    scores = np.array([])\n",
    "    for seed in seeds:\n",
    "        X_train, X_test, y_train, y_test = train_test_split(df, df['target'],\n",
    "                                                            test_size=0.3, random_state=seed)\n",
    "        model.fit(X_train, y_train)\n",
    "        y_pred = model.predict_proba(X_test)[:, 1]\n",
    "        threshold = find_best_threshold(y_test, y_pred, verbose=False)\n",
    "        scores = np.append(scores, f1_score(y_test, y_pred >= threshold, average='macro'))\n",
    "    print(scores)\n",
    "    print(f'F1-macro = {scores.mean():.3f} +/- {scores.std(ddof=1) * 2:.3f}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1c001f6b-1216-463c-989c-5ae5a09bf110",
   "metadata": {},
   "source": [
    "Conducting search of CatBoost model hyperparameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "ad5b24a1-0593-4eb9-adbb-c7687bcf3849",
   "metadata": {},
   "outputs": [],
   "source": [
    "params = {'iterations': [100, 200, 300],\n",
    "          'eta': [0.1, 0.15, 0.2],\n",
    "          'depth': [4, 5, 6]\n",
    "          }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "9ebd92c1-4e6f-4282-b23e-04ae7cd18051",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'iterations': 100, 'eta': 0.1, 'depth': 4}\t\t0.753\n",
      "{'iterations': 100, 'eta': 0.1, 'depth': 5}\t\t0.752\n",
      "{'iterations': 100, 'eta': 0.1, 'depth': 6}\t\t0.753\n",
      "{'iterations': 100, 'eta': 0.15, 'depth': 4}\t\t0.753\n",
      "{'iterations': 100, 'eta': 0.15, 'depth': 5}\t\t0.752\n",
      "{'iterations': 100, 'eta': 0.15, 'depth': 6}\t\t0.752\n",
      "{'iterations': 100, 'eta': 0.2, 'depth': 4}\t\t0.752\n",
      "{'iterations': 100, 'eta': 0.2, 'depth': 5}\t\t0.752\n",
      "{'iterations': 100, 'eta': 0.2, 'depth': 6}\t\t0.752\n",
      "{'iterations': 200, 'eta': 0.1, 'depth': 4}\t\t0.753\n",
      "{'iterations': 200, 'eta': 0.1, 'depth': 5}\t\t0.752\n",
      "{'iterations': 200, 'eta': 0.1, 'depth': 6}\t\t0.752\n",
      "{'iterations': 200, 'eta': 0.15, 'depth': 4}\t\t0.752\n",
      "{'iterations': 200, 'eta': 0.15, 'depth': 5}\t\t0.752\n",
      "{'iterations': 200, 'eta': 0.15, 'depth': 6}\t\t0.751\n",
      "{'iterations': 200, 'eta': 0.2, 'depth': 4}\t\t0.752\n",
      "{'iterations': 200, 'eta': 0.2, 'depth': 5}\t\t0.751\n",
      "{'iterations': 200, 'eta': 0.2, 'depth': 6}\t\t0.750\n",
      "{'iterations': 300, 'eta': 0.1, 'depth': 4}\t\t0.752\n",
      "{'iterations': 300, 'eta': 0.1, 'depth': 5}\t\t0.752\n",
      "{'iterations': 300, 'eta': 0.1, 'depth': 6}\t\t0.751\n",
      "{'iterations': 300, 'eta': 0.15, 'depth': 4}\t\t0.752\n",
      "{'iterations': 300, 'eta': 0.15, 'depth': 5}\t\t0.751\n",
      "{'iterations': 300, 'eta': 0.15, 'depth': 6}\t\t0.749\n",
      "{'iterations': 300, 'eta': 0.2, 'depth': 4}\t\t0.752\n",
      "{'iterations': 300, 'eta': 0.2, 'depth': 5}\t\t0.750\n",
      "{'iterations': 300, 'eta': 0.2, 'depth': 6}\t\t0.746\n",
      "CPU times: total: 11min 11s\n",
      "Wall time: 4min 21s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "cb_score, cb_params, cb_tops = find_best_params(cb_model, params, X_train, X_test, y_train, y_test, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f6cb828-5395-4394-8730-64a7f23928ed",
   "metadata": {},
   "source": [
    "The maximum F1-score macro was achieved with the following combination of  \n",
    "hyperparameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "993e8d72-6167-4498-9407-53f11ce49e6b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'iterations': 100, 'eta': 0.1, 'depth': 4}"
      ]
     },
     "execution_count": 79,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cb_params"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e9a68ff6-57e1-413b-ba38-79818074d280",
   "metadata": {},
   "source": [
    "The best hyperparameters for stability check:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "2483d453-6b2c-437d-8c6a-8d5e0fcda5b7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'iterations': 100, 'eta': 0.1, 'depth': 4},\n",
       " {'iterations': 100, 'eta': 0.1, 'depth': 5},\n",
       " {'iterations': 100, 'eta': 0.1, 'depth': 6},\n",
       " {'iterations': 100, 'eta': 0.15, 'depth': 4},\n",
       " {'iterations': 100, 'eta': 0.15, 'depth': 5},\n",
       " {'iterations': 100, 'eta': 0.2, 'depth': 4},\n",
       " {'iterations': 200, 'eta': 0.1, 'depth': 4},\n",
       " {'iterations': 200, 'eta': 0.1, 'depth': 5},\n",
       " {'iterations': 200, 'eta': 0.15, 'depth': 4},\n",
       " {'iterations': 200, 'eta': 0.2, 'depth': 4},\n",
       " {'iterations': 300, 'eta': 0.1, 'depth': 4},\n",
       " {'iterations': 300, 'eta': 0.1, 'depth': 5},\n",
       " {'iterations': 300, 'eta': 0.15, 'depth': 4}]"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cb_tops"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "652d2116-62e2-4a0f-ab92-b65eb838a355",
   "metadata": {},
   "source": [
    "Let's assess the metrics stability of these sets of hyperparameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "cb1c84b8-8c3d-466b-bdc1-c13e75b3a916",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'iterations': 100, 'eta': 0.1, 'depth': 4}\n",
      "[0.69022583 0.69280399 0.69152844 0.69080213 0.68966583]\n",
      "F1-macro = 0.691 +/- 0.002\n",
      "\n",
      "{'iterations': 100, 'eta': 0.1, 'depth': 5}\n",
      "[0.68995405 0.69327948 0.69278981 0.69335497 0.6888734 ]\n",
      "F1-macro = 0.692 +/- 0.004\n",
      "\n",
      "{'iterations': 100, 'eta': 0.1, 'depth': 6}\n",
      "[0.68945777 0.69156074 0.69269797 0.6908133  0.68917551]\n",
      "F1-macro = 0.691 +/- 0.003\n",
      "\n",
      "{'iterations': 100, 'eta': 0.15, 'depth': 4}\n",
      "[0.69086909 0.69134426 0.69315481 0.69144504 0.68934098]\n",
      "F1-macro = 0.691 +/- 0.003\n",
      "\n",
      "{'iterations': 100, 'eta': 0.15, 'depth': 5}\n",
      "[0.69039813 0.68936167 0.6908397  0.69063467 0.6887364 ]\n",
      "F1-macro = 0.690 +/- 0.002\n",
      "\n",
      "{'iterations': 100, 'eta': 0.2, 'depth': 4}\n",
      "[0.68992715 0.69237699 0.69066914 0.6909324  0.68893213]\n",
      "F1-macro = 0.691 +/- 0.003\n",
      "\n",
      "{'iterations': 200, 'eta': 0.1, 'depth': 4}\n",
      "[0.6900214  0.69175593 0.69048336 0.69074985 0.68966343]\n",
      "F1-macro = 0.691 +/- 0.002\n",
      "\n",
      "{'iterations': 200, 'eta': 0.1, 'depth': 5}\n",
      "[0.68807063 0.69178913 0.69210924 0.69035734 0.68931007]\n",
      "F1-macro = 0.690 +/- 0.003\n",
      "\n",
      "{'iterations': 200, 'eta': 0.15, 'depth': 4}\n",
      "[0.69077669 0.69101366 0.69108219 0.69099362 0.68906438]\n",
      "F1-macro = 0.691 +/- 0.002\n",
      "\n",
      "{'iterations': 200, 'eta': 0.2, 'depth': 4}\n",
      "[0.68942495 0.68986491 0.69021426 0.69019422 0.6888704 ]\n",
      "F1-macro = 0.690 +/- 0.001\n",
      "\n",
      "{'iterations': 300, 'eta': 0.1, 'depth': 4}\n",
      "[0.68930191 0.69172552 0.69202335 0.69044022 0.68925836]\n",
      "F1-macro = 0.691 +/- 0.003\n",
      "\n",
      "{'iterations': 300, 'eta': 0.1, 'depth': 5}\n",
      "[0.68823604 0.69171051 0.69025599 0.69096944 0.68813119]\n",
      "F1-macro = 0.690 +/- 0.003\n",
      "\n",
      "{'iterations': 300, 'eta': 0.15, 'depth': 4}\n",
      "[0.68864641 0.69034233 0.68999775 0.69026735 0.68846827]\n",
      "F1-macro = 0.690 +/- 0.002\n",
      "\n",
      "CPU times: total: 27min 36s\n",
      "Wall time: 11min 33s\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "for param_set in cb_tops:\n",
    "    print(param_set)\n",
    "    model = cb_set_params(param_set)\n",
    "    check_stability(model, df_train)\n",
    "    print()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f56eb9a-309f-4822-a47f-ad210d42d139",
   "metadata": {},
   "source": [
    "We'll take the hyperparameters which were acquired earlier, because they show  \n",
    "decent stability."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e3f25c4-82d5-4412-86e4-9c754172c477",
   "metadata": {},
   "source": [
    "Let's look at the overall picture of this model's metrics on these hyperparameters:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 115,
   "id": "5c7cfaee-9065-4ebe-a882-bc429ac7c7e6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.88      0.93    225378\n",
      "         1.0       0.43      0.87      0.57     22202\n",
      "\n",
      "    accuracy                           0.88    247580\n",
      "   macro avg       0.71      0.88      0.75    247580\n",
      "weighted avg       0.94      0.88      0.90    247580\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Precision</th>\n",
       "      <th>Recall</th>\n",
       "      <th>F1-score</th>\n",
       "      <th>F1-macro</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>CatBoostClassifier</th>\n",
       "      <td>0.426</td>\n",
       "      <td>0.874</td>\n",
       "      <td>0.573</td>\n",
       "      <td>0.753</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                    Precision  Recall  F1-score  F1-macro\n",
       "CatBoostClassifier      0.426   0.874     0.573     0.753"
      ]
     },
     "execution_count": 115,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "cb_model = cb_set_params(cb_params)\n",
    "get_metrics(cb_model, X_train, y_train, X_test, y_test, verbose=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "886e44df-4208-4691-9ab5-d908e024a77e",
   "metadata": {},
   "source": [
    "Looking at the features' importance of the trained model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 116,
   "id": "41635799-40d2-45d0-9fec-7554b95e90d6",
   "metadata": {},
   "outputs": [],
   "source": [
    "f_cat_ohe = set(cb_model['features'].transformer_list[1][1]['encoder'].get_feature_names_out(list(f_categorical)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 117,
   "id": "fff4270d-eef1-463c-ade3-de423ef3ffe7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>coefficient</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>127_0.089549416646279</th>\n",
       "      <td>53.612286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>245_-24.99626861041084</th>\n",
       "      <td>35.378724</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133_-0.2172831953516344</th>\n",
       "      <td>4.372331</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>147_0.1385394173731442</th>\n",
       "      <td>1.742602</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>193_6.0709518035123535</th>\n",
       "      <td>0.667632</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>209_-0.6229007332475278</th>\n",
       "      <td>0.493697</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>145</th>\n",
       "      <td>0.376649</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>133_0.3827168046483656</th>\n",
       "      <td>0.237680</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>110_3.5265538518542154</th>\n",
       "      <td>0.216618</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>58</th>\n",
       "      <td>0.192642</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                         coefficient\n",
       "127_0.089549416646279      53.612286\n",
       "245_-24.99626861041084     35.378724\n",
       "133_-0.2172831953516344     4.372331\n",
       "147_0.1385394173731442      1.742602\n",
       "193_6.0709518035123535      0.667632\n",
       "209_-0.6229007332475278     0.493697\n",
       "145                         0.376649\n",
       "133_0.3827168046483656      0.237680\n",
       "110_3.5265538518542154      0.216618\n",
       "58                          0.192642"
      ]
     },
     "execution_count": 117,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "features = pd.DataFrame(cb_model['classifier'].feature_importances_, \n",
    "                        list(f_binary) + list(f_cat_ohe) + list(f_numeric),\n",
    "                        columns=['coefficient'])\n",
    "\n",
    "features.sort_values('coefficient', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "34356f16-e9da-4df5-8f1e-f7e98760ab4e",
   "metadata": {},
   "source": [
    "The model has obviously overfitted on certain values of 2 features. We don't have  \n",
    "information of what these features are, so it's impossible to logically explain this  \n",
    "fact and also justify another method of their processing, apart from one-hot-encoding."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5507c956-f4fa-43d8-b0ed-1141ac837899",
   "metadata": {},
   "source": [
    "## 8. Saving the model"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1673bc60-4c4e-4276-ae2c-c4b2e318920c",
   "metadata": {},
   "source": [
    "Saving classification threshold for future use:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 120,
   "id": "22940258-435d-411e-b4e3-d37a47a14c62",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Best Threshold=0.8481932512274507\n"
     ]
    }
   ],
   "source": [
    "y_pred = cb_model.predict_proba(X_test)[:, 1]\n",
    "threshold = find_best_threshold(y_test, y_pred)\n",
    "\n",
    "with open('./models/thresholds/threshold_final.pkl', 'wb') as f:\n",
    "    pickle.dump(threshold, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "974fe742-485f-4d71-be3e-77a74a2e8ff2",
   "metadata": {},
   "source": [
    "Before saving the final model and predicting on the test dataset we'll retrain  \n",
    "the model on the whole training set:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 121,
   "id": "95c6360a-3584-4a30-abcb-db815f9acb7f",
   "metadata": {},
   "outputs": [],
   "source": [
    "cb_model.fit(df_train, df_train['target']);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "id": "52746593-effe-4ed5-a04a-b2bb67bea173",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./models/final_model.pkl', 'wb') as f:\n",
    "    pickle.dump(cb_model, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22002126-8df3-4e9e-968d-477c8faf3ef3",
   "metadata": {},
   "source": [
    "## 9. Prediction"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "433d1b28-348a-4985-8124-fe3615e09539",
   "metadata": {},
   "source": [
    "Loading the test dataset:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "id": "541acaea-782c-47e6-a94b-2e128df4bd32",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(71231, 3)"
      ]
     },
     "execution_count": 123,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test = pd.read_csv('data_test.csv', index_col='Unnamed: 0', dtype={'id': str})\n",
    "df_test.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 124,
   "id": "6530f8dc-6d45-43de-8cc0-3da83f78ce5d",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>vas_id</th>\n",
       "      <th>buy_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>3130519</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1548018000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2000860</td>\n",
       "      <td>4.0</td>\n",
       "      <td>1548018000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1099444</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1546808400</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1343255</td>\n",
       "      <td>5.0</td>\n",
       "      <td>1547413200</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1277040</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1546808400</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        id  vas_id    buy_time\n",
       "0  3130519     2.0  1548018000\n",
       "1  2000860     4.0  1548018000\n",
       "2  1099444     2.0  1546808400\n",
       "3  1343255     5.0  1547413200\n",
       "4  1277040     2.0  1546808400"
      ]
     },
     "execution_count": 124,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_test.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e8cfa927-cddb-485b-8bab-32426bcb7619",
   "metadata": {},
   "source": [
    "Merging with the features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 125,
   "id": "fe9cebce-0876-4f8d-a810-4ad5ac2c168d",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_test = merge_datasets(df, df_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9ec0ce04-6a67-4fb5-b3be-dd6772c296f6",
   "metadata": {},
   "source": [
    "Loading the trained model and the classification threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 126,
   "id": "4d4b0e20-0325-4daa-a601-7696e843a1df",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./models/final_model.pkl', 'rb') as f:\n",
    "    model = pickle.load(f)\n",
    "    \n",
    "with open('./models/thresholds/threshold_final.pkl', 'rb') as f:\n",
    "    threshold = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e774de4e-081d-4c47-9a55-dffa552c78e7",
   "metadata": {},
   "source": [
    "Predicting probabilities:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 127,
   "id": "4c0483fd-e6bb-41e1-bac6-351ac1d2fdce",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = model.predict_proba(df_test)[:, 1]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "aeac3e30-f740-4d5f-b7a7-b32c0626d5be",
   "metadata": {},
   "source": [
    "Using the adjusted classification threshold to predict the target labels:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 128,
   "id": "3ff77332-c95f-4777-8b16-9a7b2f4a3fa6",
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = (y_pred >= threshold).astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 129,
   "id": "082ff655-aab6-4457-a396-04241af37790",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "array([0, 0, 1, 0, 0])"
      ]
     },
     "execution_count": 129,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "prediction[:5]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "40b3a92c-91cb-4caf-983c-bc5c286b7aa3",
   "metadata": {},
   "source": [
    "Adjusting the look of the predictions dataset to the needed format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 130,
   "id": "2aae06b2-9efb-4e63-9b57-35b1b1b44c39",
   "metadata": {},
   "outputs": [],
   "source": [
    "prediction = pd.Series(prediction, name='target', index=df_test.index)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 131,
   "id": "34bfe48c-4bc1-493c-8186-a2f3fdff62d8",
   "metadata": {},
   "outputs": [],
   "source": [
    "answers = pd.concat((df_test[['id', 'vas_id', 'buy_time_x']], prediction), axis=1).rename(columns={'buy_time_x': 'buy_time'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 132,
   "id": "06c394c9-6a97-4c52-aee8-6fe2cf63136a",
   "metadata": {},
   "outputs": [],
   "source": [
    "answers['buy_time'] = answers['buy_time'].astype('int')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 133,
   "id": "3d07ee18-1d93-44cf-950f-280a14982937",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>id</th>\n",
       "      <th>vas_id</th>\n",
       "      <th>buy_time</th>\n",
       "      <th>target</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>56191</th>\n",
       "      <td>1669643</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1546808400</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>55976</th>\n",
       "      <td>3451445</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1546808400</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13469</th>\n",
       "      <td>4051049</td>\n",
       "      <td>6.0</td>\n",
       "      <td>1547413200</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>48607</th>\n",
       "      <td>311691</td>\n",
       "      <td>7.0</td>\n",
       "      <td>1546808400</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8115</th>\n",
       "      <td>3456489</td>\n",
       "      <td>2.0</td>\n",
       "      <td>1547413200</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "            id  vas_id    buy_time  target\n",
       "56191  1669643     2.0  1546808400       0\n",
       "55976  3451445     2.0  1546808400       0\n",
       "13469  4051049     6.0  1547413200       1\n",
       "48607   311691     7.0  1546808400       0\n",
       "8115   3456489     2.0  1547413200       0"
      ]
     },
     "execution_count": 133,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "answers.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b11c3ee-8e26-402f-8b4f-03764e090715",
   "metadata": {},
   "source": [
    "Saving the predictions file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 134,
   "id": "71193fa2-82ef-4388-8b42-a0ad599375b9",
   "metadata": {},
   "outputs": [],
   "source": [
    "answers.to_csv('answers_test.csv', index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2fd2dfaa-0bea-4942-a91a-df38c2d4ef75",
   "metadata": {},
   "source": [
    "## 10. Realization of additional functionality, making model applicable to business"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9f047500-7625-4336-8f19-fd6ab5ed3c1a",
   "metadata": {},
   "source": [
    "By the course project condition, we also need to express the principle of making personal  \n",
    "proposals to selected customers.  \n",
    "At first, we'll talk about metrics. In this project we had F1-macro as the primary evaluation  \n",
    "metric. But for business task we need to decide on the importance of either Precision or Recall.  \n",
    "It all depends on the expenses, necessary to propose a service to a customer, and on the magnitude  \n",
    "of potential profit. For the current task, presumably, the expenses on a service proposal include  \n",
    "either calling customers (and we have expenses on a worker which makes calls) or SMS mailing.  \n",
    "If the expenses are high, the Precision metric becomes important for us to propose services  \n",
    "only to people, who most likely will accept them.  \n",
    "If, for example, the expenses on a service proposal are minimal, e.g. we have an algorithm,  \n",
    "which sends spam on the Internet, the Recall metric is more important, because we need maximum  \n",
    "coverage. However, even with minimal financial expenses on a service proposal, we don't want to  \n",
    "propose it to every person, because we are experiencing reputational costs when proposing a service  \n",
    "to people who will almost certainly reject it, and we'll be too pesky.  \n",
    "Because this course project doesn't give us data about expenses on a service proposal, we'll figure  \n",
    "for ourselves that we need Recall not less than 0.9 along with maximum Precision. For this, we need  \n",
    "to adjust the classification threshold for our model. For the threshold adjusting, we'll return again  \n",
    "to the model which was trained on a part of data. This way we'll have something to test the threshold on:  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 135,
   "id": "e4645c74-d133-43bb-abc6-52bcde1230a6",
   "metadata": {},
   "outputs": [],
   "source": [
    "cb_model.fit(X_train, y_train);"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 136,
   "id": "6cd3df1b-65ce-4cd9-90ba-65d4cd16063e",
   "metadata": {},
   "outputs": [],
   "source": [
    "def adjust_threshold(model, X_test, y_test, recall_goal=0.9):\n",
    "    \n",
    "    y_pred = model.predict_proba(X_test)[:, 1]\n",
    "    precision, recall, thresholds = precision_recall_curve(y_test, y_pred)\n",
    "    recall_mask = recall >= recall_goal\n",
    "    prec_idx = np.arange(len(precision))[recall_mask]\n",
    "    precs = precision[recall_mask]\n",
    "    max_prec_idx = prec_idx[np.argmax(precs)]\n",
    "    \n",
    "    return thresholds[max_prec_idx]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 137,
   "id": "dcf4ffd7-7b74-4c7d-b030-55433a9adeca",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.8313294515967059"
      ]
     },
     "execution_count": 137,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "threshold_business = adjust_threshold(cb_model, X_test, y_test)\n",
    "threshold_business"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "833c2281-f1fa-4096-9021-3e6a4d7a6847",
   "metadata": {},
   "source": [
    "Printing metrics for the new classification threshold:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "id": "3f533031-42c1-4340-b0df-eb0d48fe3833",
   "metadata": {},
   "outputs": [],
   "source": [
    "y_pred = cb_model.predict_proba(X_test)[:, 1]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "id": "26c655c0-b050-4451-b833-136794f1151e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.99      0.88      0.93    225378\n",
      "         1.0       0.42      0.90      0.57     22202\n",
      "\n",
      "    accuracy                           0.88    247580\n",
      "   macro avg       0.70      0.89      0.75    247580\n",
      "weighted avg       0.94      0.88      0.90    247580\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(classification_report(y_test, y_pred >= threshold_business))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "27123210-3b75-4458-96c0-5db4fb37e998",
   "metadata": {},
   "source": [
    "Recall is now 0.9 as was intended."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e17be0aa-fde7-414d-b8ff-bbcdc943af18",
   "metadata": {},
   "source": [
    "Saving new threshold value:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 140,
   "id": "989860a3-2118-4c52-bcfc-f3356dd3655d",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('./models/thresholds/threshold_business.pkl', 'wb') as f:\n",
    "    pickle.dump(threshold_business, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d40369d-61f3-4c57-a4a0-89a69c41facb",
   "metadata": {},
   "source": [
    "Writing a function, which will get a unique customer's number as an input and return  \n",
    "a list of services, which more likely will be accepted by the customer.  \n",
    "In our datasets, the latest service proposal date is 21 January 2019. We'll consider that  \n",
    "we propose a service on 22 January 2019. This date does not participate in training but it  \n",
    "does participate in datasets merging and we need to take the latest rows when there are  \n",
    "profile duplicates in the features dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 141,
   "id": "788b4da6-3746-47f0-a6ea-5e37ef07ba17",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict(user_id, hide_low_probas=True):\n",
    "    global df\n",
    "    \n",
    "    # Converting date to more common for the model timestamp\n",
    "    date = '2019-01-22 00:00:00'\n",
    "    date_format = datetime.datetime.strptime(date, '%Y-%m-%d %H:%M:%S')\n",
    "    date = int(datetime.datetime.timestamp(date_format))\n",
    "    \n",
    "    # Services' numbers\n",
    "    vas_id = [1.0, 2.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]\n",
    "    \n",
    "    # Creating a dataset consisting of combinations of a unique\n",
    "    # customer's number and all proposable services\n",
    "    data = np.array([np.array([user_id] * len(vas_id)),\n",
    "                     np.array(vas_id), \n",
    "                     np.array([date] * len(vas_id))]).T\n",
    "    \n",
    "    df_predict = pd.DataFrame(data=data, columns=['id', 'vas_id', 'buy_time'], \n",
    "                              index=np.arange(len(vas_id)))\n",
    "    \n",
    "    df_predict['vas_id'] = df_predict['vas_id'].astype(np.float16)\n",
    "    df_predict['buy_time'] = df_predict['buy_time'].astype(np.int64)\n",
    "    \n",
    "    # Loading the model and the adjusted classification threshold\n",
    "    with open('./models/final_model.pkl', 'rb') as f:\n",
    "        model = pickle.load(f)\n",
    "    \n",
    "    with open('./models/thresholds/threshold_business.pkl', 'rb') as f:\n",
    "        threshold = pickle.load(f)\n",
    "    \n",
    "    # Merging the datasets for prediction\n",
    "    df_predict = merge_datasets(df, df_predict)\n",
    "    \n",
    "    # Making a prediction and sorting by probability\n",
    "    df_predict['probas'] = model.predict_proba(df_predict)[:, 1]\n",
    "    df_predict['target'] = (df_predict['probas'] >= threshold).astype(np.int8)\n",
    "    df_predict.sort_values('probas', ascending=False, inplace=True)\n",
    "    \n",
    "    # Printing results\n",
    "    print(f'Абоненту {user_id} {\"ну\"*hide_low_probas}{\"мо\"*(1-hide_low_probas)}жно предложить следующие услуги:\\n')\n",
    "    df_predict.rename(columns={'vas_id': 'Номер_услуги', 'probas': 'Вероятность_принятия'}, inplace=True)\n",
    "    display(df_predict.loc[df_predict['target'].isin([1, hide_low_probas * 1]), ['Номер_услуги', 'Вероятность_принятия']])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b7c92fc4-cd49-489f-9904-3d577249a9b8",
   "metadata": {},
   "source": [
    "Let's pick several customers from the general dataset to predict the services:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 142,
   "id": "50ee3082-4060-4360-b379-35c860a29d16",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Абоненту 4021120 нужно предложить следующие услуги:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Номер_услуги</th>\n",
       "      <th>Вероятность_принятия</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.0</td>\n",
       "      <td>0.906711</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Номер_услуги  Вероятность_принятия\n",
       "4           6.0              0.906711"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Абоненту 1360724 нужно предложить следующие услуги:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Номер_услуги</th>\n",
       "      <th>Вероятность_принятия</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.0</td>\n",
       "      <td>0.901351</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Номер_услуги  Вероятность_принятия\n",
       "4           6.0              0.901351"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Абоненту 4259383 нужно предложить следующие услуги:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Номер_услуги</th>\n",
       "      <th>Вероятность_принятия</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.0</td>\n",
       "      <td>0.899476</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Номер_услуги  Вероятность_принятия\n",
       "4           6.0              0.899476"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for user_id in df['id'].sample(3, random_state=RANDOM_STATE):\n",
    "    predict(user_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "17b03369-84cc-4b18-acd5-d24faf82b15f",
   "metadata": {},
   "source": [
    "We got the same result for all customers and very narrow probability spread.  \n",
    "Let's pick the first two customers and compare the probabilities of all services:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 143,
   "id": "50da30e8-c68e-4672-b429-800ae028bd4d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Абоненту 4021120 можно предложить следующие услуги:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Номер_услуги</th>\n",
       "      <th>Вероятность_принятия</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.0</td>\n",
       "      <td>0.906711</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.819322</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>9.0</td>\n",
       "      <td>0.739898</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.248189</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.210266</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.203339</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0.201476</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>7.0</td>\n",
       "      <td>0.175678</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Номер_услуги  Вероятность_принятия\n",
       "4           6.0              0.906711\n",
       "2           4.0              0.819322\n",
       "7           9.0              0.739898\n",
       "6           8.0              0.248189\n",
       "1           2.0              0.210266\n",
       "0           1.0              0.203339\n",
       "3           5.0              0.201476\n",
       "5           7.0              0.175678"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Абоненту 1360724 можно предложить следующие услуги:\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Номер_услуги</th>\n",
       "      <th>Вероятность_принятия</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>6.0</td>\n",
       "      <td>0.901351</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>4.0</td>\n",
       "      <td>0.807610</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>9.0</td>\n",
       "      <td>0.727720</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>8.0</td>\n",
       "      <td>0.248095</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2.0</td>\n",
       "      <td>0.199706</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>5.0</td>\n",
       "      <td>0.193742</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1.0</td>\n",
       "      <td>0.192625</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>7.0</td>\n",
       "      <td>0.172546</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Номер_услуги  Вероятность_принятия\n",
       "4           6.0              0.901351\n",
       "2           4.0              0.807610\n",
       "7           9.0              0.727720\n",
       "6           8.0              0.248095\n",
       "1           2.0              0.199706\n",
       "3           5.0              0.193742\n",
       "0           1.0              0.192625\n",
       "5           7.0              0.172546"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "for user_id in df['id'].sample(2, random_state=RANDOM_STATE):\n",
    "    predict(user_id, hide_low_probas=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e080124f-20b7-4b1c-a3d6-36e74b873210",
   "metadata": {},
   "source": [
    "We see that the probabilities' spread is not too much when it comes to one  \n",
    "particular service. With our classification threshold, adjusted for Recall >= 0.9,  \n",
    "only one service out of eight could surpass it. This system obviously could use some  \n",
    "reworking. The classification threshold should be different for each service.  \n",
    "Let's train models for this and adjust personal thresholds for each one.  \n",
    "In a real task, when we know which service stands behind each service number, cost to  \n",
    "propose each service and potential profit, the Precision/Recall ratio is adjusted  \n",
    "individually. Here for each service we'll set Recall >= 0.7 with maximum Precision."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "id": "1a5f6cd4-915c-4c2b-92a6-1db3da825b6d",
   "metadata": {},
   "outputs": [],
   "source": [
    "vas_ids = [1.0, 2.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]\n",
    "\n",
    "# To get more stratified samples we'll split the dataset\n",
    "# randomly, not by months like before\n",
    "X_train_shuffled, X_test_shuffled, y_train_shuffled, y_test_shuffled = train_test_split(\n",
    "                                                                    df_train, df_train['target'],\n",
    "                                                                    test_size=0.3, stratify=df_train['vas_id'],\n",
    "                                                                    random_state=RANDOM_STATE)\n",
    "\n",
    "for vas_id in vas_ids:\n",
    "    \n",
    "    X_train_part = X_train_shuffled.loc[X_train_shuffled['vas_id'] == vas_id]\n",
    "    y_train_part = X_train_part['target']\n",
    "    X_test_part = X_test_shuffled.loc[X_test_shuffled['vas_id'] == vas_id]\n",
    "    y_test_part = X_test_part['target']\n",
    "    \n",
    "    cb_model.fit(X_train_part, y_train_part)\n",
    "    \n",
    "    with open(f'./models/cb_model_{vas_id}', 'wb') as f:\n",
    "        pickle.dump(cb_model, f)\n",
    "    \n",
    "    threshold = adjust_threshold(cb_model, X_test_part, y_test_part, recall_goal=0.7)\n",
    "    \n",
    "    with open(f'./models/thresholds/threshold_business_{vas_id}', 'wb') as f:\n",
    "        pickle.dump(threshold, f)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15cec391-e6d9-4a4a-895f-5f42fd16f82a",
   "metadata": {},
   "source": [
    "Checking metrics of one of the models:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "id": "0d334ab4-078c-4d8c-b719-d96759868c8d",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "              precision    recall  f1-score   support\n",
      "\n",
      "         0.0       0.75      0.27      0.40     19168\n",
      "         1.0       0.26      0.74      0.38      6559\n",
      "\n",
      "    accuracy                           0.39     25727\n",
      "   macro avg       0.51      0.51      0.39     25727\n",
      "weighted avg       0.63      0.39      0.39     25727\n",
      "\n"
     ]
    }
   ],
   "source": [
    "with open(f'./models/cb_model_4.0', 'rb') as f:\n",
    "    model = pickle.load(f)\n",
    "X_train_part = X_train_shuffled.loc[X_train_shuffled['vas_id'] == 4.0]\n",
    "y_train_part = X_train_part['target']\n",
    "X_test_part = X_test_shuffled.loc[X_test_shuffled['vas_id'] == 4.0]\n",
    "y_test_part = X_test_part['target']\n",
    "\n",
    "with open(f'./models/thresholds/threshold_business_4.0', 'rb') as f:\n",
    "    threshold = pickle.load(f)\n",
    "    \n",
    "y_pred = model.predict_proba(X_test_part)[:, 1]\n",
    "print(classification_report(y_test_part, y_pred >= threshold))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2982cf72-97ae-494f-84a2-7cbd85597215",
   "metadata": {},
   "source": [
    "Metrics are not very good, this once again tells us that each of these models  \n",
    "should be worked with individually: extract features, choose a model, adjust its  \n",
    "hyperparameters, carefully choose a classification threshold by business metrics.  \n",
    "Also there is a large classes disbalance for some services, which makes a necessity  \n",
    "of using oversampling or undersampling. In the frame of this course project these  \n",
    "operations are not provided also because the time to run the whole notebook would be  \n",
    "indecently big."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b601efd-c5bf-4edb-9291-af6db707011c",
   "metadata": {},
   "source": [
    "Actualizing prediction function:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "id": "fdf663bd-dc5a-4cda-a477-d1fd834cb6d4",
   "metadata": {},
   "outputs": [],
   "source": [
    "def predict_final(user_id):\n",
    "    global df\n",
    "    \n",
    "    # Converting date to more common for the model timestamp\n",
    "    date = '2019-01-22 00:00:00'\n",
    "    date_format = datetime.datetime.strptime(date, '%Y-%m-%d %H:%M:%S')\n",
    "    date = int(datetime.datetime.timestamp(date_format))\n",
    "    \n",
    "    # Services' numbers\n",
    "    vas_ids = [1.0, 2.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]\n",
    "    \n",
    "    # Creating a dataset consisting of combinations of a unique\n",
    "    # customer's number and all proposable services\n",
    "    data = np.array([np.array([user_id] * len(vas_ids)),\n",
    "                     np.array(vas_ids), \n",
    "                     np.array([date] * len(vas_ids))]).T\n",
    "    \n",
    "    df_predict = pd.DataFrame(data=data, columns=['id', 'vas_id', 'buy_time'], \n",
    "                              index=np.arange(len(vas_ids)))\n",
    "    \n",
    "    df_predict['vas_id'] = df_predict['vas_id'].astype(np.float16)\n",
    "    df_predict['buy_time'] = df_predict['buy_time'].astype(np.int64)\n",
    "    \n",
    "    # Merging the datasets for prediction\n",
    "    df_predict = merge_datasets(df, df_predict)\n",
    "    \n",
    "    # Making predictions\n",
    "    services = []\n",
    "    for vas_id in vas_ids:\n",
    "        with open(f'./models/cb_model_{vas_id}', 'rb') as f:\n",
    "            model = pickle.load(f)\n",
    "        \n",
    "        with open(f'./models/thresholds/threshold_business_{vas_id}', 'rb') as f:\n",
    "            threshold = pickle.load(f)\n",
    "        \n",
    "        prediction = model.predict_proba(df_predict.loc[df_predict['vas_id'] == vas_id])[:, 1]\n",
    "        \n",
    "        # Saving predictions which exceeded classification threshold\n",
    "        if prediction >= threshold:\n",
    "            services.append(int(vas_id))\n",
    "    \n",
    "    # Printing result\n",
    "    print(f'Абоненту {user_id} нужно предложить следующие услуги:')\n",
    "    for service in services:\n",
    "        print(service, end=' ')\n",
    "    print(end='\\n\\n')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "id": "c4e0f051-7959-41d3-8170-0c5f14668945",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Абоненту 4021120 нужно предложить следующие услуги:\n",
      "4 5 6 7 9 \n",
      "\n",
      "Абоненту 1360724 нужно предложить следующие услуги:\n",
      "1 4 5 6 7 9 \n",
      "\n",
      "Абоненту 4259383 нужно предложить следующие услуги:\n",
      "1 2 5 6 7 8 9 \n",
      "\n"
     ]
    }
   ],
   "source": [
    "for user_id in df['id'].sample(3, random_state=RANDOM_STATE):\n",
    "    predict_final(user_id)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}