A:B Testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Scenario:\n",
    "> *PT Maju Makmur Sentosa Raya ingin mengetahui apakah **desain web baru mereka dapat meningkatkan conversion rate web** mereka. Perusahaan tersebut memiliki **data historis conversion rate dari desain lama dan desain baru selama 30 hari ke belakang**. Data tersebut diberikan dalam format csv dengan kolom **conversion_a adalah conversion rate harian dari desain web lama** dan kolom **conversion_b adalah conversion rate harian dari desain web baru**. Perusahaan ingin mengetahui apakah rata-rata conversion rate dari desain baru lebih besar dari desain lama.*\n",
    "\n",
    "**A/B Testing steps:**\n",
    "1. Designing our experiment\n",
    "2. Collecting and preparing the data\n",
    "3. Visualising the results\n",
    "4. Testing the hypothesis\n",
    "5. Drawing conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Designing our experiment"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Formulating a hypothesis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the scenario above the company wants to know whether the average conversion rate of the new design is greater than the old design. Given that case, we'll choose a **one-tailed test**. So our hypothesis will be:\n",
    "\n",
    "  - H0 : µ0 >= µ (The average conversion rate of the old design is higher than or equal to the new design)\n",
    "  - H1 : µ > µ0  (The average conversion rate of the new design is higher than the old design)\n",
    "\n",
    "Where **µ** and **µ0** stands for the average conversion rate of the new and old design respectively.\n",
    "\n",
    "We will calculate our hypothesis with two sample t-test, and use **α = 0.05** which means our confidence level is **95%**. Since Scipy provides two sample t-test function called ttest_ind, but which can only do two-tailed t-tests. Closest information on the topic I found in this link, https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html\n",
    "\n",
    "> *because the one-sided tests can be backed out from the two-sided tests. (With symmetric distributions one-sided p-value is just half of the two-sided pvalue)*\n",
    "\n",
    "Given the p and t values from a two-tailed test, so we would reject the null hypothesis when **p/2 < alpha and t > 0**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Collecting and preparing the data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns### Designing our experiment\n",
    "from scipy import stats\n",
    "\n",
    "sns.set()\n",
    "sns.set_theme(style = 'whitegrid')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Day</th>\n",
       "      <th>Conversion_A</th>\n",
       "      <th>Conversion_B</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>0.15</td>\n",
       "      <td>0.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>0.12</td>\n",
       "      <td>0.20</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>0.18</td>\n",
       "      <td>0.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>0.19</td>\n",
       "      <td>0.22</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>0.17</td>\n",
       "      <td>0.19</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   Day  Conversion_A  Conversion_B\n",
       "0    1          0.15          0.19\n",
       "1    2          0.12          0.20\n",
       "2    3          0.18          0.18\n",
       "3    4          0.19          0.22\n",
       "4    5          0.17          0.19"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "data = pd.read_csv('hw_data.csv')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 30 entries, 0 to 29\n",
      "Data columns (total 3 columns):\n",
      " #   Column        Non-Null Count  Dtype  \n",
      "---  ------        --------------  -----  \n",
      " 0   Day           30 non-null     int64  \n",
      " 1   Conversion_A  30 non-null     float64\n",
      " 2   Conversion_B  30 non-null     float64\n",
      "dtypes: float64(2), int64(1)\n",
      "memory usage: 848.0 bytes\n"
     ]
    }
   ],
   "source": [
    "data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Group</th>\n",
       "      <th>Conversion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Conversion_A</td>\n",
       "      <td>0.15</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Conversion_A</td>\n",
       "      <td>0.12</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Conversion_A</td>\n",
       "      <td>0.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Conversion_A</td>\n",
       "      <td>0.19</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Conversion_A</td>\n",
       "      <td>0.17</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Group  Conversion\n",
       "0  Conversion_A        0.15\n",
       "1  Conversion_A        0.12\n",
       "2  Conversion_A        0.18\n",
       "3  Conversion_A        0.19\n",
       "4  Conversion_A        0.17"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Melt our table for the easier look\n",
    "data = data.melt(id_vars = 'Day', var_name = 'Group', value_name = 'Conversion')\n",
    "data = data.drop(columns = 'Day')\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Sampling"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Group</th>\n",
       "      <th>Conversion</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Conversion_A</td>\n",
       "      <td>0.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Conversion_A</td>\n",
       "      <td>0.21</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Conversion_A</td>\n",
       "      <td>0.11</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Conversion_A</td>\n",
       "      <td>0.18</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Conversion_A</td>\n",
       "      <td>0.11</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "          Group  Conversion\n",
       "0  Conversion_A        0.21\n",
       "1  Conversion_A        0.21\n",
       "2  Conversion_A        0.11\n",
       "3  Conversion_A        0.18\n",
       "4  Conversion_A        0.11"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Sample our data\n",
    "control_sample = data[data['Group'] == 'Conversion_A'].sample(n = 30, random_state = 42)\n",
    "treatment_sample = data[data['Group'] == 'Conversion_B'].sample(n = 30, random_state = 42)\n",
    "\n",
    "ab_test = pd.concat([control_sample, treatment_sample], axis = 0)\n",
    "ab_test.reset_index(drop = True, inplace = True)\n",
    "\n",
    "ab_test.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Conversion_A    30\n",
       "Conversion_B    30\n",
       "Name: Group, dtype: int64"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ab_test[\"Group\"].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Visualising the results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>avg_conversion</th>\n",
       "      <th>std_deviation</th>\n",
       "      <th>std_error</th>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Group</th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "      <th></th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>Conversion_A</th>\n",
       "      <td>0.162000</td>\n",
       "      <td>0.036141</td>\n",
       "      <td>0.006599</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>Conversion_B</th>\n",
       "      <td>0.193333</td>\n",
       "      <td>0.027334</td>\n",
       "      <td>0.004990</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              avg_conversion  std_deviation  std_error\n",
       "Group                                                 \n",
       "Conversion_A        0.162000       0.036141   0.006599\n",
       "Conversion_B        0.193333       0.027334   0.004990"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "conversion_rates = ab_test.groupby(\"Group\")[\"Conversion\"].agg([np.mean, np.std, stats.sem])\n",
    "conversion_rates.columns = [\"avg_conversion\", \"std_deviation\", \"std_error\"]\n",
    "\n",
    "conversion_rates"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that **the mean of Conversion_B is greater than the mean of Conversion_A**. This means that the new design has greater conversion rate than the old design. But let's check if this result is statistically significant."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "      "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<AxesSubplot:xlabel='Group', ylabel='Conversion'>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAZAAAAEKCAYAAAA8QgPpAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjMuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8vihELAAAACXBIWXMAAAsTAAALEwEAmpwYAAApNElEQVR4nO3df1RU9b7/8Se/E+GIGAMevde6R9ObSN6kZGEHKxMUoSmSjj8KOh5ROnnwUlF0/FX+zvxB18yszjr3dNTMFQhRhFhdyi6USllQdqtz+uEiG0A0gRRmmP39w+V8m1QYRyZQX4+1Wou9P5/Pnvd27eY1nz2z9/YyDMNARETkHHl3dwEiInJhUoCIiIhbFCAiIuIWBYiIiLhFASIiIm5RgIiIiFt8Pbnx4uJiNm7ciM1mIz09nenTpzu1v/HGG6xfvx7DMBg4cCArVqygT58+fPfdd+Tk5HD48GGuvPJKVq9eTe/evTl27BgPPvggBw8eJDQ0lLy8PMLCwjqtw26309LSgp+fH15eXp7aXRGRi4phGFitVnr37o239+nzDS9PXQdisViYOnUqBQUF+Pv7M2XKFNauXcvgwYMBaG5uZsKECeTn5xMeHs6TTz5JU1MT8+fPZ/bs2dx6661MmjSJDRs28OOPP5KTk8PixYuJiIhg1qxZFBYWUl5eTl5eXqe1NDU18fnnn3tiN0VELnpXXXUVwcHBp6332AykoqKCmJgYQkJCAEhISKC0tJQ5c+YAYLVaWbRoEeHh4QAMHTqU4uJirFYre/fuZcOGDQCkpKRw1113kZOTQ3l5OVu2bAEgKSmJxYsXY7Va8fPz67CWU+1XXXUV/v7+nthdEZGLTltbG59//vlZ32M9FiB1dXVOp5dMJhMff/yxY7lv376MHz8egBMnTvDss89y9913c+TIEYKCgvD1PVlaWFgYFovltG36+voSFBREY2OjI4TO5tRpK39/fwICArpuJ0VELgFnO/XvsQCx2+1OL2oYxhmLaGpq4r777mPYsGHcfvvtWCyW0/qdrXjDMM54Xu5sampqXO4rIiId81iAREREsG/fPsdyfX09JpPJqU9dXR1/+MMfiImJ4c9//jMAoaGhNDU10d7ejo+Pj9M4k8lEQ0MDERER2Gw2WlpaHKfIXBEZGakZiIiIi1pbWzv84O2xn/HGxsZSWVlJY2Mjx48fp6ysjLi4OEd7e3s7mZmZTJw4kXnz5jlmGX5+fkRHR1NSUgJAYWGhY9zYsWMpLCwEoKSkhOjo6E6//xAREc/w2K+w4OTPeDdt2oTVamXy5MlkZGSQkZFBVlYW33//PX/6058YOnSoo39kZCTLli2jtraW3NxcDh8+TP/+/Vm7di19+vTh6NGj5ObmcvDgQYKDg1m9ejUDBw7stI5TKaoZiIiI6zp77/RogPQUChARkXPX2XunrkQXERG3KEBERMQtChCRi0Bbu7W7S5AeyNPHhUfvhSUivwx/Hz/itz3S3WVID1M2ZYVHt68ZiIiIuEUBIiIiblGAiIiIWxQgIiLiFgWIiIi4RQEiIiJuUYCIiIhbFCAiIuIWBYiIiLhFASIiIm5RgIiIiFsUICIi4hYFiIiIuEUBIiIibvHo7dyLi4vZuHEjNpuN9PR0pk+ffsZ+Dz30EDExMaSkpHD48GFmzJjhaGtqauLIkSN8+OGH7Nmzhz/96U9EREQAcPXVV7NihWdvVywiImfmsQCxWCysW7eOgoIC/P39mTJlCqNHj2bw4MFOfRYtWkRlZSUxMTEA9OvXj6KiIgDsdjvp6elkZ2cDUFNTw4wZM5g9e7anyhYRERd57BRWRUUFMTExhISEEBgYSEJCAqWlpU59iouLGTduHBMnTjzjNvLz8+nVqxfJyckAVFdX8+6775KcnExmZiaHDh3yVPkiItIJjwVIXV0dYWFhjmWTyYTFYnHqM3PmTFJTU884vr29nWeeeYYHHnjAsS44OJi7776b4uJixo4d65iZiIjIL89jp7DsdjteXl6OZcMwnJY7s3v3bq644gqGDh3qWLd48WLH31OnTmXNmjU0NTURHBzs0jZrampcfn2RC8moUaO6uwTpoaqqqjy2bY8FSEREBPv27XMs19fXYzKZXB7/xhtvkJiY6Fi22+1s2rSJWbNm4ePj41j/0787ExkZSUBAgMv9RUQudOfz4aK1tbXDD94eO4UVGxtLZWUljY2NHD9+nLKyMuLi4lwev3//fqKjox3L3t7e7Nq1i507dwJQWFjINddcQ2BgYJfXLiIinfNYgISHh5OdnU1aWhq33XYbSUlJREVFkZGRQXV1dafjDx486Pi57imPP/44L7zwApMmTSI/P5+lS5d6qnwREemEl2EYRncX4WmnpmE6hSUXs/htj3R3CdLDlE05v+vkOnvv1JXoIiLiFgWIiIi4RQEiIiJuUYCIiIhbFCAiIuIWBYiIiLhFASIiIm5RgIiIiFsUICIi4hYFyDlotdq6uwTpgXRcyKXKo4+0vdgE+Pny29lLursM6WF2b1rQ3SWIdAvNQERExC0KEBERcYsCRERE3KIAERERtyhARETELQoQERFxiwJERETc4tEAKS4uJjExkfj4eLZs2XLWfg899BAFBQWO5R07dnDDDTdgNpsxm82sW7cOgGPHjjFr1iwmTpzI9OnTqa+v92T5IiLSAY9dSGixWFi3bh0FBQX4+/szZcoURo8ezeDBg536LFq0iMrKSmJiYhzra2pqyM3NJSkpyWmbeXl5REdH8+yzz1JYWMiyZcvIy8vz1C6IiEgHPDYDqaioICYmhpCQEAIDA0lISKC0tNSpT3FxMePGjWPixIlO66urq9mxYwfJyck8+OCD/PDDDwCUl5eTnJwMQFJSEu+88w5Wq9VTuyAiIh3wWIDU1dURFhbmWDaZTFgsFqc+M2fOJDU19bSxYWFh/PGPf+SVV16hf//+LF68+LRt+vr6EhQURGNjo6d2QUREOuCxU1h2ux0vLy/HsmEYTssd2bBhg+PvmTNnMn78+DP2MwwDb2/XM7CmpsblvmcyatSo8xovF6+qqqpufX0dm3I2njw2PRYgERER7Nu3z7FcX1+PyWTqdFxTUxP5+fncc889wMmQ8PHxAU7OYhoaGoiIiMBms9HS0kJISIjLNUVGRhIQEHBO+yHiCr2BS091Psdma2trhx+8PXYKKzY2lsrKShobGzl+/DhlZWXExcV1Oi4wMJDnn3+ejz76CIDNmzc7ZiBjx46lsLAQgJKSEqKjo/Hz8/PULoiISAc8NgMJDw8nOzubtLQ0rFYrkydPJioqioyMDLKyshgxYsQZx/n4+JCXl8ejjz7KiRMnuOKKK1i1ahUAc+fOJTc3l0mTJhEcHMzq1as9Vb6IiHTCyzAMo7uL8LRT07CuOIWl54HIz/WU54HEb3uku0uQHqZsyorzGt/Ze6euRBcREbcoQERExC0KEBERcYsCRERE3KIAERERtyhARETELQoQERFxiwJERETcogARERG3KEBERMQtChAREXGLAkRERNyiABEREbcoQERExC0KEBERcYsCRERE3KIAERERtyhARETELR4NkOLiYhITE4mPj2fLli1n7ffQQw9RUFDgWK6qqmLy5MmYzWbS09Opra0FYM+ePYwePRqz2YzZbOaRR/QITxGR7uLrSqd//vOfPPfccxw9epSfPkL9mWeeOesYi8XCunXrKCgowN/fnylTpjB69GgGDx7s1GfRokVUVlYSExPjWJ+Tk8PTTz/NsGHDePnll1m6dCkbN26kpqaGGTNmMHv2bHf2VUREupBLAZKbm0tUVBTXXXcdXl5eLm24oqKCmJgYQkJCAEhISKC0tJQ5c+Y4+hQXFzNu3DhHH4C2tjbmzp3LsGHDABg6dCibN28GoLq6moaGBl599VUGDBjAokWL6N+/v0v1iIhI13IpQI4fP878+fPPacN1dXWEhYU5lk0mEx9//LFTn5kzZwInT1md4u/vj9lsBsBut/PUU09xyy23ABAcHMzEiROJj4/nxRdfJDs7m23btrlcU01NzTntw8+NGjXqvMbLxeunx3B30LEpZ+PJY9OlABk0aBB1dXWYTCaXN2y3251mK4ZhuDx7gZMzkdzcXGw2m+OU1eLFix3tU6dOZc2aNTQ1NREcHOzSNiMjIwkICHC5BhFX6Q1ceqrzOTZbW1s7/ODtUoDY7XaSkpIYPny40xtwR9+BREREsG/fPsdyfX29ywHU0tLCvffeS0hICBs3bsTPzw+73c6mTZuYNWsWPj4+jr4//VtERH45LgXI+PHjGT9+/DltODY2lvXr19PY2EivXr0oKytjyZIlLo3Nyclh0KBBPPbYY3h7n/yhmLe3N7t27WLQoEEkJiZSWFjINddcQ2Bg4DnVJSIiXcOlALn99tupra1lz5492Gw2rr/+egYNGtThmPDwcLKzs0lLS8NqtTJ58mSioqLIyMggKyuLESNGnHHcp59+yptvvsngwYO5/fbbgZPfnzz33HM8/vjjLFiwgA0bNhAaGsqqVavOcXdFRKSreBk//V3uWezevZucnBxGjRpFe3s7VVVVrFixwvHldk936jxeV3wH8tvZrs2i5NKxe9OC7i4BgPhtui5KnJVNWXFe4zt773RpBvLkk0+yefNmxzUcX3zxBTk5ORdMgIiISNdz6Up0q9XqdAHgkCFDaG9v91hRIiLS87kUIJdddhnV1dWO5erqanr16uWxokREpOdz6RRWTk4OmZmZji/Ov/rqK5588kmPFiYiIj2bSwESHR3Na6+9xkcffYTdbmfkyJH07dvX07WJiEgP1mGAFBUVYTab+etf/+q0/uuvvwbg97//vccKExGRnq3DAPnmm28A+Pzzz3+RYkRE5MLRYYBkZWUBsGLF//8tcVtbGw0NDfz617/2bGUiItKjufQrrF27drFkyRKam5uZMGECZrOZv/3tb56uTUREejCXAmTTpk3ceeedlJWVMXLkSP7nf/6HoqIiT9cmIiI9mEsBYhgGQ4cOpaKigri4OIKCgnDhDigiInIRcylAvL29KSkp4d1332XMmDG8/fbb5/RsDxERufi4FCC5ubls376d+++/n7CwMDZu3HjOTygUEZGLi0sXEpaXl/Pf//3fjuVzeYysiIhcnFyagZSXl3u4DBERudC4NAMZOHAgM2bM4Nprr6V3796O9boSXUTk0uVSgISEhABQW1vryVpEROQC4lKAnLoS/dixY/zqV79yeePFxcVs3LgRm81Geno606dPP2O/hx56iJiYGFJSUgD47rvvyMnJ4fDhw1x55ZWsXr2a3r17c+zYMR588EEOHjxIaGgoeXl5hIWFuVyPiIh0HZe+A/nqq69ITExk0qRJWCwWJk6cyD/+8Y8Ox1gsFtatW8fWrVspLCzkpZde4ssvvzytT2ZmJjt37nRa/9hjjzFt2jRKS0uJjIzk6aefBiAvL4/o6Ghef/11UlNTWbZs2bnsq4iIdCGXAmTJkiXMmzePfv36ER4ezl133cXChQs7HFNRUUFMTAwhISEEBgaSkJBAaWmpU5/i4mLGjRvHxIkTHeusVit79+4lISEBgJSUFMe48vJykpOTAUhKSuKdd97BarW6vrciItJlXAqQo0ePMmbMGMfy9OnTaW5u7nBMXV2d0+klk8mExWJx6jNz5kxSU1Od1h05coSgoCB8fU+eXQsLC3OM++k2fX19CQoKorGx0ZVdEBGRLubSdyAAra2tjqvP6+vrsdvtHfa32+1OV6sbhuHS1etn6ne2cYZh4O3tUgYCUFNT43LfMxk1atR5jZeLV1VVVbe+vo5NORtPHpsuBcjUqVP5wx/+wOHDh1mzZg2vvfYaM2fO7HBMREQE+/btcyzX19djMpk6fa3Q0FCamppob2/Hx8fHaZzJZKKhoYGIiAhsNhstLS2OX4i5IjIykoCAAJf7i7hKb+DSU53Psdna2trhB2+XPr6npqYyd+5ckpOTsdlsLFmyhGnTpnU4JjY2lsrKShobGzl+/DhlZWXExcV1+lp+fn5ER0dTUlICQGFhoWPc2LFjKSwsBKCkpITo6Gj8/Pxc2QUREeliLs1A1qxZw5133klOTo7LGw4PDyc7O5u0tDSsViuTJ08mKiqKjIwMsrKyGDFixFnHLlq0iNzcXDZu3Ej//v1Zu3YtAHPnziU3N5dJkyYRHBzM6tWrXa5HRES6lksBYhgG06dP58orryQ1NZX4+Hj8/f07HZecnOz41dQpzz333Gn9Vq5c6bQ8YMAA/v73v5/WLyQkhGeeecaVkkVExMNcOoX14IMPUl5ezowZM9i1axe33HILy5cv93RtIiLSg7n8Kyxvb28iIyP56quv+Prrr52+IBcRkUuPSwGya9cu8vPz2b9/PxMmTGD58uUMHz7c07WJiEgP5lKA/OUvf+HOO+8kLy+Pyy67zNM1iYjIBcClANm2bRttbW0cP36cEydOONafyzUYIiJycXE5QJYvX47VasUwDODk1eEHDhzwaHEiItJzuRQgzz//PC+++KK+9xAREQeXfsZ7+eWXKzxERMSJSwFyww03sHXrViwWC0ePHnX8JyIily6XTmE9++yztLW1sXjxYsc6fQciInJpcylAPv74Y0/XISIiFxiXAsRut/OXv/yFd955B5vNxpgxY8jMzHQ89ElERC49Ln0HsmbNGt577z3S09P5/e9/z4cffsjjjz/u6dpERKQHc2kKsXv3bvLz8x3P3rjxxhu59dZbPVqYiIj0bC7NQAzDcHpwk7+/vx7kJCJyiXMpQIYNG8by5cv59ttvOXjwIMuXL+eqq67ydG0iItKDuRQgixYt4tixY0yZMoXU1FSOHDnCggULPF2biIj0YB0GSFtbGw8//DCVlZWsXLmSiooKoqKi8PHxISgo6JeqUUREeqAOA+S//uu/aG5u5tprr3WsW7JkCceOHWP9+vWdbry4uJjExETi4+PZsmXLae0HDhwgJSWFhIQE5s2bh81m4/Dhw5jNZsd/N998M//xH/8BwJ49exg9erSj7ZFHHjnX/RURkS7S4a+wysvLefnll52eARIeHs6qVav43e9+R3Z29lnHWiwW1q1bR0FBAf7+/kyZMoXRo0czePBgR5+cnByWLl3KyJEj+fOf/8z27duZNm0aRUVFwMnrT9LT0x2vU1NTw4wZM5g9e/Z57bSIiJy/Dmcgfn5+Z3yAVFBQEP7+/h1uuKKigpiYGEJCQggMDCQhIYHS0lJHe21tLSdOnGDkyJEApKSkOLUD5Ofn06tXL5KTkwGorq7m3XffJTk5mczMTA4dOuTSToqISNfrMEC8vb1pbm4+bX1zczM2m63DDdfV1REWFuZYNplMWCyWs7aHhYU5tbe3t/PMM8/wwAMPONYFBwdz9913U1xczNixYzucAYmIiGd1eAorKSmJ+fPns3z5cgIDAwH48ccfmT9/PvHx8R1u2G634+Xl5Vg2DMNpubP23bt3c8UVVzB06FDHup/ezHHq1KmsWbOGpqYmgoODO9tP4OQpsPMxatSo8xovF6+qqqpufX0dm3I2njw2OwyQ9PR0Fi1axJgxYxgyZAh2u51//OMfJCcnc99993W44YiICPbt2+dYrq+vx2QyObXX19c7lhsaGpza33jjDRITEx3LdrudTZs2MWvWLHx8fBzrf/p3ZyIjIwkICHC5v4ir9AYuPdX5HJutra0dfvDuMEC8vb1ZsmQJmZmZfPLJJ3h7exMVFeX0Rn82sbGxrF+/nsbGRnr16kVZWRlLlixxtA8YMICAgACqqqoYNWoURUVFxMXFOdr3799PRkaGUy27du1i0KBBJCYmUlhYyDXXXOOYGYmIyC/LpXthDRgwgAEDBpzThsPDw8nOziYtLQ2r1crkyZOJiooiIyODrKwsRowYwerVq5k/fz7Nzc0MHz6ctLQ0x/iDBw8SERHhtM3HH3+cBQsWsGHDBkJDQ1m1atU51SQiIl3Ho/djT05OdvyC6pTnnnvO8fewYcN4+eWXzzj2o48+Om3dkCFD2LZtW9cWKSIibnHpViYiIiI/pwARERG3KEBERMQtChAREXGLAkRERNyiABEREbcoQERExC0KEBERcYsCRERE3KIAERERtyhARETELQoQERFxiwJERETcogARERG3KEBERMQtChAREXGLAkRERNyiABEREbd4NECKi4tJTEwkPj6eLVu2nNZ+4MABUlJSSEhIYN68edhsNgB27NjBDTfcgNlsxmw2s27dOgCOHTvGrFmzmDhxItOnT6e+vt6T5YuISAc8FiAWi4V169axdetWCgsLeemll/jyyy+d+uTk5LBw4UJ27tyJYRhs374dgJqaGnJzcykqKqKoqIjs7GwA8vLyiI6O5vXXXyc1NZVly5Z5qnwREemExwKkoqKCmJgYQkJCCAwMJCEhgdLSUkd7bW0tJ06cYOTIkQCkpKQ42qurq9mxYwfJyck8+OCD/PDDDwCUl5eTnJwMQFJSEu+88w5Wq9VTuyAiIh3wWIDU1dURFhbmWDaZTFgslrO2h4WFOdrDwsL44x//yCuvvEL//v1ZvHjxaWN8fX0JCgqisbHRU7sgIiId8PXUhu12O15eXo5lwzCcljtq37Bhg2P9zJkzGT9+/BlfwzAMvL1dz8CamhqX+57JqFGjzmu8XLyqqqq69fV1bMrZePLY9FiAREREsG/fPsdyfX09JpPJqf2nX4I3NDRgMploamoiPz+fe+65BzgZEj4+PsDJWUxDQwMRERHYbDZaWloICQlxuabIyEgCAgLOb8dEzkBv4NJTnc+x2dra2uEHb4+dwoqNjaWyspLGxkaOHz9OWVkZcXFxjvYBAwYQEBDgSMeioiLi4uIIDAzk+eef56OPPgJg8+bNjhnI2LFjKSwsBKCkpITo6Gj8/Pw8tQsiItIBj81AwsPDyc7OJi0tDavVyuTJk4mKiiIjI4OsrCxGjBjB6tWrmT9/Ps3NzQwfPpy0tDR8fHzIy8vj0Ucf5cSJE1xxxRWsWrUKgLlz55Kbm8ukSZMIDg5m9erVnipfREQ64WUYhtHdRXjaqWlYV5zC+u3sJV1UlVwsdm9a0N0lABC/7ZHuLkF6mLIpK85rfGfvnboSXURE3KIAERERtyhARETELQoQERFxiwJERETcogARERG3KEBERMQtChAREXGLAkRERNyiABEREbcoQERExC0KEBERcYsCRERE3KIAERERtyhARETELQoQERFxiwJERETcogARERG3eDRAiouLSUxMJD4+ni1btpzWfuDAAVJSUkhISGDevHnYbDYAqqqqmDx5MmazmfT0dGprawHYs2cPo0ePxmw2YzabeeQRPcJTRKS7eCxALBYL69atY+vWrRQWFvLSSy/x5ZdfOvXJyclh4cKF7Ny5E8Mw2L59u2P90qVLKSoqIjk5maVLlwJQU1PDjBkzKCoqoqioiBUrzu95vyIi4j6PBUhFRQUxMTGEhIQQGBhIQkICpaWljvba2lpOnDjByJEjAUhJSaG0tJS2tjbmzp3LsGHDABg6dCiHDh0CoLq6mnfffZfk5GQyMzMd60VE5JfnsQCpq6sjLCzMsWwymbBYLGdtDwsLw2Kx4O/vj9lsBsBut/PUU09xyy23ABAcHMzdd99NcXExY8eOJTs721Pli4hIJ3w9tWG73Y6Xl5dj2TAMp+XO2tva2sjNzcVmszF79mwAFi9e7GifOnUqa9asoampieDgYJdqqqmpcXt/AEaNGnVe4+XiVVVV1a2vr2NTzsaTx6bHAiQiIoJ9+/Y5luvr6zGZTE7t9fX1juWGhgZHe0tLC/feey8hISFs3LgRPz8/7HY7mzZtYtasWfj4+DjG/fTvzkRGRhIQEHA+uyVyRnoDl57qfI7N1tbWDj94e+wUVmxsLJWVlTQ2NnL8+HHKysqIi4tztA8YMICAgABHOhYVFTnac3JyGDRoEHl5efj7+58s1NubXbt2sXPnTgAKCwu55pprCAwM9NQuiIhIBzw2AwkPDyc7O5u0tDSsViuTJ08mKiqKjIwMsrKyGDFiBKtXr2b+/Pk0NzczfPhw0tLS+PTTT3nzzTcZPHgwt99+O3Dy+5PnnnuOxx9/nAULFrBhwwZCQ0NZtWqVp8oXEZFOeBmGYXR3EZ52ahrWFaewfjt7SRdVJReL3ZsWdHcJAMRv03VR4qxsyvld6tDZe6euRBcREbcoQERExC0KEBERcYsCRERE3KIAERERtyhARETELQoQERFxiwJERETcogARERG3KEBERMQtChAREXGLAkRERNyiABEREbcoQERExC0KEBERcYsCRERE3KIAERERt3g0QIqLi0lMTCQ+Pp4tW7ac1n7gwAFSUlJISEhg3rx52Gw2AL777jumT5/OhAkTuPfee2lpaQHg2LFjzJo1i4kTJzJ9+nTq6+s9Wb6IiHTAYwFisVhYt24dW7dupbCwkJdeeokvv/zSqU9OTg4LFy5k586dGIbB9u3bAXjssceYNm0apaWlREZG8vTTTwOQl5dHdHQ0r7/+OqmpqSxbtsxT5YuISCc8FiAVFRXExMQQEhJCYGAgCQkJlJaWOtpra2s5ceIEI0eOBCAlJYXS0lKsVit79+4lISHBaT1AeXk5ycnJACQlJfHOO+9gtVo9tQsiItIBX09tuK6ujrCwMMeyyWTi448/Pmt7WFgYFouFI0eOEBQUhK+vr9P6n4/x9fUlKCiIxsZGwsPDO6zFMAwA2traznu/QoN6nfc25OLS2tra3SUAEOIb2N0lSA9zvsfmqffMU++hP+exALHb7Xh5eTmWDcNwWj5b+8/7Aact/3SMt3fnk6hTs5TPP//8nPbhTBZOu+m8tyEXl5qamu4uAYCHr0zs7hKkh+mqY9NqtXLZZZedtt5jARIREcG+ffscy/X19ZhMJqf2n34J3tDQgMlkIjQ0lKamJtrb2/Hx8XEaZzKZaGhoICIiApvNRktLCyEhIZ3W0rt3b6666ir8/PzOGkYiIuLMMAysViu9e/c+Y7vHAiQ2Npb169fT2NhIr169KCsrY8mSJY72AQMGEBAQQFVVFaNGjaKoqIi4uDj8/PyIjo6mpKSE5ORkCgsLiYuLA2Ds2LEUFhaSmZlJSUkJ0dHR+Pn5dVqLt7c3wcHBntpVEZGL1plmHqd4GWc7udUFiouL2bRpE1arlcmTJ5ORkUFGRgZZWVmMGDGCzz77jPnz59Pc3Mzw4cNZsWIF/v7+1NbWkpuby+HDh+nfvz9r166lT58+HD16lNzcXA4ePEhwcDCrV69m4MCBnipfREQ64NEAERGRi5euRBcREbcoQERExC0KEBERcYsCRERE3KIAERERtyhARETELQqQC1BzczOPPfYYSUlJmM1m7r77bj755JPuLguA6upq5s2b55Ftv/XWWwwdOrTH3DpETnepHZsFBQVcf/31mM1mbr31VhITEykpKenS1+jJdB3IBcZutzN9+nRGjx7NnDlz8PX15b333uP+++/ntddeo2/fvt1dosfMmTMHHx8ffvWrXznd1UB6hkvx2CwoKGDPnj2sXLkSOHnLpoSEBN5+++1L4u4XHruViXjG+++/z6FDh8jKynLcSDImJoYVK1Zgt9t55plneOWVV/Dx8WHMmDHk5ORw6NAh5syZw5AhQzhw4AD9+vXjySef5JVXXuGbb75hwYIFAKxcuZKIiAhSU1NZvHgxX3zxBe3t7WRkZJCUlERBQQE7duzg6NGj3HTTTQwZMoTnn38eHx8fBg4cyBNPPMH+/ft56qmn+Pvf/85XX33FwoULOXr0KIGBgcybN4+oqChyc3MJCgrik08+wWKxcN9993HHHXd0uN+NjY289957FBYWctttt/Hwww8TFBTk8X9vcd2lemz+VEtLC4GBgQQEBHjk37jHMeSC8vzzzxuzZ88+Y1t5ebmRmppq/Pjjj4bVajUyMzONzZs3GwcPHjSGDh1qfPLJJ4ZhGMacOXOMF154wTh8+LDx29/+1rDZbIbdbjduuukmo66uznjiiSeMv/3tb4ZhGEZTU5MxadIk49tvvzXy8/ON8ePHG1ar1TAMw7j55puNhoYGwzAMY+XKlcann35qvPfee8Zdd91lGIZh3HHHHcbOnTsNwzCMDz/80LjxxhuN1tZW4+GHHzbuu+8+w263G5999plx/fXXd7rff/3rX42srCzDMAwjPT3d2LJly3n8K4onXIrHZn5+vnHdddcZt956qzFp0iRj+PDhRl5e3vn/Y14gNAO5wHh7e5/10817773HpEmT6NXr5DNL7rjjDgoLCxk7diz9+vXj6quvBmDIkCH88MMPhIaGMmzYMN5//338/Py48sorCQsLo6KighMnTpCfnw/Ajz/+yBdffAHA1Vdf7XhWy0033cTUqVO55ZZbSEhI4N///d95//33gZOfxL799lvi4+MBGDlyJH369OGf//wnAGPGjMHLy4urrrqKo0ePdrrfO3bsYM6cOQAkJiayefNmpk2b5s4/oXjIpXps3nzzzY5TWHV1ddx111385je/ISkpyZ1/xguKAuQCExkZydatW097bsratWuprKzk9ttvd+p/6jnzP/0f+9RzVwDMZjMlJSX4+fk5nvZot9t54oknGD58OHDyVvt9+vShuLjY6c6c8+fP57PPPuPtt98mJyeHOXPmEBERAZz5ATSGYdDe3u5Ujyu31//kk0/4/PPPWbZsGStWrKC9vZ26ujr279/veKKldL9L8dj8OZPJxI033sgHH3xwSQSIfoV1gYmOjqZfv3489dRTjgN+9+7dFBQUkJ6ezmuvvcaJEyew2Wzk5+cTExPT4fbGjRvH3r17+d///V/Gjx8PnDxv/eKLLwInP1HdeuutHDp0yGmczWYjPj6evn37Mnv2bMxmMwcOHHC0BwUFMXDgQMrKygDYv38/DQ0NDBky5Jz3uaCggDvvvJPy8nLeeust3n77bcxmM9u2bTvnbYnnXIrH5s+1tbXxwQcfOGZUFzvNQC4wXl5ePP3006xYsYKkpCR8fX3p27cvzz77LFdffTWHDh3ijjvuwGazccMNN3DXXXfx/fffn3V7l112Gddeey1tbW2Oh8bMmTOHRx99lKSkJNrb28nJyeFf//VfnR4Q5uvrS1ZWFjNmzCAgIIB+/fqxcuVKvvzyS0efJ554gkcffZT169fj5+fH+vXr8ff3P6f9bWtr49VXX+WFF15wWn/PPffwu9/9jkceeYQ+ffqc0zbFMy61Y/OUt956C7PZjJeXF21tbcTGxpKSkuLWti40+hmviIi4RTMQ6RFKSkrYtGnTGduKiop+4WpE/j8dm2enGYiIiLhFX6KLiIhbFCAiIuIWfQci0oVefvllXnrpJVpaWmhra+Nf/uVf+M///E+uueaa7i5NpMspQES6yNq1a9m7dy95eXkMGDAAgMrKSmbPnk1BQQG//vWvu7lCka6lL9FFukBDQwPjxo1j165dmEwmp7bCwkJGjBhBRkYGUVFR/N///R/3338/V1xxBYsXL+bo0aN4eXkxY8YMbrvtNt5//32WLFnCq6++CuC0vH79er755hu+//576uvrGTZsGMuWLdONJaVbaAYi0gX279/Pb37zm9PCA+C2225z/D1kyBDy8vKw2WxMmDCBhx56iPj4eCwWC6mpqQwaNKjT19q7dy/5+fmEhoaSk5PDhg0bePjhh7tyd0Rcoi/RRbrAzyfyzc3NmM1mzGYz48ePZ+3atcDJ230AfP3117S2tjpu6BceHk58fDy7d+/u9LUmTJjA5Zdfjre3N5MnT+bdd9/t4r0RcY1mICJdICoqiq+++oojR47Qt29fgoKCHBeZrV+/niNHjgAQGBgIQHt7+2k36zMMA5vN5nRDQQCr1erUz8fHx/G33W53PHtD5JemI0+kC4SHh5OWlsbcuXP57rvvHOtra2v54IMPTnuT/7d/+zd8fX0dN/SzWCzs3LmT2NhYQkND+e677zh8+DCGYfDaa685jX3zzTdpamrCbrezfft2brrpJs/voMgZaAYi0kWys7N55ZVXeOCBBzh+/DhNTU306dOHxMREpk+fzltvveXo6+fnx9NPP83SpUtZv3497e3t3HfffY471E6ZMoU77riDsLAwbrzxRqqrqx1jL7/8cjIyMjhy5AjXXXcdmZmZv/i+ioB+hSVyQTl1OmzhwoXdXYqITmGJiIh7NAMRERG3aAYiIiJuUYCIiIhbFCAiIuIWBYiIiLhFASIiIm5RgIiIiFv+H6J3SqgiH2L7AAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<Figure size 432x288 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "sns.barplot(x = ab_test['Group'], y = ab_test['Conversion'], ci = False, palette = 'viridis')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Testing the hypothesis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "t-value: 3.7873679309192925\n",
      "p-value: 0.00018189800641437955\n"
     ]
    }
   ],
   "source": [
    "control_results = ab_test[ab_test['Group'] == 'Conversion_A']['Conversion']\n",
    "treatment_results = ab_test[ab_test['Group'] == 'Conversion_B']['Conversion']\n",
    "\n",
    "\n",
    "tval, pval = stats.ttest_ind(a=treatment_results, b=control_results)\n",
    "\n",
    "print(f't-value: {tval}')\n",
    "print(f'p-value: {pval/2}')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Drawing conclusions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Our ***p*-value = 0.0002 is way less than α = 0.05 and the t-value is greater than 0**. This means that we success to reject the null hypothesis, which means that **the avarage conversion rate of the new design is significantly greater than the old one**."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "      "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}