bayesian_classification.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "2gMnY4ESvfEA"
   },
   "source": [
    "# Bayesian Classification"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "7wHXE1ij1bIA"
   },
   "source": [
    "## Objectives"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "5ZLVbMKZ1bIA"
   },
   "source": [
    "* **Understand** Bayes Theorem,\n",
    "* **Learn and Explore** how Bayes theorem can aid in the construction of bayesian models,\n",
    "* **Create** algorithms for bayesian classification,\n",
    "* **Observe** the results of this new models."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "OJ4Teb2m1bIA"
   },
   "source": [
    "## Bayes Theroem, the start:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Bayes' Theorem, formulated by Thomas Bayes, is a fundamental concept in probability and statistics that describes how to update the probability of a hypothesis (or event) based on new evidence. It relates the probability of occurrence of two distinct events in relation to each other and finds wide applications in various fields, including data science, machine learning, medicine, economics, and more.\n",
    "\n",
    "Suppose we have two events: \"$H$\" and \"$E$\". Bayes' Theorem allows us to calculate the conditional probability of \"$H$\" occurring, given that \"$E$\" has already happened, using the following formula **(1)**:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "P(H|E) = \\frac{P(E|H) \\cdot P(H)}{P(E)}\n",
    "\\end{equation}\n",
    "\n",
    "where:\n",
    "\n",
    "$P(H|E)$, also known as \"**Posterior**\", is the probability of the hypothesis $H$ occurring given that evidence $E$ has occurred.\n",
    "\n",
    "$P(E|H)$, also known as \"**Likelihood**\", is the probability of observing evidence $E$ given that hypothesis $H$ has occurred.\n",
    "\n",
    "$P(H)$, also known as \"**Prior**\", is the initial probability of the hypothesis $H$ occurring before considering any evidence.\n",
    "\n",
    "$P(E)$, also known as \"**Marginal Likelihood**\" or \"**Evidence**\", is the probability of observing evidence $E$."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "ST3ronAy1bIA"
   },
   "source": [
    "From this theorem, two significant solutions in the field of machine learning and artificial intelligence were developed: Bayesian Networks (also known as Probability Networks) and Bayesian Classifiers. Do not forget to watch [this video](https://www.youtube.com/watch?v=HZGCoVF3YvM&ab_channel=3Blue1Brown) to get a better understading how to use Bayes theorem.\n",
    "\n",
    "It's important to notice that although neural networks and Bayesian models (such as the Bayesian classifier and Bayesian network) are distinct approaches to tackling machine learning problems, they are interconnected conceptually. Neural networks can be trained using the maximum likelihood method, which is a technique related to the principle of maximum likelihood employed in statistics and probabilistic modeling, including Bayesian methods. Consequently, neural networks can be viewed as universal function approximators and possess the capability to learn underlying probability distributions, despite it not being their primary approach.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iydhbcAqJtFs"
   },
   "source": [
    "## ☆ Challenge #1: Protect the Oranges! (Bayesian Network) ☆ "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A Bayesian network, is a graphical representation of probabilistic relationships between different variables. It's a powerful tool used to model uncertain situations and make predictions between a lot of non-indepent variables.\n",
    "\n",
    "The strength of Bayesian networks lies in their ability to update probabilities when new information is available. Let's say one event occurs, and we want to know how it affects the likelihood of another event happening. Bayesian networks allow us to do this by applying Bayes' theorem, which takes into account the prior probabilities and the new evidence. To make predictions using a Bayesian network, we follow a process called \"inference.\" It involves using the network to calculate the probabilities of events, given the evidence we have. Let's discorver more with a example:\n",
    "\n",
    "The farmer faces concerns about the security of his valuable orange farm during the night while he is **sleeping** because he fears the possibility of **thieves** invading the property to steal his precious fruits. To protect his plantation, he has installed a sensitive **alarm** system that is triggered if it detects the presence of intruders on the farm.\n",
    "\n",
    "However, the farmer realizes that the alarm can be triggered not only by **thieves** but also by the farm's **sheeps** or **thunders** during the night.\n",
    "\n",
    "In order to make more informed decisions about when to activate the alarm and ensure the **safety** of the oranges, the farmer has decided to use a probabilistic model known as \"Bayesian Network.\" This Bayesian Network consists of five random variables:\n",
    "\n",
    "- **IntruderPresence**: This variable represents the presence of intruders on the farm during the night. It can take two possible states: \"True\" if intruders are present and \"False\" if the farm is secure.\n",
    "\n",
    "- **SheepPresence**: This variable represents the presence of sheep on the farm during the night. Sheep roaming around could potentially trigger the alarm. It can take two states: \"True\" if sheep are present and \"False\" if there are no sheep.\n",
    "\n",
    "- **Thunder**: This variable represents the occurrence of thunder during the night. Thunder can also trigger the sensitive alarm system. It can take two states: \"True\" if there is thunder and \"False\" if there is no thunder.\n",
    "\n",
    "- **AlarmTriggered**: This variable represents whether the alarm system is triggered or not. It can take two states: \"True\" if the alarm is triggered and \"False\" if it remains inactive.\n",
    "\n",
    "- **FarmerWakeUp**: This variable represents the farmer's decision to wake up and defend the oranges in the event of a potential threat. It can take two states: \"True\" if the farmer wakes up and defends the oranges, and \"False\" if the farmer remains asleep.\n",
    "\n",
    "- **OrangeSafety**: This variable represents the safety of the oranges on the farm. It can take two states: \"1\" if the oranges are secure and \"0\" if there is a potential threat to the oranges.\n",
    "\n",
    "Our task is to develop a solution to predict the probability of each event within the Bayesian Network, to do that, the following were probabilities collected by the farmer, using his dataset ```./datasets/night_farm_dataset.csv```. Now, it's yout challenge to:\n",
    "\n",
    "\n",
    "**a)** Model the Bayesian Network for the problem.\n",
    "\n",
    "**b)** Based on the network created in **a)**, construct the useful tables for relating variables states and probabilities.\n",
    "\n",
    "**c)** Calculate the probabilities for each variable, with no prior evidence.\n",
    "\n",
    "**d)** What is the probability that the alarm has been activated due to a sheep in the plantation? Additionally, what is the probability of the farmer waking up if there is an intruder at the farm?\n",
    "\n",
    "**e)** Verify your results at [this](http://bayesjs.github.io/bayesjs-editor/) link.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8_Ch1ZPO1bIC"
   },
   "source": [
    "### ☆ Solution ☆"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**a)** We can start modeling the network using any node. It is possible to model a Bayesian network by choosing an initial random variable and establishing the relationships on a case-by-case basis. However, it is worth noting that there is not just one way to model a Bayesian network. Below, we present the most intuitive way to model it for this problem:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<div align=\"center\">\n",
    "\t<img src = \"./assets/network_1.png\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice that the intruders, the thunder, and the sheeps are exclusively linked to the alarm, as if the alarm goes on, one of them will be the cause. Similarly, the alarm and the farmer waking up is exclusively connected to the Orange safety, as if the farmer woke up and the alarm goes on, oranges are not in rink anymore. With data collected and the network done, we can start calculating the probabilities in each scenario."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** To construct the tables, we should create them based on the modeled Bayesian network, linking the variables according to the hierarchy determined in the network. However, before that, we need to analyze the dataset to collect the initial probabilities for each table, using pandas we can start calculating marginal probabilities for the independent variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "\n",
    "df = pd.read_csv('./datasets/night_farm_dataset.csv')\n",
    "    \n",
    "print('IntruderPresence marginal probabilities:')\n",
    "p_IntruderPresence = df['IntruderPresence'].value_counts(normalize=True).to_dict()\n",
    "print(p_IntruderPresence)\n",
    "print()\n",
    "\n",
    "print('SheepPresence marginal probabilities:')\n",
    "p_SheepPresence = df['SheepPresence'].value_counts(normalize=True).to_dict()\n",
    "print(p_SheepPresence)\n",
    "print()\n",
    "\n",
    "print('Thunder marginal probabilities:')\n",
    "p_Thunder = df['Thunder'].value_counts(normalize=True).to_dict()\n",
    "print(p_Thunder)\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After that, calculate the conditional probabilities for non-independent variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# AlarmTriggered conditional probability\n",
    "\n",
    "p_AlarmTriggered = {}\n",
    "print('AlarmTriggered probabilities:')\n",
    "for intruder in [True, False]:\n",
    "    for sheep in [True, False]:\n",
    "        for thunder in [True, False]:\n",
    "            sub_df = df[(df.IntruderPresence == intruder) & \n",
    "                        (df.SheepPresence == sheep) & \n",
    "                        (df.Thunder == thunder)]\n",
    "            key = f'Intruder: {intruder}, Sheep: {sheep}, Thunder: {thunder}'\n",
    "            p_AlarmTriggered[key] = sub_df.AlarmTriggered.value_counts(normalize=True).to_dict()\n",
    "            print(key)\n",
    "            print(p_AlarmTriggered[key])\n",
    "            print()\n",
    "\n",
    "# FarmerWakeUp conditional probability \n",
    "\n",
    "p_FarmerWakeUp = {}\n",
    "print('FarmerWakeUp probabilities:')\n",
    "for alarm in [True, False]:\n",
    "    key = f'Alarm: {alarm}'    \n",
    "    p_FarmerWakeUp[key] = df.FarmerWakeUp[df.AlarmTriggered == alarm].value_counts(normalize=True).to_dict()\n",
    "    print(key)\n",
    "    print(p_FarmerWakeUp[key])\n",
    "    print()\n",
    "\n",
    "# OrangeSafety conditional probability \n",
    "\n",
    "p_OrangeSafety = {}\n",
    "print('OrangeSafety probabilities:')\n",
    "for intruder in [True, False]:\n",
    "    for wake_up in [True, False]:\n",
    "        sub_df = df[(df.IntruderPresence == intruder) & (df.FarmerWakeUp == wake_up)]\n",
    "        key = f'Intruder: {intruder}, WakeUp: {wake_up}'\n",
    "        p_OrangeSafety[key] = sub_df.OrangeSafety.value_counts(normalize=True).to_dict()\n",
    "        print(key)\n",
    "        print(p_OrangeSafety[key]) \n",
    "        print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the end, you're gonna end up with this tables (values were round off):"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "| IntruderPresence | Probability |\n",
    "|:----------------:|:-----------:|\n",
    "| True             |    0.05     |\n",
    "| False            |    0.95     |\n",
    "\n",
    "\n",
    "| SheepPresence | Probability |\n",
    "|:------------:|:-----------:|\n",
    "|     True     |    0.30     |\n",
    "|    False     |    0.70     |\n",
    "\n",
    "|  Thunder  | Probability |\n",
    "|:---------:|:-----------:|\n",
    "|   True    |    0.10     |\n",
    "|   False   |    0.90     |\n",
    "\n",
    "| IntruderPresence | SheepPresence | Thunder | Alarm(True) | Alarm(False) |\n",
    "|:----------------:|:-------------:|:-------:|:-----------:|:-----------:|\n",
    "|       True       |     True      |   True  |    0.95     |    0.05     |\n",
    "|       True       |     True      |  False  |    0.94     |    0.06     |\n",
    "|       True       |    False      |   True  |    0.93     |    0.07     |\n",
    "|       True       |    False      |  False  |    0.92     |    0.08     |\n",
    "|      False       |     True      |   True  |    0.80     |    0.20     |\n",
    "|      False       |     True      |  False  |    0.75     |    0.25     |\n",
    "|      False       |    False      |   True  |    0.60     |    0.40     |\n",
    "|      False       |    False      |  False  |    0.01     |    0.99     |\n",
    "\n",
    "\n",
    "| Alarm | FarmerWakeUp(True) | FarmerWakeUp(False) |\n",
    "|:-----:|:-----------------:|:------------------:|\n",
    "|  True |       0.80        |       0.20         |\n",
    "|  False |       0.05        |       0.95         |\n",
    "\n",
    "| IntruderPresence | FarmerWakeUp | OrangeSafety(Safe) | OrangeSafety(NotSafe) | \n",
    "|:----------------:|:------------:|:------------------:|:--------------------:|\n",
    "|      False       |     True     |        1.00        |         0.00         |\n",
    "|       True       |    False     |        0.00        |         1.00         |\n",
    "|       True       |     True     |        1.00        |         0.00         |\n",
    "|      False       |    False     |        1.00        |         0.00         |\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** This time, using the probabilities calculated at **b)** item, we can calculate the probabilities for OrangeSafety, FarmerWakeUp and AlarmTriggered for no prior evidence using the Law of Total Probability:\n",
    "\n",
    "\\begin{equation}\n",
    "P(A) = \\sum_{i=1}^{n} P(A|B_i) \\cdot P(B_i)\n",
    "\\end{equation}\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "print('AlarmTriggered no prior evidence probability:')\n",
    "\n",
    "nep_AlarmTriggered = 0\n",
    "for intruder in [True, False]:\n",
    "    for sheep in [True, False]:\n",
    "        for thunder in [True, False]:\n",
    "            value = p_AlarmTriggered[f'Intruder: {intruder}, Sheep: {sheep}, Thunder: {thunder}']\n",
    "            nep_AlarmTriggered += value[True] * p_IntruderPresence[intruder] * p_SheepPresence [sheep] * p_Thunder[thunder]\n",
    "\n",
    "print(f'True: {nep_AlarmTriggered}')\n",
    "print(f'False: {1 - nep_AlarmTriggered}')\n",
    "print()\n",
    "\n",
    "print('FarmerWakeUp no prior evidence probability:')\n",
    "\n",
    "\n",
    "nep_FarmerWakeUp = {True: 0, False: 0}\n",
    "\n",
    "value = p_FarmerWakeUp[f'Alarm: {True}']\n",
    "nep_FarmerWakeUp[True] += value[True] * nep_AlarmTriggered\n",
    "print(nep_FarmerWakeUp[True])\n",
    "value = p_FarmerWakeUp[f'Alarm: {False}']\n",
    "nep_FarmerWakeUp[True] += value[True] * (1-nep_AlarmTriggered)\n",
    "\n",
    "nep_FarmerWakeUp[False] = 1 - nep_FarmerWakeUp[True]  \n",
    "\n",
    "print(f'True: {nep_FarmerWakeUp[True]}')\n",
    "print(f'False: {nep_FarmerWakeUp[False]}')\n",
    "print()\n",
    "\n",
    "print('OrangeSafety no prior evidence probability:')\n",
    "p_OrangeSafety[f'Intruder: {True}, WakeUp: {False}'][True] = 0\n",
    "\n",
    "nep_OrangeSafety = 0\n",
    "for intruder in [True, False]:\n",
    "    for wake_up in [True, False]:\n",
    "        value = p_OrangeSafety[f'Intruder: {intruder}, WakeUp: {wake_up}']\n",
    "        nep_OrangeSafety += value[True] * p_IntruderPresence[intruder] * nep_FarmerWakeUp[wake_up]\n",
    "\n",
    "print(f'True: {nep_OrangeSafety}')\n",
    "print(f'False: {1 - nep_OrangeSafety}')\n",
    "print()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d)** At the first scenario, we can use Bayes Theorem to update our beliefes about other variables with this new evidence:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Sheep|Alarm = True) = \\frac{P(Alarm|Sheep = True) \\cdot P(Sheep)}{P(Alarm)} = \\frac{P(Alarm \\cap Sheep)}{P(Alarm)}\n",
    "\\end{equation}\n",
    "\n",
    "At the second scenario:\n",
    "\n",
    "\\begin{equation}\n",
    "P(Farmer|Intruder = True) = \\frac{P(Intruder|Farmer = True) \\cdot P(Farmer)}{P(Intruder)} = \\frac{P(Farmer \\cap Intruder)}{P(Intruder)}\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "alarm_sheep = 0\n",
    "for intruder in [True, False]:\n",
    "    for sheep in [True]:\n",
    "        for thunder in [True, False]:\n",
    "            value = p_AlarmTriggered[f'Intruder: {intruder}, Sheep: {sheep}, Thunder: {thunder}']\n",
    "            alarm_sheep += value[True] * p_IntruderPresence[intruder] * p_SheepPresence [sheep] * p_Thunder[thunder]\n",
    "            \n",
    "p_1 = alarm_sheep / nep_AlarmTriggered\n",
    "print(p_1)\n",
    "\n",
    "alarm_intruder = 0\n",
    "for intruder in [True]:\n",
    "    for sheep in [True, False]:\n",
    "        for thunder in [True, False]:\n",
    "            for alarm in [True, False]:\n",
    "                value = p_AlarmTriggered[f'Intruder: {intruder}, Sheep: {sheep}, Thunder: {thunder}']\n",
    "                alarm_intruder += nep_FarmerWakeUp[True] * value[alarm] * p_IntruderPresence[True] * p_SheepPresence [sheep] * p_Thunder[thunder]\n",
    "\n",
    "p_2 = alarm_intruder/p_IntruderPresence[True]\n",
    "print(p_2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "iydhbcAqJtFs"
   },
   "source": [
    "## Bayesian Classifier"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style='text-align: justify;'>\n",
    "    \n",
    "Also based on Bayes' Theorem, the Bayesian Classifier classifier is a widely used supervised machine learning algorithm for categorizing objects into distinct categories based on attributes or features. The most simple variation it is called Naive Bayesian Classifier because it's different from the Bayesian Network, it makes a simplified assumption of independence between the variables of the problem, which simplifies the calculation of probabilities.\n",
    "\n",
    "<div align=\"center\">\n",
    "\t<a href=\"rep_icon\">\n",
    "\t<img height = \"400em\" src = \"./assets/bayes.png\" />\n",
    "    </a>\n",
    "</div>\n",
    "\n",
    "The classifier receives a training set and then calculates the probability of an example belonging to each class, based on the frequency of features in the training examples belonging to each class. From there, and assumes the assumption of conditional independence of variables (or features) given the class value. This means that, given the value of class $C_k$, the features $X_1, X_2, \\ldots, X_n$ are considered independent of each other. Described as the Naive Bayes Assumption **(5)**:\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "P(X_1=x_1, X_2=x_2, \\ldots, X_n=x_n | C_k) = \\prod_{i=1}^{n} P(X_i=x_i | C_k)\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4j4pF7PHJzj3"
   },
   "source": [
    "<p style='text-align: justify;'>\n",
    "   \n",
    "This assumption greatly simplifies the process of calculating probabilities and makes the algorithm more efficient, but it may not reflect reality in some cases. With this in mind, the classifier uses Bayes' Theorem to calculate the probability of belonging to each class. It then selects the class with the highest probability as the final classification. By calculating the probabilities for each class, the classifier assigns the test example to the class with the highest probability, like describe in this formula **(6)**:\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\\begin{equation}\n",
    "\\text{Class: } \\arg\\max_{C_k} P(C_k) \\prod_{i=1}^{n} P(x_i | C_k)\n",
    "\\end{equation}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<p style='text-align: justify;'>\n",
    "\n",
    "Bayesian classifiers are particularly useful in situations where data is available for training the classifier, such as in spam detection in emails. In this case, the classifier can be trained using a set of labeled emails (spam and non-spam), where features of the emails, such as keywords, term frequencies, or syntactic characteristics, are extracted. This approach is particularly efficient when dealing with categorical data, such as words in a text. Additionally, spam detection requires a quick response, as unwanted emails need to be filtered in real-time. Trained Bayesian classifiers demonstrate fast computational speed, making them a suitable choice for this type of problem.\n",
    "\n",
    "</p>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "s5kh8V2iLl0e"
   },
   "source": [
    "## ☆ Challenge #2: Plague on orange farm is back. ☆"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "xzdm1mJhNWCX"
   },
   "source": [
    "The same farmer owns an orange farm and is concerned about the possibility of his oranges being affected by an unknown pest. He has decided to use the knowledge of Bayesian Classifier to make informed decisions regarding pest control on his plantation.\n",
    "\n",
    "The farmer noticed that some oranges show suspicious symptoms, while others appear healthy. He is also considering the application of pesticides to control the possible pest infestation. To assist him in his decisions, let's model this situation with a Bayesian Classifiers, with the dataset avaliable in ```./datasets/orange_farm.csv```, that contains the following variables:\n",
    "\n",
    "- **Infestation**: A binary variable indicating whether the oranges are infected with the pest or not. \"1\" represents infected oranges, and \"0\" indicates healthy oranges.\n",
    "- **Symptoms**: A binary variable representing the presence or absence of symptoms in the oranges. \"1\" indicates oranges with suspicious symptoms, and \"0\" represents oranges without visible symptoms.\n",
    "- **Pesticides**: A binary variable indicating whether the farmer applied pesticides to the plantation or not. \"1\" represents the application of pesticides, and \"0\" indicates that no pesticides were used.\n",
    "- **Rainfall**: A numerical variable indicating the amount of recent rainfall in millimeters.\n",
    "- **Soil Moisture**: A numerical variable representing the soil moisture in the plantation as a percentage.\n",
    "- **Temperature**: A numerical variable representing the average temperature in degrees Celsius.\n",
    "- **Management Practices**: A categorical variable indicating the management practices adopted in the plantation, \"Conventional\" and \"Organic\".\n",
    "\n",
    "This is a perfect example of using a Bayesian classifier. All variables do not have any relation between them, except with our \"target\" (**Infestation**). Now, let's address the four parts of the farmer's request:\"\n",
    "\n",
    "**a)** Make an exploratory analysis of the data, in that way the farmer should examine which variables seem to have the greatest influence on pest infestation. For example, he can use data visualization techniques like histograms, scatter plots, and correlation matrices to understand the relationships between variables and the infestation status.After that, model the bayesian network model."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**b)** Calculate the marginal probabilities of infestation (I), symptoms (S), and pesticide application (P), the farmer needs to count the occurrences of each state and divide them by the total number of observations in the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**c)** Determine the conditional probabilities, the farmer needs to calculate the probabilities of pest infestation (I) given the presence of symptoms (S), the application of pesticides (P), and different levels of rainfall (C), soil moisture (U), temperature (T), and management practices (M). He can do this by dividing the number of occurrences of each combination by the total number of observations for the given condition."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**d)** Create a Bayesian Classification model, the farmer can use the conditional probabilities he calculated in part c) to estimate the likelihood of infestation (I) given the values of other variables. He can then use Bayes' theorem to compute the posterior probabilities and make predictions on new data. The farmer should use the test dataset ```./datasets/orange_farm_test.csv``` into training and testing sets to evaluate the model's accuracy. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "8_Ch1ZPO1bIC"
   },
   "source": [
    "### ☆ Solution ☆"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As requested in item **a)**, we can begin the exploratory analysis to help the farmer visualize the data and gain a better understanding of the problem. It is essential to plot the data on graphs and create a correlation matrix with the variables. This will enable a more in-depth comprehension of the situation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "import matplotlib.pyplot as plt\n",
    "import seaborn as sns\n",
    "\n",
    "# Load the dataset\n",
    "data = pd.read_csv('./datasets/plague_farm_dataset.csv')\n",
    "\n",
    "# Calculate the marginal probabilities\n",
    "total_samples = len(data)\n",
    "\n",
    "df = pd.DataFrame(data)\n",
    "\n",
    "print(df.head())\n",
    "print(df.describe())\n",
    "print(df.info())\n",
    "\n",
    "# Set up subplots\n",
    "fig, axes = plt.subplots(2, 2, figsize=(12, 10))\n",
    "\n",
    "# Infestation\n",
    "sns.countplot(x='Infestation', data=df, ax=axes[0, 0])\n",
    "axes[0, 0].set_title('Infestation Distribution')\n",
    "\n",
    "# Symptoms\n",
    "sns.countplot(x='Symptoms', data=df, ax=axes[0, 1])\n",
    "axes[0, 1].set_title('Symptoms Distribution')\n",
    "\n",
    "# Rainfall\n",
    "axes[1, 0].hist(df['Rainfall'], bins=20, edgecolor='k')\n",
    "axes[1, 0].set_xlabel('Rainfall (mm)')\n",
    "axes[1, 0].set_ylabel('Frequency')\n",
    "axes[1, 0].set_title('Rainfall Distribution')\n",
    "\n",
    "# Soil Moisture\n",
    "axes[1, 1].hist(df['Soil Moisture'], bins=20, edgecolor='k')\n",
    "axes[1, 1].set_xlabel('Soil Moisture (%)')\n",
    "axes[1, 1].set_ylabel('Frequency')\n",
    "axes[1, 1].set_title('Soil Moisture Distribution')\n",
    "\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n",
    "fig, axes = plt.subplots(2, 2, figsize=(12, 10))\n",
    "\n",
    "# Temperature\n",
    "axes[0, 0].hist(df['Temperature'], bins=20, edgecolor='k')\n",
    "axes[0, 0].set_xlabel('Temperature (°C)')\n",
    "axes[0, 0].set_ylabel('Frequency')\n",
    "axes[0, 0].set_title('Temperature Distribution')\n",
    "\n",
    "# Pesticides\n",
    "sns.countplot(x='Pesticides', data=df, ax=axes[0, 1])\n",
    "axes[0, 1].set_title('Pesticides Application')\n",
    "\n",
    "# Management Practices\n",
    "sns.countplot(x='Management Practices', hue='Infestation', data=df, ax=axes[1, 0])\n",
    "axes[1, 0].set_title('Management Practices and Infestation')\n",
    "\n",
    "# Compute the correlation matrix\n",
    "correlation_matrix = df[['Infestation', 'Symptoms', 'Pesticides', 'Rainfall', 'Soil Moisture', 'Temperature']].corr()\n",
    "\n",
    "# Plot the correlation matrix as a heatmap\n",
    "sns.heatmap(correlation_matrix, annot=True, cmap='coolwarm', fmt='.2f', ax=axes[1, 1])\n",
    "axes[1, 1].set_title('Correlation Matrix')\n",
    "\n",
    "plt.tight_layout()\n",
    "plt.show()\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Upon initial observation, it becomes evident that there is no correlation among the variables, except for their relationship with infestation. Based on this insight, we can construct the following Bayesian network:\n",
    "\n",
    "\n",
    "\n",
    "<div align=\"center\">\n",
    "\t<img height = \"400em\" src = \"./assets/network_2.png\" />\n",
    "</div>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As requested in item **b)**, we can calculate the marginal probabilities using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Count occurrences of each state for 'Infestation'\n",
    "infestation_counts = data['Infestation'].value_counts()\n",
    "p_infestation = infestation_counts[1] / total_samples\n",
    "\n",
    "# Count occurrences of each state for 'Symptoms'\n",
    "symptoms_counts = data['Symptoms'].value_counts()\n",
    "p_symptoms = symptoms_counts[1] / total_samples\n",
    "\n",
    "# Count occurrences of each state for 'Pesticides'\n",
    "pesticides_counts = data['Pesticides'].value_counts()\n",
    "p_pesticides = pesticides_counts[1] / total_samples\n",
    "\n",
    "# Display the marginal probabilities\n",
    "print(f\"Marginal Probability of Infestation (I): {p_infestation:.2f}\")\n",
    "print(f\"Marginal Probability of Symptoms (S): {p_symptoms:.2f}\")\n",
    "print(f\"Marginal Probability of Pesticide Application (P): {p_pesticides:.2f}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As requested in item **c)**, we can calculate and store the conditional probabilities in using the following code:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "infestation_given_symptoms = data[data['Infestation'] == 1]['Symptoms'].value_counts() / data['Symptoms'].value_counts()\n",
    "\n",
    "infestation_given_pesticides = data[data['Infestation'] == 1]['Pesticides'].value_counts() / data['Pesticides'].value_counts()\n",
    "\n",
    "rainfall_intervals = pd.cut(data['Rainfall'], bins=[0, 50, 100, 150, 200])\n",
    "infestation_given_rainfall = data[data['Infestation'] == 1].groupby(rainfall_intervals)['Infestation'].count() / data.groupby(rainfall_intervals)['Infestation'].count()\n",
    "\n",
    "moisture_intervals = pd.cut(data['Soil Moisture'], bins=[0, 25, 50, 75, 100])\n",
    "infestation_given_moisture = data[data['Infestation'] == 1].groupby(moisture_intervals)['Infestation'].count() / data.groupby(moisture_intervals)['Infestation'].count()\n",
    "\n",
    "temperature_intervals = pd.cut(data['Temperature'], bins=[10, 20, 30, 40])\n",
    "infestation_given_temperature = data[data['Infestation'] == 1].groupby(temperature_intervals)['Infestation'].count() / data.groupby(temperature_intervals)['Infestation'].count()\n",
    "\n",
    "infestation_given_management = data[data['Infestation'] == 1]['Management Practices'].value_counts() / data['Management Practices'].value_counts()\n",
    "\n",
    "print(\"Conditional Probability of Infestation (I) given Symptoms (S):\")\n",
    "print(infestation_given_symptoms)\n",
    "\n",
    "print(\"\\nConditional Probability of Infestation (I) given Pesticides (P):\")\n",
    "print(infestation_given_pesticides)\n",
    "\n",
    "print(\"\\nConditional Probability of Infestation (I) given Rainfall (C):\")\n",
    "print(infestation_given_rainfall)\n",
    "\n",
    "print(\"\\nConditional Probability of Infestation (I) given Soil Moisture (U):\")\n",
    "print(infestation_given_moisture)\n",
    "\n",
    "print(\"\\nConditional Probability of Infestation (I) given Temperature (T):\")\n",
    "print(infestation_given_temperature)\n",
    "\n",
    "print(\"\\nConditional Probability of Infestation (I) given Management Practices (M):\")\n",
    "print(infestation_given_management)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now it's time to begin item **d)**, where we will develop the Bayesian model. First, let's define the function to calculate joint probabilities between the target variable (Infestation) and the selected features (Other variables). It does so by using the information collected at c) item and dividing the occurrences of specific target-feature combinations by the total occurrences of each target value. The resulting joint probabilities provide crucial information for Bayesian analysis and modeling."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Define the features and the target\n",
    "features = ['Symptoms', 'Pesticides', 'Rainfall', 'Soil Moisture', 'Temperature', 'Management Practices']\n",
    "target = 'Infestation'\n",
    "\n",
    "# Function to calculate joint Naive Bayes Assumption formula probabilities\n",
    "def calculate_joint_probabilities(data, features, target):\n",
    "    joint_probabilities = {}\n",
    "    \n",
    "    target_values = data[target].unique()\n",
    "    feature_values = [data[feature].unique() for feature in features]\n",
    "\n",
    "    for target_value in target_values:\n",
    "        # Compute P(target)\n",
    "        p_target = len(data[data[target] == target_value]) / len(data)\n",
    "\n",
    "        # Initialize dictionary for this target value\n",
    "        joint_probabilities[target_value] = {}\n",
    "\n",
    "        for feature, feature_value in zip(features, feature_values):\n",
    "            # Initialize dictionary for this feature\n",
    "            joint_probabilities[target_value][feature] = {}\n",
    "\n",
    "            for value in feature_value:\n",
    "                # Compute P(feature | target)\n",
    "                conditional_probability = len(data[(data[target] == target_value) & (data[feature] == value)]) / len(data[data[target] == target_value])\n",
    "\n",
    "                # Compute joint probability and store it\n",
    "                joint_probabilities[target_value][feature][value] = p_target * conditional_probability\n",
    "                \n",
    "    return joint_probabilities"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After creating the function to calculate probabilities, it's time to train the classifier and test it with a new dataset to assess its performance and compare the results. We can create a classify function takes a new data point (sample) and predicts its class by calculating the joint probabilities for each class and each feature. It then assigns the sample to the class with the highest joint probability."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Calculate joint probabilities\n",
    "joint_probabilities = calculate_joint_probabilities(data, features, target)\n",
    "\n",
    "# Function to classify a new sample\n",
    "def classify(sample, joint_probabilities, features):\n",
    "    target_values = joint_probabilities.keys()\n",
    "    max_prob = -1\n",
    "    predicted_class = None\n",
    "\n",
    "    for target_value in target_values:\n",
    "        prob = 1\n",
    "        for feature in features:\n",
    "            prob *= joint_probabilities[target_value][feature].get(sample[feature], 1e-10)  # Assume probability is 1e-10 if the feature value was not seen in the training set\n",
    "        if prob > max_prob:\n",
    "            max_prob = prob\n",
    "            predicted_class = target_value\n",
    "            \n",
    "    return predicted_class\n",
    "\n",
    "# Load the test dataset\n",
    "test_data = pd.read_csv('./datasets/plague_farm_dataset_test.csv')\n",
    "\n",
    "# Use the classifier to predict the target for each sample in the test set\n",
    "test_data['Predicted Infestation'] = test_data.apply(lambda row: classify(row, joint_probabilities, features), axis=1)\n",
    "\n",
    "# Calculate the accuracy of the classifier\n",
    "accuracy = (test_data['Predicted Infestation'] == test_data['Infestation']).mean()\n",
    "print(f'Accuracy: {accuracy:.2f}')\n",
    "\n",
    "# Count the occurrences of each class in the real and predicted data\n",
    "actual_counts = test_data['Infestation'].value_counts().sort_index()\n",
    "predicted_counts = test_data['Predicted Infestation'].value_counts().sort_index()\n",
    "\n",
    "# Create a DataFrame from these counts\n",
    "counts_df = pd.DataFrame({'Actual': actual_counts, 'Predicted': predicted_counts})\n",
    "\n",
    "# Plot a stacked bar chart\n",
    "counts_df.plot(kind='bar', color=['skyblue', 'orange'])\n",
    "plt.xlabel('Infestation')\n",
    "plt.ylabel('Count')\n",
    "plt.title('Comparison of actual and predicted infestation')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "4FolLqE_1bID"
   },
   "source": [
    "## Clear the Memory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "tXiePpX31bID"
   },
   "source": [
    "Please execute the following cell to clear up the CPU memory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "kMB90WpX1bID",
    "outputId": "2ccc1609-8376-49b6-efc9-df55c0838266"
   },
   "outputs": [],
   "source": [
    "import IPython\n",
    "app = IPython.Application.instance()\n",
    "app.kernel.do_shutdown(True)"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}