5.- Naive Bayes.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Naive Bayes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The naive Bayes approach is based on the premise that the probability of prior events can be a good estimate of the probability of future events. For example, when forecasting the probability\n",
    "of rain for today, we would report on the proportion of prior days with the same weather conditions as today, in which it rained. So, if it rained 4 out of 10 of those days, then we estimate a 40 percent chance of rain today. This approach is useful in several domains and\n",
    "problem areas.\n",
    "\n",
    "\n",
    "The naive Bayes method is named after 18th century clergy and mathematician Thomas\n",
    "Bayes who developed mathematical principles for describing the probability of events\n",
    "and how those probabilities are to be revised in light of additional information. Those\n",
    "foundational mathematical principles are known today as Bayesian methods. Applied to\n",
    "machine learning, an event is the expected outcome (or class) such as “true” or “false,”,  “yes” or “no”. A classifier based on Bayesian methods is one that\n",
    "attempts to predict the class of unlabeled data by answering this question: “Based on\n",
    "prior evidence, what is the most likely class of a new unlabeled instance?” It does this by\n",
    "doing the following:\n",
    "- Finding all existing instances with the same feature values (or profile) as the\n",
    "unlabeled instance.\n",
    "- Determining the most likely class that those instances belong to.\n",
    "- Assigning the identified class label to the unlabeled instance.\n",
    "This classification approach uses the concept of conditional probability to determine\n",
    "the most likely class of an instance.\n",
    "\n",
    "\n",
    "Reference:\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Probability\n",
    "\n",
    "The probability of an event is how likely the event is to happen. Since most events cannot\n",
    "be predicted with total certainty, the chance that an event will occur is often described\n",
    "in terms of the probability of the event. For example, when a coin is tossed, there are\n",
    "two possible outcomes: heads or tails. The probability of one of those outcomes, heads\n",
    "for example, is the number of outcomes we care about (heads) divided by the total\n",
    "number of possible outcomes (heads or tails). Therefore, the probability of heads is $\\frac{1}{2}$.\n",
    "The mathematical notation for this is $P(head)=\\frac{1}{2}$\n",
    "\n",
    "\n",
    "# Conditional Probability\n",
    "\n",
    "For dependent events, instead of simply evaluating the probability that events A and\n",
    "B occurred, we determine the probability of event A given that event B occurred. This is\n",
    "known as conditional probability, because the probability of event A is conditioned on the\n",
    "probability of event B. The notation for this is P AB | , which reads the probability of A\n",
    "given B. This relationship can be represented using Bayes theorem, which describes the\n",
    "relationship between dependent events A and B as follows:\n",
    "\n",
    "\\begin{equation*}\n",
    "P(A|B) = \\frac{P(A)P(B \\cap A)}{P(B)}\n",
    "\\end{equation*}\n",
    "\n",
    "There are four parts to this formula. The first part is the conditional probability of A\n",
    "given that B occurred. This is written as $P(A|B)$ and is known as the posterior probability. The second part of the Bayes formula is known as the prior probability. It is written\n",
    "asP A and describes the probability of event A by itself $P(A)$.\n",
    "\n",
    "The next part of the Bayes formula represents the inverse of the posterior probability.\n",
    "It is the probability of B given that A occurred. This is known as the likelihood and is\n",
    "written as $P(B|A)$ . The fourth part of the Bayes formula is called the marginal likelihood. It represents the probability of event B alone and is written as $P(B)$.\n",
    "\n",
    "\n",
    "# Classification\n",
    "Suppose our dataset consists of $n$ predictors denoted as $x1$, $x_2$, $x_3$, $...$, $x_n$, and $m$ distinct class values, which are represented as $C_1$, $C_2$, $...$, $C_m$; then using the Bayes theorem, the conditional probability that an instance belongs to class $C_k$ is denoted as follows:\n",
    "\n",
    "\\begin{equation*}\n",
    "P(C|x_1,x_2,...,x_n)  =  \\frac{P(C)P(x_1,x_2,...,x_n|C)}{P(x_1,x_2,...,x_n)}\n",
    "\\end{equation*}\n",
    "\n",
    "assuming independence\n",
    "\\begin{align*}\n",
    "P(C|x_1,x_2,...,x_n) & =  \\frac{P(C)P(x_1|C)P(x_2|C),...,P(x_n|C)}{P(x_1,x_2,...,x_n)} \\\\\n",
    "P(C|x_1,x_2,...,x_n) & =  \\frac{P(C) \\prod_{i=1}^{n}P(x_i|C)}{P(x_1,x_2,...,x_n)}\n",
    "\\end{align*}\n",
    "\n",
    "Considering a binary classification problem, if we have \n",
    "\n",
    "\\begin{equation*}\n",
    "\\frac{P(C_1) P(x_1|C_1)P(x_2|C_1)}{P(x_1,x_2)} >\\frac{P(C_2) P(x_1|C_2)P(x_2|C_2)}{P(x_1,x_2)}\n",
    "\\end{equation*}\n",
    "\n",
    "Then \n",
    "\\begin{equation*}\n",
    "P(C_1) P(x_1|C_1)P(x_2|C_1) > P(C_2) P(x_1|C_2)P(x_2|C_2)\n",
    "\\end{equation*}\n",
    " "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example:\n",
    "\n",
    "Suppose a tennis team has recorded certain characteristics of the days that are good for playing as well as those that are not. Among the characteristics are: the outlook, temperature, humidity and wind. \n",
    "\n",
    "\n",
    "| Outllok(x_1)  | Temperature(x_2) | Humidity(x_3) | Windy(x_4)  | Clase(C)\n",
    "| --- | --- | --- | --- | --- |\n",
    "| sunny    |  Hot        | High    | False   | No  |\n",
    "| sunny    |  Hot        | High    | Frue    | No  |\n",
    "| Overcast |  Hot        | High    | False   | Yes |\n",
    "| Rain     |  Mild       | High    | False   | Yes |\n",
    "| Rain     |  Cool       | Normal  | False   | Yes |\n",
    "| Rain     |  Cool       | Normal  | True    | No  |\n",
    "| Overcast |  Cool       | Normal  | True    | Yes |\n",
    "| Sunny    | Mild        | High    | False   | No  |\n",
    "| Sunny    | Cool        | Normal  | False   | Yes |\n",
    "| Rain     | Mild        | Normal  | False   | Yes |\n",
    "| Sunny    | Mild        | Normal  | True    | Yes |\n",
    "| Overcast | Mild        | High    | True    | Yes |\n",
    "| Overcast | Hot         | Normal  | False   | Yes |\n",
    "| Rain     | Mild        | High    | True    | No  |\n",
    "\n",
    "Based on the above, the team would like to know, given today's conditions (Outlook=rain, Temperature=hot, Humidity=high and Windy=false), whether or not the game will be played.\n",
    "\n",
    "The procedure consists in:\n",
    "- Calculate the a posteriori probabilities, and\n",
    "- Assign the pattern to the class $C_i$ where P($C_i$|x) is greater.\n",
    "\n",
    "## Solution:\n",
    "\n",
    "\n",
    "Outlook:\n",
    "\n",
    "\n",
    "- $P(sunny|yes) = 2/9$,  $P(sunny|no) = 3/5$ \n",
    "- $P(overcast|yes) = 4/9$,  $P(overcast|no) = 0/5 = 0$\n",
    "- $P(Rain|yes) = 3/9$,  $P(Rain|no) = 2/5$\n",
    "\n",
    "\n",
    "Temperature:\n",
    "\n",
    "- $P(hot|yes) = 2/9$,  $P(hot|no) = 2/5$\n",
    "- $P(mild|yes) = 4/9$,  $P(mild|no) = 1/5$\n",
    "- $P(cool|yes) = 3/9$,   $P(cool|no) = 1/5$\n",
    "\n",
    "Humidity:\n",
    "\n",
    "- $P(high|yes) = 3/9$,   $P(high|no) = 4/5 $\n",
    "- $P(normal|yes) = 6/9$,  $P(normal|no) = 1/5 $\n",
    "\n",
    "Windy:\n",
    "\n",
    "- $P(false|yes) = 6/9$, $P(false|no) = 2/5$\n",
    "- $P(true|yes) = 3/9$,  $P(true|no) = 3/5$\n",
    "\n",
    "Total:\n",
    "\n",
    "- $P(yes) = 9/14$, $P(false) = 5/14$\n",
    "\n",
    "then, given the condition: Outlook=rain, Temperature=hot, Humidity=high and Windy=false,\n",
    "\n",
    "\\begin{align*}\n",
    "P(yes|x) & =  P(x|yes)P(yes) \\\\\n",
    "P(yes|x) & =  P(rain|yes)P(hot|yes)P(high|yes)P(false|yes)P(yes) \\\\\n",
    "P(yes|x) & =  \\frac{3}{9} \\frac{2}{9} \\frac{3}{9} \\frac{6}{9} \\frac{9}{14} \\\\\n",
    "P(yes|x) & =  0.01058 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "and, \n",
    "\n",
    "\\begin{align*}\n",
    "P(no|x) & =  P(x|no)P(no) \\\\\n",
    "P(no|x) & =  P(rain|no)P(hot|no)P(high|no)P(false|no)P(no) \\\\\n",
    "P(no|x) & =  \\frac{2}{5} \\frac{2}{5} \\frac{4}{5} \\frac{2}{5} \\frac{5}{14} \\\\\n",
    "P(no|x) & =  0.018 \\\\\n",
    "\\end{align*}\n",
    "\n",
    "Since $0.018 > 0.010$ then, the test pattern is assigned to class $No$, which means that the game will not be played.\n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example with R"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "#install.packages(\"rsample\")\n",
    "library('rsample')  # data splitting \n",
    "library(\"ISLR\")\n",
    "library(\"caTools\")\n",
    "library(\"dplyr\")\n",
    "library(\"class\") # knn\n",
    "library(\"ggplot2\")\n",
    "options(repr.plot.width = 16, repr.plot.height = 6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "'data.frame':\t5822 obs. of  86 variables:\n",
      " $ MOSTYPE : num  33 37 37 9 40 23 39 33 33 11 ...\n",
      " $ MAANTHUI: num  1 1 1 1 1 1 2 1 1 2 ...\n",
      " $ MGEMOMV : num  3 2 2 3 4 2 3 2 2 3 ...\n",
      " $ MGEMLEEF: num  2 2 2 3 2 1 2 3 4 3 ...\n",
      " $ MOSHOOFD: num  8 8 8 3 10 5 9 8 8 3 ...\n",
      " $ MGODRK  : num  0 1 0 2 1 0 2 0 0 3 ...\n",
      " $ MGODPR  : num  5 4 4 3 4 5 2 7 1 5 ...\n",
      " $ MGODOV  : num  1 1 2 2 1 0 0 0 3 0 ...\n",
      " $ MGODGE  : num  3 4 4 4 4 5 5 2 6 2 ...\n",
      " $ MRELGE  : num  7 6 3 5 7 0 7 7 6 7 ...\n",
      " $ MRELSA  : num  0 2 2 2 1 6 2 2 0 0 ...\n",
      " $ MRELOV  : num  2 2 4 2 2 3 0 0 3 2 ...\n",
      " $ MFALLEEN: num  1 0 4 2 2 3 0 0 3 2 ...\n",
      " $ MFGEKIND: num  2 4 4 3 4 5 3 5 3 2 ...\n",
      " $ MFWEKIND: num  6 5 2 4 4 2 6 4 3 6 ...\n",
      " $ MOPLHOOG: num  1 0 0 3 5 0 0 0 0 0 ...\n",
      " $ MOPLMIDD: num  2 5 5 4 4 5 4 3 1 4 ...\n",
      " $ MOPLLAAG: num  7 4 4 2 0 4 5 6 8 5 ...\n",
      " $ MBERHOOG: num  1 0 0 4 0 2 0 2 1 2 ...\n",
      " $ MBERZELF: num  0 0 0 0 5 0 0 0 1 0 ...\n",
      " $ MBERBOER: num  1 0 0 0 4 0 0 0 0 0 ...\n",
      " $ MBERMIDD: num  2 5 7 3 0 4 4 2 1 3 ...\n",
      " $ MBERARBG: num  5 0 0 1 0 2 1 5 8 3 ...\n",
      " $ MBERARBO: num  2 4 2 2 0 2 5 2 1 3 ...\n",
      " $ MSKA    : num  1 0 0 3 9 2 0 2 1 1 ...\n",
      " $ MSKB1   : num  1 2 5 2 0 2 1 1 1 2 ...\n",
      " $ MSKB2   : num  2 3 0 1 0 2 4 2 0 1 ...\n",
      " $ MSKC    : num  6 5 4 4 0 4 5 5 8 4 ...\n",
      " $ MSKD    : num  1 0 0 0 0 2 0 2 1 2 ...\n",
      " $ MHHUUR  : num  1 2 7 5 4 9 6 0 9 0 ...\n",
      " $ MHKOOP  : num  8 7 2 4 5 0 3 9 0 9 ...\n",
      " $ MAUT1   : num  8 7 7 9 6 5 8 4 5 6 ...\n",
      " $ MAUT2   : num  0 1 0 0 2 3 0 4 2 1 ...\n",
      " $ MAUT0   : num  1 2 2 0 1 3 1 2 3 2 ...\n",
      " $ MZFONDS : num  8 6 9 7 5 9 9 6 7 6 ...\n",
      " $ MZPART  : num  1 3 0 2 4 0 0 3 2 3 ...\n",
      " $ MINKM30 : num  0 2 4 1 0 5 4 2 7 2 ...\n",
      " $ MINK3045: num  4 0 5 5 0 2 3 5 2 3 ...\n",
      " $ MINK4575: num  5 5 0 3 9 3 3 3 1 3 ...\n",
      " $ MINK7512: num  0 2 0 0 0 0 0 0 0 1 ...\n",
      " $ MINK123M: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ MINKGEM : num  4 5 3 4 6 3 3 3 2 4 ...\n",
      " $ MKOOPKLA: num  3 4 4 4 3 3 5 3 3 7 ...\n",
      " $ PWAPART : num  0 2 2 0 0 0 0 0 0 2 ...\n",
      " $ PWABEDR : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PWALAND : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PPERSAUT: num  6 0 6 6 0 6 6 0 5 0 ...\n",
      " $ PBESAUT : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PMOTSCO : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PVRAAUT : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PAANHANG: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PTRACTOR: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PWERKT  : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PBROM   : num  0 0 0 0 0 0 0 3 0 0 ...\n",
      " $ PLEVEN  : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PPERSONG: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PGEZONG : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PWAOREG : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PBRAND  : num  5 2 2 2 6 0 0 0 0 3 ...\n",
      " $ PZEILPL : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PPLEZIER: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PFIETS  : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PINBOED : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ PBYSTAND: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ AWAPART : num  0 2 1 0 0 0 0 0 0 1 ...\n",
      " $ AWABEDR : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ AWALAND : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ APERSAUT: num  1 0 1 1 0 1 1 0 1 0 ...\n",
      " $ ABESAUT : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ AMOTSCO : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ AVRAAUT : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ AAANHANG: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ ATRACTOR: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ AWERKT  : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ ABROM   : num  0 0 0 0 0 0 0 1 0 0 ...\n",
      " $ ALEVEN  : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ APERSONG: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ AGEZONG : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ AWAOREG : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ ABRAND  : num  1 1 1 1 1 0 0 0 0 1 ...\n",
      " $ AZEILPL : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ APLEZIER: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ AFIETS  : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ AINBOED : num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ ABYSTAND: num  0 0 0 0 0 0 0 0 0 0 ...\n",
      " $ Purchase: Factor w/ 2 levels \"No\",\"Yes\": 1 1 1 1 1 1 1 1 1 1 ...\n"
     ]
    }
   ],
   "source": [
    "str(Caravan)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<style>\n",
       ".dl-inline {width: auto; margin:0; padding: 0}\n",
       ".dl-inline>dt, .dl-inline>dd {float: none; width: auto; display: inline-block}\n",
       ".dl-inline>dt::after {content: \":\\0020\"; padding-right: .5ex}\n",
       ".dl-inline>dt:not(:first-of-type) {padding-left: .5ex}\n",
       "</style><dl class=dl-inline><dt>No</dt><dd>5474</dd><dt>Yes</dt><dd>348</dd></dl>\n"
      ],
      "text/latex": [
       "\\begin{description*}\n",
       "\\item[No] 5474\n",
       "\\item[Yes] 348\n",
       "\\end{description*}\n"
      ],
      "text/markdown": [
       "No\n",
       ":   5474Yes\n",
       ":   348\n",
       "\n"
      ],
      "text/plain": [
       "  No  Yes \n",
       "5474  348 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# check classes\n",
    "summary(Caravan$Purchase)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "FALSE"
      ],
      "text/latex": [
       "FALSE"
      ],
      "text/markdown": [
       "FALSE"
      ],
      "text/plain": [
       "[1] FALSE"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# check NA values\n",
    "any(is.na(Caravan))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "set.seed(123)\n",
    "# stratified split 70% for training, and the rest for testing\n",
    "split <- initial_split(Caravan, prop = 0.7, strata = \"Purchase\")\n",
    "train <- training(split)\n",
    "test  <- testing(split)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "  No  Yes \n",
       "3825  250 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# distribution of train\n",
    "table(train$Purchase) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "\n",
       "  No  Yes \n",
       "1649   98 "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "# distribution of test set\n",
    "table(test$Purchase)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "# create stratified training and testing\n",
    "features <- setdiff(names(train), \"Purchase\")\n",
    "# training\n",
    "x_train <- train[, features]\n",
    "y_train <- train$Purchase\n",
    "# testing\n",
    "x_test <- test[,features]\n",
    "y_test <- test$Purchase"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "library(\"caret\") \n",
    "library(\"klaR\") # naive bayes"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "# set up 10-fold cross validation procedure\n",
    "train_control <- trainControl(method = \"cv\", number = 10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning message:\n",
      "\"model fit failed for Fold01: usekernel=FALSE, fL=0, adjust=1 Error in NaiveBayes.default(x, y, usekernel = FALSE, fL = param$fL, ...) : \n",
      "  Zero variances for at least one class in variables: PVRAAUT, PWERKT, AVRAAUT, AWERKT\n",
      "\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 118\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 266\"\n",
      "Warning message:\n",
      "\"model fit failed for Fold02: usekernel=FALSE, fL=0, adjust=1 Error in NaiveBayes.default(x, y, usekernel = FALSE, fL = param$fL, ...) : \n",
      "  Zero variances for at least one class in variables: PVRAAUT, PWERKT, AVRAAUT, AWERKT\n",
      "\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 16\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 109\"\n",
      "Warning message:\n",
      "\"model fit failed for Fold03: usekernel=FALSE, fL=0, adjust=1 Error in NaiveBayes.default(x, y, usekernel = FALSE, fL = param$fL, ...) : \n",
      "  Zero variances for at least one class in variables: PVRAAUT, PWERKT, AVRAAUT, AWERKT\n",
      "\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 328\"\n",
      "Warning message:\n",
      "\"model fit failed for Fold04: usekernel=FALSE, fL=0, adjust=1 Error in NaiveBayes.default(x, y, usekernel = FALSE, fL = param$fL, ...) : \n",
      "  Zero variances for at least one class in variables: PVRAAUT, PWERKT, AVRAAUT, AWERKT\n",
      "\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 165\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 257\"\n",
      "Warning message:\n",
      "\"model fit failed for Fold05: usekernel=FALSE, fL=0, adjust=1 Error in NaiveBayes.default(x, y, usekernel = FALSE, fL = param$fL, ...) : \n",
      "  Zero variances for at least one class in variables: PVRAAUT, PWERKT, AVRAAUT, AWERKT\n",
      "\"\n",
      "Warning message:\n",
      "\"model fit failed for Fold06: usekernel=FALSE, fL=0, adjust=1 Error in NaiveBayes.default(x, y, usekernel = FALSE, fL = param$fL, ...) : \n",
      "  Zero variances for at least one class in variables: PVRAAUT, PWERKT, PZEILPL, AVRAAUT, AWERKT, AZEILPL\n",
      "\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 113\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 200\"\n",
      "Warning message:\n",
      "\"model fit failed for Fold07: usekernel=FALSE, fL=0, adjust=1 Error in NaiveBayes.default(x, y, usekernel = FALSE, fL = param$fL, ...) : \n",
      "  Zero variances for at least one class in variables: PVRAAUT, PWERKT, PZEILPL, AVRAAUT, AWERKT, AZEILPL\n",
      "\"\n",
      "Warning message:\n",
      "\"model fit failed for Fold08: usekernel=FALSE, fL=0, adjust=1 Error in NaiveBayes.default(x, y, usekernel = FALSE, fL = param$fL, ...) : \n",
      "  Zero variances for at least one class in variables: PVRAAUT, PWERKT, PPERSONG, AVRAAUT, AWERKT, APERSONG\n",
      "\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 71\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 126\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 295\"\n",
      "Warning message:\n",
      "\"model fit failed for Fold09: usekernel=FALSE, fL=0, adjust=1 Error in NaiveBayes.default(x, y, usekernel = FALSE, fL = param$fL, ...) : \n",
      "  Zero variances for at least one class in variables: PVRAAUT, PWERKT, AVRAAUT, AWERKT\n",
      "\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 243\"\n",
      "Warning message:\n",
      "\"model fit failed for Fold10: usekernel=FALSE, fL=0, adjust=1 Error in NaiveBayes.default(x, y, usekernel = FALSE, fL = param$fL, ...) : \n",
      "  Zero variances for at least one class in variables: PVRAAUT, PWERKT, AVRAAUT, AWERKT\n",
      "\"\n",
      "Warning message in nominalTrainWorkflow(x = x, y = y, wts = weights, info = trainInfo, :\n",
      "\"There were missing values in resampled performance measures.\"\n",
      "Warning message in train.default(x = x_train, y = y_train, method = \"nb\", trControl = train_control):\n",
      "\"missing values found in aggregated results\"\n"
     ]
    }
   ],
   "source": [
    "# train model\n",
    "naive.bayes <- train(x = x_train, y = y_train, method = \"nb\", trControl = train_control)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Cross-Validated (10 fold) Confusion Matrix \n",
       "\n",
       "(entries are percentual average cell counts across resamples)\n",
       " \n",
       "          Reference\n",
       "Prediction   No  Yes\n",
       "       No  93.9  6.1\n",
       "       Yes  0.0  0.0\n",
       "                            \n",
       " Accuracy (average) : 0.9387\n"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "confusionMatrix(naive.bayes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 240\"\n",
      "Warning message in FUN(X[[i]], ...):\n",
      "\"Numerical 0 probability for all classes with observation 1441\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "Confusion Matrix and Statistics\n",
       "\n",
       "          Reference\n",
       "Prediction   No  Yes\n",
       "       No  1649   98\n",
       "       Yes    0    0\n",
       "                                          \n",
       "               Accuracy : 0.9439          \n",
       "                 95% CI : (0.9321, 0.9542)\n",
       "    No Information Rate : 0.9439          \n",
       "    P-Value [Acc > NIR] : 0.5268          \n",
       "                                          \n",
       "                  Kappa : 0               \n",
       "                                          \n",
       " Mcnemar's Test P-Value : <2e-16          \n",
       "                                          \n",
       "            Sensitivity : 1.0000          \n",
       "            Specificity : 0.0000          \n",
       "         Pos Pred Value : 0.9439          \n",
       "         Neg Pred Value :    NaN          \n",
       "             Prevalence : 0.9439          \n",
       "         Detection Rate : 0.9439          \n",
       "   Detection Prevalence : 1.0000          \n",
       "      Balanced Accuracy : 0.5000          \n",
       "                                          \n",
       "       'Positive' Class : No              \n",
       "                                          "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "y_pred <- predict(naive.bayes, x_test, type=\"raw\")\n",
    "\n",
    "confusionMatrix(y_pred, y_test)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R",
   "language": "R",
   "name": "ir"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.1.0"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}