module-3-statistics/Day2_Distributions_Central_Limit_Hypothesis_Testing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a64b3269-eb72-42d6-9b12-76da3a19770d",
   "metadata": {},
   "source": [
    "### CMM262: Statistics, Day 2 - Part 1 - Distributions"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "edd598de-d4d2-429b-a28f-1a7dea80f5cb",
   "metadata": {},
   "outputs": [],
   "source": [
    "library(ggplot2)\n",
    "library(palmerpenguins)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "24b0894e-9746-4706-8ca2-9f55649c6a60",
   "metadata": {},
   "source": [
    "Want to understand how all the pieces fit together? Read R for Data Science: https://r4ds.had.co.nz/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "11376ac2-9224-4112-b645-059aa4075008",
   "metadata": {},
   "source": [
    "### Exercise 3 - Discrete Distributions - Binomial Distribution\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1af9f7b7-7c16-4ba4-95ad-2b1aff6fc46c",
   "metadata": {},
   "source": [
    "R has many built-in functions for working with distributions. These generally follow these conventions:\n",
    "\n",
    "* Probability Density Functions (PDFs and PMFs) - start with 'd' for 'density'.\n",
    "\n",
    "* Cumulative Distribution Functions (CDFs) start with 'p' for 'probability'.\n",
    "\n",
    "* Quantile Functions start with q. \n",
    "\n",
    "* Functions to generate random values from a distribution start with 'r' for random.\n",
    "\n",
    "Let's learn how to use these functions in the context of the binomial distribution. The binomial distribution describes the number of 'successes' (k) observed from n Bernoulli trials.\n",
    "\n",
    "Let's plot the binomial PMF using parameters: n=10 and p=0.1.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f9169cb1-5ab0-4ea4-baee-db3d0027df69",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 10\n",
    "k = seq(0, n, by=1)\n",
    "p = 0.1\n",
    "y = dbinom(k, n, p)\n",
    "\n",
    "binom.pmf <- data.frame(k=k, mass=y)\n",
    "\n",
    "ggplot(binom.pmf,  aes(k, mass)) + geom_point(color=\"blue\", size=3) + theme_bw(base_size=22)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d8ee14a7-2f64-480a-ac8e-3af81f0132ce",
   "metadata": {},
   "source": [
    "Now let's plot the binomial PMF for n=15, and p=0.5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "afedceed-d24c-40c2-b3a0-bc88e219335e",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = 15\n",
    "p=0.5\n",
    "k = seq(0, n, by=1)\n",
    "y = dbinom(k, n, p)\n",
    "\n",
    "binom.pmf <- data.frame(k=k, mass=y)\n",
    "\n",
    "ggplot(binom.pmf,  aes(k, mass)) + geom_point(color=\"blue\", size=3) + theme_bw(base_size=22)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4eb48b1-94dd-45f8-bd92-bf1cdce08696",
   "metadata": {},
   "source": [
    "Now let's randomly sample from the binomial distribution with the same parameters as above.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4a384818-8575-4847-91d1-5bd816c2c3a0",
   "metadata": {},
   "outputs": [],
   "source": [
    "num.samples <- 100\n",
    "k <- rbinom(num.samples, 15, 0.5)\n",
    "\n",
    "# Plot a histogram of the random sample\n",
    "hist(k, breaks=10)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a4023234-dabf-46ba-b92c-e4c368b0fcde",
   "metadata": {},
   "source": [
    "Revisiting the penguin dataset from last class, how many male penguins are there in our sample?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd64031f-a200-4770-bf88-4050cfc1d41a",
   "metadata": {},
   "outputs": [],
   "source": [
    "library(palmerpenguins)\n",
    "\n",
    "# remove samples where sex is unknown\n",
    "penguins <- penguins[!is.na(penguins$sex),]\n",
    "\n",
    "\n",
    "n.male = sum(penguins$sex == \"male\")\n",
    "n.female = sum(penguins$sex == \"female\")\n",
    "n.penguin = n.male + n.female\n",
    "\n",
    "cat(\"There are\", n.male, \"male and\", n.female, \"female penguins\" )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6d1ceb46-895d-4a52-b60b-811a8bd92da8",
   "metadata": {},
   "source": [
    "Assuming proportion parameter for male/female is 50%, what is the probability of observing exactly this number of male penguins in our sample?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd7b2fbf-8034-4dfb-aaf0-45cc9fced6f6",
   "metadata": {},
   "outputs": [],
   "source": [
    "dbinom(n.male, n.penguin, 0.5)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0e04ccee-cddf-4296-ab2f-d0aa89b3ee01",
   "metadata": {},
   "source": [
    "Let's plot the theoretical probability density function for the sample of n penguins, assuming p=0.5"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a123d1ce-092a-407e-be65-0ee43fc45edb",
   "metadata": {},
   "outputs": [],
   "source": [
    "n = n.penguin\n",
    "p=0.5\n",
    "k = seq(0, n, by=1)\n",
    "y = dbinom(k, n, p)\n",
    "\n",
    "binom.pmf <- data.frame(k=k, mass=y)\n",
    "\n",
    "ggplot(binom.pmf,  aes(k, mass)) + geom_point(color=\"blue\", size=3) + theme_bw(base_size=22)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a31d0ad0-4db9-4df2-bd70-230406b8ddfc",
   "metadata": {},
   "source": [
    "Is it surprising that we observed this many male penguins? How might we test this? "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a3720e83-50c1-499b-aae2-e7728224c6d1",
   "metadata": {},
   "source": [
    "### Exercise 4 - Central Limit Theorem"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c25f5764-621e-482d-b51b-cb733f535d8a",
   "metadata": {},
   "source": [
    "Let's assume that the entire population of penguins are the ones in our dataset, and that we are only able to obtain small samples of penguins for research purposes.\n",
    "\n",
    "We are going to focus on the body mass of the penguins for this exercise. Let's limit ourselves to the set of penguins which have a measurement for this trait."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "360a4e95-720a-44cf-87cd-f381da5bfdf2",
   "metadata": {},
   "outputs": [],
   "source": [
    "# remove rows where body mass is not defined\n",
    "penguins <- penguins[!is.na(penguins$body_mass_g),]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "826fd6a4-59ba-431d-b2b8-f9a0811881fe",
   "metadata": {},
   "source": [
    "Let's first take a look at the population distribution"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c36b1085-492a-4deb-8efc-6f7b84641782",
   "metadata": {},
   "outputs": [],
   "source": [
    "ggplot(data=penguins, aes(body_mass_g)) + geom_histogram(binwidth = 100)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6e80fd48-eb19-4486-9fb5-bd1c65461fc5",
   "metadata": {},
   "source": [
    "The data do not look all that normal. But that's OK! What is the population mean?\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d5523920-5fbb-4409-863e-149c8ba34028",
   "metadata": {},
   "outputs": [],
   "source": [
    "mean(penguins$body_mass_g)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "21f4ee35-96c2-4d89-86b3-986e43b2cc58",
   "metadata": {},
   "source": [
    "Now let's pretend that our study is not very well funded and we are only able to sample and weigh 10 penguins from the population each expedition. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c154d8a0-233a-4c9f-898f-8cfa74eaefd3",
   "metadata": {},
   "outputs": [],
   "source": [
    "samp.size = 10\n",
    "samp.mass = sample(penguins$body_mass_g, samp.size)\n",
    "mean(samp.mass)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "00b1a072-838c-4161-8b29-cb4e89beba41",
   "metadata": {},
   "source": [
    "The Central Limit Theorem says that sample means taken in this way should be normally distributed around the true population mean. And that this distribution should have standard deviation equal to the population standard deviation / sqrt(sample size). \n",
    "\n",
    "Let's test this idea out by repeating our sampling process 1000 times and looking at the resulting distribution of sample means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5c9015fd-c466-4fd1-9f4e-2852c99a7a11",
   "metadata": {},
   "outputs": [],
   "source": [
    "samp.size = 10\n",
    "samp.means = c()\n",
    "for(i in 1:1000) {\n",
    "  samp.mass = sample(penguins$body_mass_g, samp.size)\n",
    "  samp.means[i] = mean(samp.mass)}"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5843b9ba-f615-4e59-89b6-d9de603892b6",
   "metadata": {},
   "source": [
    "Now let's look at the histogram of sample means. Does it line up with the theoretical distribution?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "51d74cf8-6b00-4438-880a-b5484e7cf069",
   "metadata": {},
   "outputs": [],
   "source": [
    "# compute theoretical normal distribution with mean equal to pop. mean, and stdev equal to S.E.\n",
    "standard.err = sd(penguins$body_mass_g) / sqrt(samp.size)\n",
    "x = seq(min(penguins$body_mass_g), max(penguins$body_mass_g), by=10)\n",
    "predicted.density = dnorm(x, mean=mean(penguins$body_mass_g), sd=standard.err)\n",
    "\n",
    "# make a histogram of the sample means\n",
    "hist(samp.means, breaks=30, freq=F)\n",
    "\n",
    "# draw a curve of the theoretical distribution over top\n",
    "lines(x, predicted.density, col=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6bf7f00d-c7e7-4612-841a-5f88e056bde0",
   "metadata": {},
   "source": [
    "Pretty cool! What if we had a larger budget and we could sample 40 penguins each expedition? This should reduce the standard error about 2 fold (sqrt(40)/sqrt(10) = 2)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2f6e23ee-253d-4c38-a6de-c99086200bc0",
   "metadata": {},
   "outputs": [],
   "source": [
    "samp.size = 40\n",
    "samp.means = c()\n",
    "for(i in 1:1000) {\n",
    "  samp.mass = sample(penguins$body_mass_g, samp.size)\n",
    "  samp.means[i] = mean(samp.mass)\n",
    "}\n",
    "# compute theoretical normal distribution with mean equal to pop. mean, and stdev equal to S.E.\n",
    "standard.err = sd(penguins$body_mass_g) / sqrt(samp.size)\n",
    "x = seq(min(penguins$body_mass_g), max(penguins$body_mass_g), by=10)\n",
    "predicted.density = dnorm(x, mean=mean(penguins$body_mass_g), sd=standard.err)\n",
    "\n",
    "# make a histogram of the sample means\n",
    "hist(samp.means, breaks=30, freq=F)\n",
    "\n",
    "# draw a curve of the theoretical distribution over top\n",
    "lines(x, predicted.density, col=\"red\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b92c6811-a1df-4201-b264-6f7ae2001133",
   "metadata": {},
   "source": [
    "# Excercise 6 - Hypothesis Testing"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ff566911-00ef-4bba-b0e5-e4052a615b75",
   "metadata": {},
   "source": [
    "We want to test whether the mass of adelie and chinstrap penguins differ.\n",
    "First, let's compute a z-score for the difference in means."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b8ed245f-9afc-4bd6-8792-4cf89a069b32",
   "metadata": {},
   "outputs": [],
   "source": [
    "adelie.mean = mean(penguins$body_mass_g[penguins$species==\"Adelie\"] )\n",
    "adelie.sd = sd(penguins$body_mass_g[penguins$species==\"Adelie\"])\n",
    "adelie.n = sum(penguins$species==\"Adelie\")\n",
    "\n",
    "chinstrap.mean = mean(penguins$body_mass_g[penguins$species==\"Chinstrap\"] )\n",
    "chinstrap.sd = sd(penguins$body_mass_g[penguins$species==\"Chinstrap\"])\n",
    "chinstrap.n = sum(penguins$species==\"Chinstrap\")\n",
    "\n",
    "z = (adelie.mean - chinstrap.mean) / sqrt((adelie.sd**2)/adelie.n + (chinstrap.sd**2)/chinstrap.n)\n",
    "z"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ef1fb97-f140-49e1-9380-ff51f2334433",
   "metadata": {},
   "source": [
    "What fraction of the standard normal distribution is to the left of this z-score?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85a8a038-f178-44c2-98b2-f12e87b4b67d",
   "metadata": {},
   "outputs": [],
   "source": [
    "pnorm(z, mean=0, sd=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e3cebd0-1086-45fa-88f1-ba31ed185e5f",
   "metadata": {},
   "source": [
    "What fraction of the standard normal distribution is to the right of -z?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "bd01c0be-ddeb-4272-a10d-5906ddae061e",
   "metadata": {},
   "outputs": [],
   "source": [
    "pnorm(-z, mean=0, sd=1, lower.tail=F)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "174c4690-ee59-49d5-9f0a-1a61c0c4e38d",
   "metadata": {},
   "source": [
    "The distribution is symmetric, so we could have just multiplied by 2 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "85af2929-ec9a-4ab7-b3e0-2457e33f322a",
   "metadata": {},
   "outputs": [],
   "source": [
    "p.value = pnorm(z, mean=0, sd=1)*2\n",
    "cat(\"The p-value from the two-sided two-sample z-test is\", p.value)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "61778b71-9d8f-416d-9bf3-62362021a770",
   "metadata": {},
   "source": [
    "We fail to reject the null hypothesis.\n",
    "What about if we use a t-test instead?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "397aa9c8-b28e-4100-a082-45dabb86031c",
   "metadata": {},
   "outputs": [],
   "source": [
    "t.test(penguins$body_mass_g[penguins$species==\"Adelie\"],\n",
    "       penguins$body_mass_g[penguins$species==\"Chinstrap\"])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5d2a670-aabd-4973-99b1-9644299ab119",
   "metadata": {},
   "source": [
    "Note that the t-statistic is very similar to the z-statistic and so are the p-values. T-tests are very similar to z-tests for large sample sizes (n). They are most important to use (instead of z-tests) when sample sizes are small."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R [conda env:stats]",
   "language": "R",
   "name": "conda-env-stats-r"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.1.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}