module-3-statistics/Day1_Stats_and_Data_Visualization.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c70e901d-1776-4a5c-a1f5-24036feee887",
   "metadata": {},
   "source": [
    "### Statistics Module - Day 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9287e646-52f4-4fe1-8ce5-63d681e635d0",
   "metadata": {},
   "source": [
    "**Credits**: Written by Graham McVicker. Some parts based on a prior notebook developed by TAs Michelle Franc Ragsac (mragsac@eng.ucsd.edu) and Clarence Mah (ckmah@ucsd.edu) ."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "be85757e-463f-4761-9771-100a51e3bc3c",
   "metadata": {},
   "source": [
    "The Palmer Penguins dataset is based on data points collected and made available by [Dr. Kristen Gorman](https://www.uaf.edu/cfos/people/faculty/detail/kristen-gorman.php) and the [Palmer Station, Antarctica LTER](https://pal.lternet.edu/), a member of the [Long Term Ecological Research Network](https://lternet.edu/)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e16bdc4-eca6-40bf-9afa-bddd42ac0e13",
   "metadata": {},
   "source": [
    "Source: https://allisonhorst.github.io/palmerpenguins/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "930981b5-dae4-4d49-bb41-7d7d23c79bc3",
   "metadata": {},
   "source": [
    "### Exercise 1 - Summary Statistics"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0ad9e348-903c-4f98-a3c9-192d4e0fc81e",
   "metadata": {},
   "source": [
    "Let's import the Palmer penguins library and load the penguins dataset\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9c0efc2e-6f94-4908-9c59-fd1b1a665a01",
   "metadata": {},
   "outputs": [],
   "source": [
    "library(palmerpenguins)\n",
    "data(penguins, package='palmerpenguins')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8ae4a9a8-c685-40c1-96ac-c4cd5f272ec8",
   "metadata": {},
   "source": [
    "The penguins dataset is in a [tibble](https://www.rstudio.com/blog/tibble-1-0-0/) which is a more user-friendly version of a data frame. We can view the data in the table:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99415615-a2c5-4bec-a292-bd44cb45671e",
   "metadata": {},
   "outputs": [],
   "source": [
    "head(penguins, 20)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3d2c3b5f-29ef-40a0-807b-64d4f7fd885b",
   "metadata": {},
   "source": [
    "We can get a quick summary of the whole dataset using 'summary':"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5aa54cd8-7812-4a74-8f33-e164d21f5ab8",
   "metadata": {},
   "outputs": [],
   "source": [
    "summary(penguins)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d4525cb3-4cc1-4ea1-89b4-994aed13883c",
   "metadata": {},
   "source": [
    "first lets create vectors that can be used to pull out male or female chinstrap penguins\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "35493669-4f8a-4ea1-89f5-b8bd91696f63",
   "metadata": {},
   "outputs": [],
   "source": [
    "male.chinstraps = (penguins$species == \"Chinstrap\") & (penguins$sex == \"male\")\n",
    "female.chinstraps = (penguins$species == \"Chinstrap\") & (penguins$sex == \"female\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "27977450-0466-4c2e-95c7-7cb7c4c7fedd",
   "metadata": {},
   "outputs": [],
   "source": [
    "n.male.chinstraps = sum(male.chinstraps)\n",
    "cat(\"There are\", n.male.chinstraps, \"male chinstrap penguins\\n\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "535689ae-03fb-4ad3-9dd6-471eab3f9bb7",
   "metadata": {},
   "outputs": [],
   "source": [
    "n.female.chinstraps = sum(female.chinstraps)\n",
    "cat(\"There are\", n.female.chinstraps, \"female chinstrap penguins\\n\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7d31dc87-74b8-4782-bf41-ed8a42db2792",
   "metadata": {},
   "source": [
    "What are the sample mean bill lengths for the male and female chinstrap penguins?\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f35085e4-8332-4668-983d-15b7f9b1187f",
   "metadata": {},
   "source": [
    "Males: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b0429f83-9dcc-48ec-8f2d-eedb5fbb3a79",
   "metadata": {},
   "outputs": [],
   "source": [
    "# can compute manually\n",
    "sum(penguins$bill_length_mm[male.chinstraps])/n.male.chinstraps\n",
    "\n",
    "# or using the mean function\n",
    "mean(penguins$bill_length_mm[male.chinstraps])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "beb945e0-5a6f-43d9-990d-6d18d947372e",
   "metadata": {},
   "source": [
    "Females:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f76aa3d5-b207-4b39-9f99-8d532419f69e",
   "metadata": {},
   "outputs": [],
   "source": [
    "mean(penguins$bill_length_mm[female.chinstraps])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "38d5b0b3-769c-4f52-bb8d-dca43f4bd1fb",
   "metadata": {},
   "source": [
    "Now compute the sample standard deviation for female penguins:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5dc7480-6cee-464b-9505-b3e6af674203",
   "metadata": {},
   "outputs": [],
   "source": [
    "sample.var <- sum((penguins$bill_length_mm[female.chinstraps] -\n",
    "                  mean(penguins$bill_length_mm[female.chinstraps]))**2)/(n.female.chinstraps-1)\n",
    "sqrt(sample.var)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5cb3bb9a-f7c7-4029-a5f4-f6f24291cc65",
   "metadata": {},
   "source": [
    "The n-1 in the denominator of the sample standard deviation is \"Bessel's correction\". It ensures that the sample standard deviation is an unbiased estimate of the population standard deviation.\n",
    "\n",
    "We can also use the sd function to get the same result:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b22f475b-9513-4fa7-9816-0304059efa76",
   "metadata": {},
   "outputs": [],
   "source": [
    "sd(penguins$bill_length_mm[female.chinstraps])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "08849ce6-9be2-44d7-b2ef-a8fcded046ab",
   "metadata": {},
   "source": [
    "Now compute sample standard deviation for male chinstraps:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "74085d31-0b81-440a-b0dd-3bf02d532262",
   "metadata": {},
   "outputs": [],
   "source": [
    "sd(penguins$bill_length_mm[male.chinstraps])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "49412873-6858-4058-9c02-519cab665a1a",
   "metadata": {},
   "source": [
    "Another common summary statistic is the median, which is the middle value of a sample."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c4776491-a7d1-4061-bff3-852e322fdc5c",
   "metadata": {},
   "outputs": [],
   "source": [
    "median(penguins$bill_length_mm[male.chinstraps])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0b5bffe7-efcd-41a0-bc3c-8eaf25229428",
   "metadata": {},
   "outputs": [],
   "source": [
    "median(penguins$bill_length_mm[female.chinstraps])"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e4c0acb6-b2ed-4ba0-9099-93359acc1c96",
   "metadata": {},
   "source": [
    "### Exercise 2 - Visualizing Data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6b6a93b5-7e0d-4074-90fb-8aab9f5d30a5",
   "metadata": {},
   "source": [
    "Summary statistics are really useful, but they often obscure characteristics of the data, such as the shape of the distribution or the presence of outliers.  Compute summary statistics, but also visualize your data!"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed838b05-2c73-4fab-b015-78690a35ba28",
   "metadata": {},
   "source": [
    "A histogram provides perhaps the simplest way to visualize data. Let's start by looking at bill length and bill depth. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3ee813d8-0c1f-4061-ab7f-1a84aaca6961",
   "metadata": {},
   "outputs": [],
   "source": [
    "# TODO: add image showing bill length and bill depth?"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2c4f9ac-7df5-42b5-82a4-0b5f136b3f39",
   "metadata": {},
   "source": [
    "The simplest way to make a histogram is with the hist function. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b1e8ccef-85f8-4e19-99fc-dc6f503d6970",
   "metadata": {},
   "outputs": [],
   "source": [
    "hist(penguins$bill_length_mm)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "153861d9-5019-48a2-98e3-3be5f8117033",
   "metadata": {},
   "source": [
    "We can use the breaks parameter to control the binning:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3fad9afd-5537-4b13-a675-f0824c4fed6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "hist(penguins$bill_length_mm, breaks=30)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "142ff5ab-d666-413a-abab-81b5a8ee5ee7",
   "metadata": {},
   "source": [
    "(https://ggplot2.tidyverse.org/index.html) provides more powerful methods for plotting. ggplots syntax can take some getting used to. I recommend using a [ggplot2 cheatsheet](https://www.maths.usyd.edu.au/u/UG/SM/STAT3022/r/current/Misc/data-visualization-2.1.pdf) for reference."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2aa2322b-e30c-4b8c-8f64-1cfc2a871bd7",
   "metadata": {},
   "outputs": [],
   "source": [
    "library(ggplot2)\n",
    "ggplot(data=penguins, aes(bill_length_mm)) + geom_histogram()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1378efe5-aa0c-4608-ac93-1599cddd7f8e",
   "metadata": {},
   "source": [
    "Notice that the distribution looks bimodal. And remember we have 3 different species of penguin in our dataset. Let's see if the bill length differs by species. Let's make separate overlapping histograms in different colors for each species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8695be3b-dc6c-4dd4-9fea-f6afa032428d",
   "metadata": {},
   "outputs": [],
   "source": [
    "ggplot(penguins, aes(x=bill_length_mm, fill=species)) + geom_histogram(position=\"identity\", alpha=0.6, binwidth=2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a12253c6-3db4-40be-92b8-d37d1d0b6f61",
   "metadata": {},
   "source": [
    "We can also visualize the distributions of the species as separate plots, using one of ggplot's faceting functions."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "99be870e-588e-4192-a31d-5c0b235e3612",
   "metadata": {},
   "outputs": [],
   "source": [
    "ggplot(penguins, aes(x=bill_length_mm, fill=species)) + geom_histogram(binwidth=2) + facet_wrap(.~species)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "81e7da3d-8d73-4f5e-b0bd-9003b42dd7c7",
   "metadata": {},
   "source": [
    "If we want a more compact display, we can use boxplots or violin plots"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "39039b0a-2c53-435b-9d9f-7d84e686d5bc",
   "metadata": {},
   "outputs": [],
   "source": [
    "ggplot(penguins, aes(x=bill_length_mm, fill=species)) + geom_boxplot()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "10c2a18d-4e9e-48be-b9bf-92669de34285",
   "metadata": {},
   "source": [
    "ggridges is a cool package that lets you plot overlapping density plots (smoothed histograms)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ac064689-a7f9-46a7-8627-09e35c428fe3",
   "metadata": {},
   "outputs": [],
   "source": [
    "library(ggridges)\n",
    "ggplot(penguins, aes(x=bill_length_mm, y=species, fill=species)) + geom_density_ridges()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "64d69720-30a3-452d-87d5-22cd17c07835",
   "metadata": {},
   "source": [
    "What if we want to view two variables at once for example, bill length and bill depth? We can use a scatter plot to view their joint distribution. We can use color and plotting shapes to indicate species and sex. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c135e0bc-141f-49ea-b90a-3d36419538bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "ggplot(penguins, aes(x=bill_length_mm, y=bill_depth_mm, color=species, shape=sex)) + geom_point()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ed1ed682-4956-4ccf-b126-66a8bedc6b7e",
   "metadata": {},
   "source": [
    "Let's plot the distributions of bill lengths of male and female penguins separately for each species."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "02ced30e-a223-45ca-986d-a7be8260ecbe",
   "metadata": {},
   "outputs": [],
   "source": [
    "ggplot(penguins, aes(x=bill_length_mm, fill=sex)) + geom_histogram(position=\"identity\", alpha=0.6, binwidth=2) + facet_wrap(.~species)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R [conda env:stats]",
   "language": "R",
   "name": "conda-env-stats-r"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".r",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "4.1.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}