introduction-to-r-part-2.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### R in a nutshell\n",
    "\n",
    "- Statistical programming environments\n",
    "- Originally designed and implemented by statisticians\n",
    "- Widely popular due to its extensive collection of community-contributed packages\n",
    "- Quickly gaining places among traditional proprietary tools such as SAS and STATA for data analytics\n",
    "\n",
    "#### Learning Objectives\n",
    "\n",
    "- Understand basic programming concepts: variables, assignment, functions, loops, conditions\n",
    "- Understand core R concepts: data loading, data types, data access, libraries\n",
    "- Understand advanced R concepts: data manipulation, visualization\n",
    "- Understand HPC concepts: running R codes on the Palmetto supercomputer via batch submission scripts\n",
    "\n",
    "#### Materials on this notebook is based on two lessons by Software Carpentry and Data Carpentry:\n",
    "\n",
    "- Introduction to Programming using R\n",
    "- Data Analysis and Visualization in R for Ecology"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Where am I?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "getwd()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys <- read.csv(\"data/combined.csv\")\n",
    "head(surveys)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Frames Manipulation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if (!require('dplyr', character.only = TRUE)){\n",
    "  install.packages('dplyr', dep = TRUE)\n",
    "  if (!require('dplyr', character.only = TRUE)){\n",
    "    print ('Package not found')\n",
    "  }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Common `dplyr` functions: **\n",
    "- `select()`\n",
    "- `filter()`\n",
    "- `mutate()`\n",
    "- `groupby()`\n",
    "- `summarize()`\n",
    "- `%>%`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Cheatsheet: **\n",
    "http://www.rstudio.com/wp-content/uploads/2015/02/data-wrangling-cheatsheet.pdf"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Selecting columns and filtering rows **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "select(surveys, plot_id, species_id, weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "filter(surveys, year == 1995)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Pipes: combining multile select and filter actions **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys %>%\n",
    "  filter(weight < 5) %>%\n",
    "  select(species_id, sex, weight)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys_sml <- surveys %>%\n",
    "  filter(weight < 5) %>%\n",
    "  select(species_id, sex, weight)\n",
    "surveys_sml"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Challenge: **\n",
    "\n",
    "Using pipes, subset the `survey` data to include individuals collected before 1995 and retain only the columns `year`, `sex`, and `weight`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Mutate: create new columns based on existing columns **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys %>%\n",
    "  mutate(weight_kg = weight / 1000) %>%\n",
    "  head  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys %>%\n",
    "  filter(!is.na(weight)) %>%\n",
    "  mutate(weight_kg = weight / 1000) %>%\n",
    "  head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `is.na`: determines whether something is NA (not available - missing values)\n",
    "- `!`: negates a logical value"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Challenge: **\n",
    "\n",
    "Create a new data frame from the survey data that meets the following criteria: \n",
    "\n",
    "contains only the species_id column and a new column called hindfoot_half containing values that are half the hindfoot_length values. In this hindfoot_half column, there are no NAs and all values are less than 30."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Split-apply-combine data analysis and the summarize() function **\n",
    "\n",
    "- split data into groups\n",
    "- apply some analysis to each group\n",
    "- combine the results"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** `group_by()` and `summarize()`: **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys %>%\n",
    "  group_by(sex) %>%\n",
    "  summarize(mean_weight = mean(weight, na.rm = TRUE))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys %>%\n",
    "  group_by(sex, species_id) %>%\n",
    "  summarize(mean_weight = mean(weight, na.rm = TRUE))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- `NaN`: not a number\n",
    "- Need filtering to remove missing values"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "surveys %>%\n",
    "  filter(!is.na(weight)) %>%\n",
    "  group_by(sex, species_id) %>%\n",
    "  summarize(mean_weight = mean(weight))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "x <- surveys %>%\n",
    "       filter(!is.na(weight)) %>%\n",
    "       group_by(sex, species_id) %>%\n",
    "       summarize(mean_weight = mean(weight))\n",
    "str(x)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "- The `tbl_data` structure is very similar to a data frame; for our purposes the only difference is that, in addition to displaying the data type of each column under its name, it only prints the first few rows of data and only as many columns as fit on one screen.\n",
    "- If you want to display more data, you use the `print()` function at the end of your chain with the argument `n` specifying the number of rows to display:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys %>%\n",
    "  filter(!is.na(weight)) %>%\n",
    "  group_by(sex, species_id) %>%\n",
    "  summarize(mean_weight = mean(weight)) %>%\n",
    "  print(n = 15)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Summarization on multiple variables at the same time is also possible"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "surveys %>%\n",
    "  filter(!is.na(weight)) %>%\n",
    "  group_by(sex, species_id) %>%\n",
    "  summarize(mean_weight = mean(weight),\n",
    "            min_weight = min(weight))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Tallying: simply counting things: **"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "surveys %>%\n",
    "  group_by(sex) %>%\n",
    "  tally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Challenge: **\n",
    "\n",
    "- How many individuals were caught in each plot_type surveyed?\n",
    "- Use `group_by()` and `summarize()` to find the mean, min, and max hindfoot length for each species (using `species_id`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "** Exporting data to file: **\n",
    "\n",
    "- Write cleaned data to file, so that data cleaning process does not have to redone\n",
    "- Output data should be stored in different location from original raw data\n",
    "\n",
    "Conditional statement:\n",
    "\n",
    "\n",
    "if (condition is true){\n",
    "\n",
    "    do something\n",
    "    \n",
    "} else {\n",
    "\n",
    "    do something else\n",
    "    \n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "current_dir <- getwd()\n",
    "output_data_dir <- 'output_data'\n",
    "\n",
    "if (!file.exists(output_data_dir)){\n",
    "    dir.create(file.path(current_dir, output_data_dir))\n",
    "} else {\n",
    "    print (\"Directory already exists\")\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "?dir.create"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "?file.path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Create a clean data set with missing observations removed*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "surveys_complete <- surveys %>%\n",
    "  filter(species_id != \"\",         # remove missing species_id\n",
    "         !is.na(weight),           # remove missing weight\n",
    "         !is.na(hindfoot_length),  # remove missing hindfoot_length\n",
    "         sex != \"\")                # remove missing sex"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Additional filters: Remove rare species (less than 50 observations)\n",
    "\n",
    "- Create index of rare species\n",
    "- Filter rare species from the cleaned data set"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## Extract the most common species_id\n",
    "species_counts <- surveys_complete %>%\n",
    "  group_by(species_id) %>%\n",
    "  tally %>%\n",
    "  filter(n >= 50)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "## Only keep the most common species\n",
    "surveys_complete <- surveys_complete %>%\n",
    "  filter(species_id %in% species_counts$species_id)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*Write data to file*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "write.csv(surveys_complete, file = file.path(output_data_dir, \"surveys_complete.csv\"),\n",
    "          row.names=FALSE)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "R 3.4.2 (Juniper)",
   "language": "R",
   "name": "juniper_r3.4.2"
  },
  "language_info": {
   "codemirror_mode": "r",
   "file_extension": ".R",
   "mimetype": "text/x-r-source",
   "name": "R",
   "pygments_lexer": "r",
   "version": "3.4.2"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}