module-1-programming/Day2_2_Terminal_Commands_and_Bash_Scripting.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<h1 style = \"font-size: 35px\">Terminal Commands and Bash Scripting</h1>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook summarizes the Unix module lecture. Here are the slides that accompanied that lecture:\n",
    "\n",
    "https://docs.google.com/presentation/d/1sYx_aTcjUQv6NfT28Ae_N3yKL7-XGC7d2JJNIB3a1DA"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Anatomy of a Command"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/command_anatomy.jpg\" width='750px'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# File System Navigation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `pwd`: where am I?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is the path to my current location (aka working directory) in the file system?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first slash in the path refers to the root directory, the folder that contains all of the other folders on your computer."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `whoami`: who am I logged in as?\n",
    "To continue the existential questions...\n",
    "\n",
    "<img src=\"img/whoami.jpg\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Who am I logged in as? What is my username?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "whoami"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `tree`: show the file system structure in a tree format\n",
    "```bash\n",
    "tree ~/module-1-programming\n",
    "```\n",
    "```\n",
    "~/module-1-programming\n",
    "├── Day1_1_Intro_to_Programming.ipynb\n",
    "├── Day1_2_Python_Packages.ipynb\n",
    "├── Day1_3_Intro_to_Pandas.ipynb\n",
    "├── Day2_1_R_Basics.ipynb\n",
    "├── Day2_2_Terminal_Commands_and_Bash_Scripting.ipynb\n",
    "├── README.md\n",
    "├── bash_playground\n",
    "│   ├── scripts\n",
    "│   │   ├── hello.sh\n",
    "│   │   ├── hello_to_all_of_you.sh\n",
    "│   │   └── hello_you.sh\n",
    "│   └── sequences.txt\n",
    "├── data\n",
    "│   ├── chrom_lengths.tsv\n",
    "│   ├── gene_chrom.tsv\n",
    "│   └── hg19.chrom.sizes.txt\n",
    "└── img\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `cd`: **c**hange my working **d**irectory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `cd` command to navigate to a directory, given the path to that directory:\n",
    "```bash\n",
    "cd [path_to_dir]\n",
    "```\n",
    "Without any arguments, `cd` will go to your home directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now where are we?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Usually, you'll want to give `cd` the path to a directory, though. For example, let's go to the `bash_playground/` folder."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd module-1-programming/bash_playground"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can go back *up* a level with the \"..\" notation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd ..\n",
    "pwd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And you can go back to your last directory by using `-` as the directory path. This will also print the new working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cd -"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `ls`: **l**i**s**t contents of a directory"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use the `ls` command to print the contents of a directory, given the path to that directory:\n",
    "```bash\n",
    "ls [path_to_dir_or_file]...\n",
    "```\n",
    "Without any arguments, `ls` will show you the contents of your current working directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But we can also give it the path to a file or directory (or multiple files or directories)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls scripts/hello.sh scripts/hello_you.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Adding flags get us more information. Let's use the short-format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -lh scripts/hello.sh scripts/hello_you.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the `-a` flag to show _all_ files, including hidden ones, prefixed with a single dot `.`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -a"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Just like with `cd`, we can also specify the path to the directory above our current one using the \"..\" notation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In any path, the tilde `~` represents our home directory, so we can view the files in that directory like this:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls ~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The tilde actually gets replaced with the path to our home directory prior to us executing the command. You can see this by putting an `echo` in front of the command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo ls ~"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Relative vs absolute paths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are two types of paths:\n",
    "1. relative - interpreted relative to your current working directory\n",
    "2. absolute - independent of your current working directory (ie relative to the root directory)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So these paths refer to the same file:\n",
    "- `scripts/hello.sh` (relative)\n",
    "- `~/module-1-programming/bash_playground/scripts/hello.sh` (absolute)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### When should I use a relative or absolute path?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whether it's best to use a relative or absolute path will depend on the context.\n",
    "\n",
    "If we changed our working directory from `bash_playground/`, then the first path would break but the second one wouldn't.\n",
    "\n",
    "But if we changed the location of the `module-1-programming/` directory, then the second path would be the one to break, instead. The first path wouldn't break, assuming our working directory is still `bash_playground/`!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Different ways of writing a path"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These are all equivalent paths!\n",
    "- `scripts/hello.sh`\n",
    "- `../bash_playground/scripts/hello.sh`\n",
    "- `scripts/../scripts/hello.sh`\n",
    "- `scripts/./hello.sh`\n",
    "\n",
    "The double-dot `..` refers to the folder above and the single-dot `.` refers to the current folder."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Understanding help and manual pages"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most commands have usage documentation available via `--help` (long format) **and/or** `-h` (short format)\n",
    "```bash\n",
    "cmd --help\n",
    "```\n",
    "```bash\n",
    "cmd -h\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Alternatively, some commands will also register their help via the built-in manual pages in the `man` command\n",
    "```bash\n",
    "man cmd\n",
    "```\n",
    "(press **q** to quit or exit-out after opening the manual)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## An example manual page: `man ls`\n",
    "<img src=\"img/man_ls.png\" width='800px'>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Wildcards and globbing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Question marks `?` and asterisks `*` are considered _wildcards_.\n",
    "\n",
    "You can use a question-mark `?` in place of any character. Bash will find all of the files that match. This is called _globbing_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls ?equences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use an asterisk `*` when you want to match zero or more characters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls *es.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is useful when you want to refer to a bunch of similarly named files simultaneously."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls scripts/*.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Wildcards get expanded to a space-separated list prior to the command executing. A great way to see this is to prepend the command with an `echo`.\n",
    "\n",
    "We'll discuss the `echo` command in more depth later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo ls scripts/*.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Tab completion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "While typing a command, you can press `<tab>` to use the terminal's auto-completions. This helps save you time and reduces the chance you'll make typos while you type. So you should make an effort to use tab completion as often as possible.\n",
    "\n",
    "For example, let's tab-complete the `scripts/` directory.\n",
    "```bash\n",
    "ls sc<tab>\n",
    "```\n",
    "## double tap when there are multiple options\n",
    "If there are multiple possible ways to complete your command, nothing will appear at first.\n",
    "```bash\n",
    "ls scripts/hello<tab>\n",
    "```\n",
    "Press `<tab>` twice in quick succession to display the possible options.\n",
    "```bash\n",
    "ls scripts/hello<tab><tab>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making use of your bash history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use the up and down arrow keys to cycle through your past commands.\n",
    "\n",
    "To view all of your past commands, use the `history` command.\n",
    "\n",
    "**Caution**: this might spit out a lot of text!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "history"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To search for a past command within the history:\n",
    "1. press _ctrl+r_\n",
    "2. start typing part of the command\n",
    "    The top ?hit will appear as you type\n",
    "3. To move onto the next best hit, press _ctrl+r_ again\n",
    "4. Once you've found the command, use _esc_ or the arrow keys (or press _enter_ if you want to execute the command immediately)\n",
    "\n",
    "**You try**: Search for the last `echo` command we executed."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Altering the file system"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `mkdir`: **m**a**k**e a new **dir**ectory\n",
    "```bash\n",
    "mkdir path_to_dir...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mkdir shiny_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Look, it's there!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `cp`: **c**o**p**y a file or directory\n",
    "```bash\n",
    "cp path_to_source... path_to_dest\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cp sequences.txt shiny_new/sequences_copy.txt\n",
    "ls shiny_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `cp -r` to copy a directory and all of its contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cp -r shiny_new same_old\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We should now have two new directories in our bash playground: `same_old/` and `shiny_new/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls same_old"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And `shiny_new/` should contain the same contents as `same_old/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls shiny_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `touch`: create a new, empty file\n",
    "```bash\n",
    "touch path_to_file...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "touch empty_inside.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Don't read into that name too much. Just trust me, it's empty. Look at the size. Or go check it yourself later!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -lh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `mv`: **m**o**v**e a file or directory\n",
    "```bash\n",
    "mv path_to_source... path_to_dest\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's move `empty_inside.txt` to the `shiny_new/` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "mv empty_inside.txt shiny_new/\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check: is it there?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls shiny_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### renaming a file\n",
    "Just use `mv` where the destination is in the same folder!\n",
    "\n",
    "Let's convert `empty_inside.txt` to `fulfilled.txt`. Hopefully, you'll be the same way after this lesson ;)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "mv shiny_new/empty_inside.txt shiny_new/fulfilled.txt\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `rm`: **r**e**m**ove a file or directory\n",
    "```bash\n",
    "rm path_to_file_or_dir...\n",
    "```\n",
    "With great power comes great responsibility... use this carefully! There is no \"undo\" button or trash can in the terminal!\n",
    "\n",
    "Let's `echo` the command first to make sure that the wildcards are expanding correctly."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo rm shiny_new/*.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm shiny_new/*.txt\n",
    "ls shiny_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `rm -r`: delete a directory and all of its contents, **r**ecursively"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm -r same_old\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `rmdir`: **r**e**m**ove an empty **dir**ectory\n",
    "```bash\n",
    "rmdir path_to_empty_dir...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rmdir shiny_new\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `ln`: create a symlink to a file or directory\n",
    "```bash\n",
    "ln -s path_to_target... path_to_link\n",
    "```\n",
    "A symlink is a shortcut to a file or directory. This can be helpful when you want to have a file in multiple locations but don't want to copy it.\n",
    "\n",
    "Recall the `data/` directory in the folder above `bash_playground/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls ../data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's create a symlink to the `data/` directory from within the `bash_playground/` directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ln -s ../data data-sym\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A symlink is a special file that just references another file (or directory). You can view the path of a symlink using `ls -l`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls -l data-sym"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Symlinks will act exactly the same as their original file or directory."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls data-sym\n",
    "cd data-sym"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls\n",
    "cd .."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm data-sym\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deleting a symlink doesn't delete the target it points to."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls .."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Printing things"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `cat`: print the contents of files\n",
    "```bash\n",
    "cat path_to_file...\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you provide more than one file, `cat` will _concatenate_ them (aka append them to each other). This is where it gets its name."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remember the hidden secret file? Let's take a look at it now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat .you_found_me.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, let's concatenate the two files together."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat .you_found_me.txt sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `wc`: count the number of lines or words in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "wc [path_to_file]...\n",
    "```\n",
    "You can use the `wc` command to obtain counts of the words, lines, or characters (respectively) in a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can use flags to get just the line count (`-l`), just the word count (`-w`), or just the character count (`-c`)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc -l sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc -w sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wc -c sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `head`: print the top of a file\n",
    "```bash\n",
    "head [path_to_file]...\n",
    "```\n",
    "By default, `head` will print the first 10 lines of a file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "But you can also specify the number of lines to the `-n` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head -n 5 sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `tail`: print the end of a file\n",
    "```bash\n",
    "tail [path_to_file]...\n",
    "```\n",
    "You can use `tail` in the exact same way as `head`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tail sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tail -n 5 sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `echo`: print a string\n",
    "```bash\n",
    "echo [string]...\n",
    "```\n",
    "You can use the `echo` command to print a string."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo hey"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `echo` command accepts multiple string arguments, so we can also do this."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo is anybody out there?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Under the hood: standard input, output, and error"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Every command has three standard files: standard input (stdin), standard output (stdout), and standard error (stderr).\n",
    "\n",
    "When you run a command, its output will be written to stdout and any error messages will be written to stderr.\n",
    "\n",
    "<img src=\"img/stdin_out_err.png\" width='450px'>\n",
    "\n",
    "By default, stdout and stderr are connected to your terminal, which is why you can see them on your screen.\n",
    "\n",
    "Most commands will not read from stdin unless they detect that they should, or you explicitly request them to. More on that later."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Redirecting stdout and stderr"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can redirect the stdout and stderr from the terminal to custom files. Standard output is represented by the number 1 and standard error by the number 2."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls scripts/ nonexistent_file"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls scripts/ nonexistent_file 1>file_out 2>file_err"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat file_out"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat file_err"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "If you omit the number prior to the redirection operator `>`, it will default to stdout (the number 1)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls scripts/ nonexistent_file >file_out_new 2>file_err_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat file_out_new"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat file_err_new"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can redirect both stdout and stderr to the same file using the ampersand `&`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls scripts/ nonexistent_file &>file_both\n",
    "cat file_both"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Whichever redirection you leave off will default to the terminal, again."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls scripts/ nonexistent_file >file_out\n",
    "cat file_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Appending instead of overwriting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Using a single redirection operator `>` will overwrite the contents of the target file. Let's overwrite the contents of `file_out`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history >file_out\n",
    "tail file_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can append to the file (ie add to the end of it) instead of overwriting it by using two redirection operators in sequence `>>`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls >>file_out\n",
    "tail file_out"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we move onto the next section, let's just clean up after ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo rm file_*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm file_*"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Piping from stdout to stdin"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/mariopipe.png\">\n",
    "\n",
    "You can use the pipe operator **|** to connect the stdout of one command to the stdin of another.\n",
    "\n",
    "<img src=\"img/piping_stdout_stdin.png\" width='675px'>\n",
    "\n",
    "Without any arguments, most commands (including `tail`) will assume that the file should be read from stdin, instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history | tail"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can chain a series of pipes to create a _pipeline_!\n",
    "\n",
    "Let's try to get the 10 most recent commands from our history after excluding the 5 most recent ones."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "history | tail | head -n5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Piped commands run in parallel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A common misconception is that commands in a pipeline run sequentially. In fact, they actually run in parallel (at the same time)!\n",
    "\n",
    "Each command waits until it receives a line of text from its stdin. It processes this line and then writes to its stdout as soon as it can. In this way, the original text is said to be _streamed_ through the pipeline. Instead of waiting for each step to finish before moving onto the next step, we are running the commands at the same time!\n",
    "\n",
    "You can think of a chain of piped commands as a bunch of trains, each with their own engine, traversing on the same railroad track. Each train can go as fast or as slow as it wants, but none can overtake the other.\n",
    "\n",
    "The ability to _stream_ text makes command pipelines one of the fastest methods to process big data, and thus, a favorite among bioinformaticians."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In addition to storing text within files, you can also store text within variables.\n",
    "\n",
    "Let's try storing the string \"hello\" inside a variable named `my_variable`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_variable=hello"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To use the variable within another command, we must prefix the variable name with a dollar-sign `$`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Concatenating (joining) variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can join variables together by just putting them next to each other. Let's create two new variables and join them for the third."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "hello_var=hello\n",
    "world_var=world\n",
    "my_variable=$hello_var$world_var\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What would it take for us to get a space between \"hello\" and \"world\"? First, let's try this and see what happens:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_variable=hello world"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It doesn't work! But why not?\n",
    "\n",
    "Spaces are important in bash. They're used to separate arguments from each other and delineate strings."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use quotes when there are spaces involved"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we just need a way to denote that both words are part of a single string. Quotes to the rescue!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_variable=\"hello world\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Returning to our original example, it should be clear why we can just do something like this, now:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_variable=\"$hello_var $world_var\"\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What about if we want to put an underscore between the two variables, instead?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_variable=$hello_var_$world_var\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It doesn't work! But why not?\n",
    "\n",
    "We're telling bash to look for a variable called `hello_var_`. But since this variable doesn't exist, bash will replace it with an empty string `\"\"`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use braces to avoid ambiguity"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "So we just need a way to denote where the variable is. The solution is to surround the variable name in braces."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_variable=${hello_var}_$world_var\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Use single quotes when you want to be interpreted literally"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_variable='$hello_var $world_var'\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Within double quotes, expressions with $ are evaluated prior to the command executing. But within single quotes, everything is kept as it is.\n",
    "\n",
    "Once again, we can prefix the command with an `echo` to see the resolved command before it gets executed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo my_variable=\"$hello_var $world_var\"\n",
    "echo my_variable='$hello_var $world_var'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Storing the stdout of a command in a variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can store the stdout of a command in a variable by wrapping the command in parantheses.\n",
    "```bash\n",
    "my_variable=$(cmd)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "my_variable=$(head -n1 sequences.txt)\n",
    "echo $my_variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's get fancy and actually try to manipulate some of these files.\n",
    "\n",
    "Note: all of these commands will also read from stdin if they aren't given a file argument."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `cut`: extract specific columns from a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "cut [path_to_file]...\n",
    "```\n",
    "To get a column of the file, provide the column number (ex: 2) to the `-f` flag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cut -f 2 sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `tr`: substitution of characters in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "tr character_set1 character_set2\n",
    "```\n",
    "You can use the `tr` command to replace all instances of a T with a U. Let's turn our DNA sequences into RNA!\n",
    "\n",
    "Note that the `tr` command doesn't have a file argument. It only accepts input from stdin. We can attach a file to stdin using the `<` operator (rather than `>` for stdout)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tr T U <sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The `tr` command can also replace more than one character at a time. Let's replace each base with its complement."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tr ACGT TGCA <sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `rev`: reverse text in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "rev [path_to_file]...\n",
    "```\n",
    "The `rev` command reverses the characters on each line of a file. Let's use it to obtain the reverse complement of our sequences."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cut -f2 sequences.txt | tr ACGT TGCA | rev"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `paste`: merge files horizontally"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "paste [path_to_file]...\n",
    "```\n",
    "The `paste` command is useful when you want to merge multiple files, each containing a table with **n** lines (rows)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "paste .you_found_me.txt sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `sort`: sort the lines in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "sort [path_to_file]...\n",
    "```\n",
    "The `sort` command will sort the lines of your file. By default, this is done alphabetically."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head -n 5 sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sort sequences.txt | head -n 5"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also sort the file by specific columns using the `-k` flag.\n",
    "\n",
    "Let's sort by the second column."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sort -k 2 sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `uniq`: filter out duplicate lines in a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "uniq [path_to_file_input [path_to_file_output]]\n",
    "```\n",
    "You can use the `uniq` command to filter out duplicated lines or count the number of unique lines.\n",
    "\n",
    "Note that the text provided to `uniq` must be sorted first! Otherwise, it won't work properly.\n",
    "\n",
    "Let's find unique sequences in the `sequences.txt` file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head -n 5 sequences.txt | cut -f2 | sort"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head -n 5 sequences.txt | cut -f2 | sort | uniq"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I bet you didn't even notice the last two sequences were duplicates until now ;P"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `grep`: search for lines containing a keyword"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "grep pattern [path_to_file]...\n",
    "```\n",
    "To search for lines in the file containing a specific pattern, you can use `grep`.\n",
    "\n",
    "Let's look for all sequences with \"TAAC\" in them."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep \"TAAC\" sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "All of these commands read from stdin and `grep` is no exception. For example, we can search through our bash history for the `echo` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "history | grep echo"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When you use the `-E` flag, `grep` will interpret the pattern `grep` as a _regex (aka regular expression)_. Regexes can get quite complicated, so we don't have the time to dive into them completely.\n",
    "\n",
    "But to give you an idea of how powerful this can be, let's try to grab sequences C through F or just sequence B and D."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep -E 'seq[C-F]' sequences.txt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "grep -E 'seq(B|D)' sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `sed`: scan through a file and make edits"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "sed script_pattern [path_to_file]...\n",
    "```\n",
    "The `tr` command made single-character substitutions, but what if you want to replace entire words or sentences? That's one of the most common uses for the `sed` command.\n",
    "\n",
    "Let's replace \"seqC\" with the label \"sample1\". The first argument to `sed` is a special pattern composed of the letter **s**, the search keyword, then the replacement keyword, where each is separated by some character -- in this case, we use a slash `/`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head -n 5 sequences.txt | sed 's/seqC/sample1/'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `awk`: scan through a file and extract text\n",
    "```bash\n",
    "awk program-text [path_to_file]...\n",
    "```\n",
    "`awk` is a much more powerful command than many of the others. It implements its own programming language which can be used to do any number of tasks, including most of those of the other commands we just learned.\n",
    "\n",
    "We don't have time to discuss `awk`, but there are tons of great tutorials online you can peruse."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `vim`: terminal-based text editor\n",
    "```bash\n",
    "vim [path_to_file]...\n",
    "```\n",
    "`vim` is a fully-featured [text editor](https://learntocodewith.me/resources/text-editors/) that runs from within the terminal. You can learn how to use it at [openvim.com](https://openvim.com)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `less`: view large files and text one page at a time\n",
    "```bash\n",
    "less [path_to_file]...\n",
    "```\n",
    "You can use `less` to view large files one page at a time. This proves to be incredibly useful in bioinformatics, where most files are very, very large."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's take a look at the `sequences.txt` file with `less`.\n",
    "```bash\n",
    "less sequences.txt\n",
    "```\n",
    "\n",
    "This will open a separate viewing window. You can quit out of it and return to the terminal prompt by pressing **q**.\n",
    "\n",
    "Now, let's take a look at our bash history. We can pipe into `less` for this.\n",
    "```bash\n",
    "history | less\n",
    "```\n",
    "\n",
    "You can use the arrow keys to scroll in `less`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### searching through the text\n",
    "The `less` command has a vast array of incredible helpful keyboard shortcuts. We don't have time to go into all of them, but one of the most useful is the ability to search within the text.\n",
    "\n",
    "To search down through the text...\n",
    "1. Press the forward-slash `/`\n",
    "2. Type your search keyword\n",
    "3. Press _enter_\n",
    "\n",
    "To search up from your current position, use a question-mark `?` instead of a forward-slash.\n",
    "\n",
    "**You try**: Search for `ls` commands within `history`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Got long-running commands? Use job control."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Scenario:** You're about to execute a commmand that will take a long time.\n",
    "\n",
    "**Solution:** Send the command to the background by appending an ampersand `&`. Keep working while it runs.\n",
    "\n",
    "We'll use the `sleep` command to illustrate how this works, but you could use any command in theory."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `sleep`: do nothing for n seconds\n",
    "```bash\n",
    "sleep num_seconds\n",
    "```\n",
    "Running `sleep 3` will cause `sleep` to execute for 3 seconds, doing absolutely nothing in the meantime."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "sleep 3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's try sending `sleep` to the background while it runs.\n",
    "```bash\n",
    "sleep 7 &\n",
    "```\n",
    "Now, we can continue working while it executes."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## backgrounding a job while it is running"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I'm generally not put-together enough to anticipate when I need to background a job. So is there a way to background a job after you've already started executing it?\n",
    "\n",
    "Yup!\n",
    "1. Press ctrl+z to temporarily pause the command\n",
    "2. Send it to the background using the `bg` command\n",
    "\n",
    "For example,\n",
    "1. `sleep 120`\n",
    "2. _ctrl+z_\n",
    "3. `bg`\n",
    "\n",
    "To bring it back to the foreground, use the `fg` command after backgrounding it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## ending a command prematurely"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "You can also permanently stop a command, if you need to. Usually, if your command isn't backgrounded, you can do this by pressing _ctrl+c_.\n",
    "\n",
    "For example,\n",
    "1. `sleep 120`\n",
    "2. _ctrl+c_"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Otherwise, if your command is backgrounded or `ctrl+c` isn't working for whatever reason, you can use the `kill` command.\n",
    "\n",
    "There are two ways of doing this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `kill <PID>`\n",
    "Every running command has a process identifier (PID). To list all processes and their PIDs, use the `ps` command. We use the `-u` option to retrieve only those processes that our **u**ser owns.\n",
    "```bash\n",
    "ps -u $(whoami)\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### `kill <job ID>`\n",
    "Every backgrounded command has a job identifier. To list all jobs and their IDs, use the `jobs` command.\n",
    "```bash\n",
    "jobs\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Working with larger files\n",
    "Bioinformaticians frequently keep commands compressed so they don't take up too much space."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `gzip`: compressing a file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "gzip [path_to_file]...\n",
    "```\n",
    "You can compress a file using the `gzip` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gzip sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how the file has a `.gz` extension now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And if we try to take a look inside, it'll just look like gibberish."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head sequences.txt.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### decompress a file\n",
    "Let's revert the file back to the way it was. We can use the `-d` switch to activate decompression."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "gzip -d sequences.txt.gz\n",
    "ls"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "head sequences.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Ok, now that we know how to deal with small files, let's level-up to \"BIG DATA\"!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `wget`: downloading a file from a URL"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "wget url...\n",
    "```\n",
    "For example, let's download chr21 from the hg38 reference genome:\n",
    "\n",
    "https://hgdownload.soe.ucsc.edu/goldenPath/hg38/chromosomes/chr21.fa.gz (also at https://bit.ly/3zjmX7L)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the URL for chr21 is stored at the bottom of the hidden file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo $(tail -n1 .you_found_me.txt)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "wget $(tail -n1 .you_found_me.txt)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check: do we have it?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ls "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `du`: print a file's **d**isk **u**sage\n",
    "```bash\n",
    "du path_to_file...\n",
    "```\n",
    "In addition to `ls`, we can use the `du` to check how big a file is."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "How big is the file we downloaded compared to the sequences file? We can use the `-h` flag to print the size in human-readable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "du -h sequences.txt chr21.fa.gz"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's several 1000 times larger!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `zcat`: output decompressed text from a gzipped file"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "zcat [path_to_file]...\n",
    "```\n",
    "We want to take a look at the first ten lines of this file, but it's compressed. Rather than uncompress it all and waste storage space, let's see if we can obtain what we want using the `zcat` command."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "zcat chr21.fa.gz | head"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first set of lines of chr21 represent unsequenced heterochromatin. Let's see if we can find real sequences by opening the file inside the `zless` command...."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## `zless`: `less` but for gzipped files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```bash\n",
    "zcat [path_to_file]...\n",
    "```\n",
    "Let's open this file in `less` and then look around for a nucleotide like a **g**.\n",
    "\n",
    "```bash\n",
    "zless chr21.fa.gz\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we finish this section, let's just clean up after ourselves."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "rm chr21.fa.gz\n",
    "ls"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Logic control\n",
    "Bash has for loops and if statements, just like python and R!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## for loops"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "A for loop is written like this:\n",
    "```bash\n",
    "for my_var in <space_separated_list>; do\n",
    "    cmd \"$my_var\"\n",
    "done\n",
    "```\n",
    "A variable, in this case called `my_var`, is set to each item in the loop. The items are specified as a space separated list. Inside the body of the loop, we can run a `cmd` using the value of each item."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for name in Dolly Kitty Poppet; do\n",
    "    echo Hello \"$name!\";\n",
    "done;"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As an example, let's iterate over all of the scripts in the `scripts/` directory and print out their paths. Recall that expressions with wildcards like the asterisk `*` get expanded to a space-separated list prior to command execution."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for file in scripts/*.sh; do\n",
    "    echo \"check out this file called $file\"\n",
    "done"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## if statements\n",
    "An if-statement is written like this:\n",
    "```bash\n",
    "if [ <logical_statement> ]; then\n",
    "    cmd\n",
    "fi\n",
    "```\n",
    "The logical statement is something that we can test for.\n",
    "\n",
    "Here's an example. First, we make a variable called `$you` and then we test whether `$you` equals \"bored\"."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "you=excited\n",
    "if [ $you != bored ]; then\n",
    "    echo \"okay onto the next adventure!\"\n",
    "fi"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Bash: also a programming language"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What is _bash_ anyways?\n",
    "1. A shell programming language used within terminals\n",
    "2. An application that reads and interpets all of the commands (ie code) that you type into the terminal\n",
    "\n",
    "**Takeaway**: You can write scripts composed from all of the commands we learned!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We've written a couple of scripts for you, already. Why don't you take a look?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat scripts/hello.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## What is a shebang?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The first line of `hello.sh` is called a _shebang_. It denotes the command that should be used to execute your script.\n",
    "\n",
    "In this case, we're using the `bash` command, since we're writing our script in the `bash` programming language. But we could have used any command -- like `python`, for example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Executing a script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To execute the script, just type the path to it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "scripts/hello.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Passing arguments to a script"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Scripts are, in fact, commands themselves! Within a script, `$1` refers to the first argument to the script, `$2` refers to the second argument, etc...\n",
    "\n",
    "We demonstrate this in `hello_you.sh`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat scripts/hello_you.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scripts/hello_you.sh me"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Calling scripts from within other scripts "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat scripts/hello_to_all_of_you.sh"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scripts/hello_to_all_of_you.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Create your own script!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Try to create your own script.\n",
    "1. Navigate to the `module-1-programming/bash_playground/scripts/` directory in DataHub, and create a new file called `my_hello.sh`.\n",
    "2. Copy the shebang from our other scripts and add your own command.\n",
    "3. Every time you want to convert a regular file into a script, you'll first need to designate the file as _executable_. You should do this now using the `chmod` command:\n",
    "\n",
    "    ```bash\n",
    "    chmod u+x scripts/my_hello.sh\n",
    "    ```\n",
    "4. Execute your script.\n",
    "\n",
    "    ```bash\n",
    "    scripts/my_hello.sh\n",
    "    ```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Making a script globally availale using the special `$PATH` environment variable"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I don’t want to have to type out the full path to `hello_you.sh` every time.\n",
    "\n",
    "How can I make it so that I can execute `hello_you.sh` from anywhere by just typing `hello_you.sh`?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "`$PATH` is a special _environment_ variable that is created by _bash_ on startup. It contains a colon-separated list of directories.\n",
    "\n",
    "Whenever you type a command, _bash_ will look through these directories to find an executable file that matches it.\n",
    "\n",
    "What does `$PATH` look like right now?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "echo $PATH"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Notice how `/usr/bin` is listed in `$PATH`? Try this:\n",
    "```bash\n",
    "ls /usr/bin | less\n",
    "```\n",
    "Do you recognize any of those?! They're the commands we've been learning about today!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "_All of the commands we've been using are just executable files in a folder on the filesystem somewhere!_\n",
    "\n",
    "Can we make our own scripts into commands?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's add the `scripts/` directory to the end of `$PATH`. Note that we can obtain the absolute path to the `scripts/` directory using `pwd`:\n",
    "```\n",
    "echo \"$(pwd)/scripts\"\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PATH=\"$PATH:$(pwd)/scripts\"\n",
    "echo \"$PATH\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "And now, we should be able to execute our scripts as commands from anywhere :D"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "scripts/hello.sh"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# We just scratched the surface! How can I learn more?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. [The missing semester of your CS education: an open MIT course](https://missing.csail.mit.edu/)\n",
    "2. [A curated list of resources for terminal-using bioinformaticians](https://docs.google.com/document/d/1cOjKC4OI4rsQUl_68PPHCn8vO11lwhpSWyJrGDZT80Q)\n",
    "3. [Software carpentry](https://swcarpentry.github.io/shell-novice/)\n",
    "4. [explainshell.com](http://explainshell.com) and help/manual pages\n",
    "5. It’s dry reading, but [the GNU bash manual](https://www.gnu.org/software/bash/manual/bash.html) is really quite good\n",
    "6. Other useful topics for bioinformaticians that we weren't able to cover today\n",
    "\t- git\n",
    "\t- ssh\n",
    "\t- sshfs\n",
    "\t- conda and conda environments\n",
    "\t- Snakemake\n",
    "\t- the screen command\n",
    "\t- process substitution\n",
    "\t- exit codes\n",
    "\t- functions\n",
    "\t- aliases\n",
    "\t- bash startup\n",
    "\t- using a computing cluster\n",
    "\t- regexes\n",
    "\t- manipulating strings\n",
    "\t- shell expansions\n",
    "\t- the diff and comm commands\n",
    "\t- file and directory permissions\n",
    "\t- the find and xargs commands\n",
    "\t- arrays\n",
    "\t- shell arithmetic\n",
    "\t- while loops\n",
    "\t- signals\n",
    "\t- the top command\n",
    "\t- the datamash tool\n",
    "\t- the /dev special files"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Bash",
   "language": "bash",
   "name": "bash"
  },
  "language_info": {
   "codemirror_mode": "shell",
   "file_extension": ".sh",
   "mimetype": "text/x-sh",
   "name": "bash"
  },
  "toc-autonumbering": false,
  "toc-showcode": false,
  "toc-showmarkdowntxt": false,
  "toc-showtags": false
 },
 "nbformat": 4,
 "nbformat_minor": 4
}