1-begin lesson.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# So you want to scrape.\n",
    "\n",
    "## Observe, orient, decide, act\n",
    "\n",
    "Before you start any scraping task, you'll want to invest a chunk of time into figuring out what you actually have to work with. Time you spend on the front end could save you from having to start completely over a good chunk of the way through.\n",
    "\n",
    "So, let's look at what we have to work with. Because you still have that **0-start web server** tab running, you have a little web server going. Click this link to open it in your browser:\n",
    "\n",
    "http://localhost/www.tdcj.texas.gov/death_row/dr_executed_offenders.html"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is a localized partial copy of a Texas Department of Criminal Justice site, showing \"Executed offenders\" -- people put to death by the state. They seem to be numbered in orderm, most recent at the top, and appear to show 558 executions in about 36 years.\n",
    "\n",
    "There's no pagination. Each person has a link for more information, and a link to the final statement. \n",
    "\n",
    "![Executed offender page\"](support/mainpage.png \"Main executed offender page\")\n",
    "\n",
    "On the main page, you see things like first name, last name, race, gender, age and TDCJ number. The page is from the Texas Department of Criminal Justice, or TDCJ. This is an inmate number -- a unqiue identifier.\n",
    "\n",
    "The newest executee is listed as Braziel, Jr., Alvin. Leave your cursor over the \"Offender Information\" link for Braziel and your browser, in the bottom-left corner, will show you it's http://localhost/www.tdcj.texas.gov/death_row/dr_info/brazielalvin.html . This is a good sign; it's not _javascript:something_. You can work with this. Let's hit that link."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "What do you want to scrape? Probably all of it.\n",
    "\n",
    "![Stephen Colbert wants it all](support/giveittome.gif)\n",
    "\n",
    "How hard will it be to grab that big bunch of narrative stuff? In most browsers, you can right-click (maybe around the \"Name\") entry in the biography) and left-click on Inspect. You should see something like this.\n",
    "\n",
    "![Output of HTML inspector](support/tableinspect.png)\n",
    "\n",
    "At first blush, this all looks good. **(Hint: It won't be.)** On the right side we see the main part of this demographic information is all in a HTML table: * &lt;table class=\"table\\_deathrow indent\"&gt; *. That's emminently scrapable.\n",
    "\n",
    "Flip back to that initial web page, and let's get Python started."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests   # external dependency\n",
    "from pyquery import PyQuery as pq   # external dependency\n",
    "\n",
    "import csv\n",
    "from collections import OrderedDict\n",
    "\n",
    "# Requests is a great way to get web pages into Python.\n",
    "# PyQuery is a great way to actually rip apart those pages.\n",
    "# And the built-in CSV module lets you save your data so you can do stuff with it later.\n",
    "# And OrderedDict is a great storage mechanism. Ignore it for now."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "# What are we scraping? Well, we start at the main page.\n",
    "hosturl = \"http://localhost/www.tdcj.texas.gov/death_row/dr_executed_offenders.html\"\n",
    "r = requests.get(hosturl)\n",
    "html = r.content"
   ]
  },
  {
   "cell_type": "raw",
   "metadata": {},
   "source": [
    "You now have a web page. You have scraping software. What are you going to scrape?\n",
    "\n",
    "<h1>Death Row Information</h1>\n",
    "<h2>Executed Offenders</h2>\n",
    "<div class=\"overflow\">\n",
    "<table class=\"tdcj_table indent\" style=\"width: 98%\">\n",
    "    <caption>Executed Offenders</caption>\n",
    "  <tr>\n",
    "    <th style=\"text-align: center\" scope=\"col\">Execution</th>\n",
    "    <th style=\"text-align: center; width: 16%\" scope=\"col\">Link</th>\n",
    "    <th style=\"text-align: center; width: 13%\" scope=\"col\">Link</th>\n",
    "    <th style=\"text-align: center\" scope=\"col\">Last Name</th>\n",
    "    <th style=\"text-align: center\" scope=\"col\">First Name</th>\n",
    "    <th style=\"text-align: center; width: 7%\" scope=\"col\">TDCJ<br>Number</th>\n",
    "    <th style=\"text-align: center\" scope=\"col\">Age</th>\n",
    "    <th style=\"text-align: center\" scope=\"col\">Date</th>\n",
    "    <th style=\"text-align: center\" scope=\"col\">Race</th>\n",
    "    <th style=\"text-align: center\" scope=\"col\">County</th>\n",
    "  </tr>\n",
    "  <tr>\n",
    "    <td style=\"text-align: center\">558</td>\n",
    "    <td style=\"text-align: center\"><a href=\"dr_info/brazielalvin.html\" title=\"Offender Information for Joseph Garcia\">Offender Information</a></td>\n",
    "    <td style=\"text-align: center\"><a href=\"dr_info/brazielalvinlast.html\" title=\"Last Statement of Joseph Garcia\">Last Statement</a></td>\n",
    "    <td style=\"text-align: center\">Braziel, Jr.</td>\n",
    "    <td style=\"text-align: center\">Alvin</td>\n",
    "    <td style=\"text-align: center\">999393</td>\n",
    "    <td style=\"text-align: center\">43</td>\n",
    "    <td style=\"text-align: center\">12/11/2018</td>\n",
    "    <td style=\"text-align: center\">Black</td>\n",
    "    <td style=\"text-align: center\">Dallas</td>\n",
    "  </tr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Planning your scrape\n",
    "\n",
    "So it looks like all the stuff we want is inside this *table* tag, with a class of *tdcj\\_table*.\n",
    "\n",
    "Our HTML has been loaded in as a variable called *html*.\n",
    "\n",
    "The most basic PyQuery syntax:\n",
    "\n",
    "* The first parenthesis is the hunk of HTML you're trying to parse\n",
    "\n",
    "* The second parenthesis is the hunk of HTML you're trying to extract, a tag. You can add on specifics, like *\"p#somename\">* or *div.someclass*.\n",
    "\n",
    "So let's isolate the table first. How?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "# You get the table here ... ?\n",
    "\n",
    "\n",
    "\n",
    "# Now, we have the table. Your actual data is inside *tr* tags, table rows.\n",
    "# But the first table row contains header stuff, not actual data. Let's pull just the first real row of data,\n",
    "# which is row 0. How?\n",
    "\n",
    "# Slice it off."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'\\n    <td style=\"text-align: center\">558</td>\\n    <td style=\"text-align: center\"><a href=\"dr_info/brazielalvin.html\" title=\"Offender Information for Joseph Garcia\">Offender Information</a></td>\\n    <td style=\"text-align: center\"><a href=\"dr_info/brazielalvinlast.html\" title=\"Last Statement of Joseph Garcia\">Last Statement</a></td>\\n    <td style=\"text-align: center\">Braziel, Jr.</td>\\n    <td style=\"text-align: center\">Alvin</td>\\n    <td style=\"text-align: center\">999393</td>\\n    <td style=\"text-align: center\">43</td>\\n    <td style=\"text-align: center\">12/11/2018</td>\\n    <td style=\"text-align: center\">Black</td>\\n    <td style=\"text-align: center\">Dallas</td>\\n  '"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# This I'll show you. Let's look:\n",
    "\n",
    "row = pq(table)(\"tr\")[1]\n",
    "pq(row).html()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "558\n",
      "557\n",
      "556\n",
      "555\n",
      "554\n"
     ]
    }
   ],
   "source": [
    "# How do we get at that first nugget of information, the execution number?\n",
    "# What about the rest? Let's pull it together a little:\n",
    "\n",
    "for row in pq(table)(\"tr\")[1:6]:      # Look only at the first five rows of actual usable data\n",
    "    ExecutionNo = pq(row)(\"td\")[0].text\n",
    "    print(ExecutionNo)\n",
    "\n",
    "# Now scrape the rest of the stuff. Easy, right?\n",
    "# What data structure are you putting this into?"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Let's take a look at those URLs now:\n",
    "for row in pq(table)(\"tr\")[1:6]:      # Look only at the first five rows of actual usable data\n",
    "    pass"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}