6-Getting-Data-from-Web.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "ISRC Python Workshop: Scrape web data\n",
    "\n",
    "___Getting data from Web: Parse by Hand!___"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<hr>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "@author: Zhiya Zuo\n",
    "\n",
    "@email: zhiya-zuo@uiowa.edu"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Introduction"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Sometimes, they have APIs but they have no well-written packages in the language you prefer (e.g. only Java but no Python libraries). Even worse, there may not be APIs for the public and we have to design a scraper to retrieve all the relevant informaiton we want. In such cases, we can manually build our own wrapper functions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Preliminiary examples"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examples from <a href=\"https://www.w3schools.com/html/tryit.asp?filename=tryhtml_basic_document\" target=\"blank_\">w3schools</a>."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "```html\n",
    "<!DOCTYPE html>\n",
    "<html>\n",
    "<body>\n",
    "\n",
    "<h1>My First Heading</h1>\n",
    "\n",
    "<p>My 1st paragraph.</p>\n",
    "<p>My 2nd paragraph.</p>\n",
    "<p>My 3rd paragraph.</p>\n",
    "\n",
    "</body>\n",
    "</html>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save this code to your disk as `sample.html` (or any other name). We will use a great library called ___`Beautiful Soup`___ to read the contents from Python. You may also need to install lxml, which is for parsing specific formats (e.g., html and xml)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T00:59:26.608656Z",
     "start_time": "2019-03-15T00:59:26.448596Z"
    }
   },
   "outputs": [],
   "source": [
    "## Do the following if you have not\n",
    "# pip install beautifulsoup4 lxml\n",
    "from bs4 import BeautifulSoup as Soup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T00:59:26.766935Z",
     "start_time": "2019-03-15T00:59:26.761404Z"
    }
   },
   "outputs": [],
   "source": [
    "with open(\"sample-data/sample.html\", \"r\") as sample:\n",
    "    sample_contents = sample.read()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The structure of HTML is not displayed properly without BeautifulSoup, which is really hand!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T00:59:28.135370Z",
     "start_time": "2019-03-15T00:59:28.123466Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'<!DOCTYPE html>\\n<html>\\n<body>\\n\\n<h1>My First Heading</h1>\\n\\n<p>My 1st paragraph.</p>\\n<p>My 2nd paragraph.</p>\\n<p>My 3rd paragraph.</p>\\n\\n</body>\\n</html>\\n'"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sample_contents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T00:59:28.612880Z",
     "start_time": "2019-03-15T00:59:28.604759Z"
    }
   },
   "outputs": [],
   "source": [
    "sample_soup = Soup(sample_contents, 'lxml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By printing it, we can see the exact contents as shown above with proper indentation"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T00:59:29.159746Z",
     "start_time": "2019-03-15T00:59:29.155620Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html>\n",
      "<html>\n",
      " <body>\n",
      "  <h1>\n",
      "   My First Heading\n",
      "  </h1>\n",
      "  <p>\n",
      "   My 1st paragraph.\n",
      "  </p>\n",
      "  <p>\n",
      "   My 2nd paragraph.\n",
      "  </p>\n",
      "  <p>\n",
      "   My 3rd paragraph.\n",
      "  </p>\n",
      " </body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "print(sample_soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Get the contents of interest: all the `p`'s\n",
    "\n",
    "_`p` means paragraph in html. Check more tag definitions on https://w3schools.org._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T00:59:30.637396Z",
     "start_time": "2019-03-15T00:59:30.634466Z"
    }
   },
   "outputs": [],
   "source": [
    "p_tags = sample_soup.find_all(\"p\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T00:59:30.973087Z",
     "start_time": "2019-03-15T00:59:30.965799Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[<p>My 1st paragraph.</p>, <p>My 2nd paragraph.</p>, <p>My 3rd paragraph.</p>]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_tags"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T00:59:31.195724Z",
     "start_time": "2019-03-15T00:59:31.187710Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<p>My 2nd paragraph.</p>"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "p_tags[1]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "For each of the `p` tag, we get the textual value out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T00:59:32.440325Z",
     "start_time": "2019-03-15T00:59:32.436504Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "My 1st paragraph.\n",
      "My 2nd paragraph.\n",
      "My 3rd paragraph.\n"
     ]
    }
   ],
   "source": [
    "for p in p_tags:\n",
    "    print(p.text)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### A real example"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As you can see, this is very straightforward. Let's use a real website for illustration. For example, if we are interested in company profiles, we can scrape from [Google Finance](https://www.google.com/finance). We will be using <a href=\"https://www.google.com/finance?q=NYSE%3AIBM&ei=Ij62WPHgGdSLmAHrja_wAQ\" target=\"_blank\">IBM's profile</a> as an example. However, you may find off-the-shelf packages. We will only use this for an introduction on how to scrape manually."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "To view the \"text style\" or the real structure of a web page, you can use ___`developer tools`___ function in your browser. For example, you can see something like this. If you move your mouse to a place, the console will show you the corresponding tags in the source html files. You will find that the description text is located within a `p` tag."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"https://i.imgur.com/BSg9ixm.png\" width=\"1000\">"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that [`requests`](http://docs.python-requests.org/) is a convenient package for sending HTTP requests."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:00:07.490430Z",
     "start_time": "2019-03-15T01:00:07.365897Z"
    }
   },
   "outputs": [],
   "source": [
    "import requests"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:03:09.166347Z",
     "start_time": "2019-03-15T01:03:08.641354Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ibm_url = \"https://finance.yahoo.com/quote/IBM/profile?p=IBM\"\n",
    "r = requests.get(ibm_url)\n",
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:05:54.889840Z",
     "start_time": "2019-03-15T01:05:54.881016Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'; Facts</title><meta name=\"keywords\" content=\"company profile,business summary,company management,company introduction,key statistics,officers\"/><meta http-equiv=\"x-dns-prefetch-control\" content=\"on\"/><meta property=\"twitter:dnt\" content=\"on\"/><meta property=\"fb:app_id\" content=\"90376669494\"/><meta name=\"theme-color\" content=\"#400090\"/><meta name=\"viewport\" content=\"width=device-width, initial-scale=1\"/><meta name=\"description\" lang=\"en-US\" content=\"See the company profile for International Business Machines (IBM) including business summary, industry/sector information, number of employees, business summary, corporate governance, key executives and their compensation.\"/><link rel=\"manifest\" '"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.text[300:1000]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2018-03-03T04:20:18.387485Z",
     "start_time": "2018-03-03T04:20:18.356444Z"
    }
   },
   "source": [
    "Convert it to a soup object"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:05:56.532482Z",
     "start_time": "2019-03-15T01:05:56.495219Z"
    }
   },
   "outputs": [],
   "source": [
    "ibm_soup = Soup(r.text, 'lxml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Find the correponding tag. Note that `class_` has a trailing underscore `_`"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:12:12.262796Z",
     "start_time": "2019-03-15T01:12:12.246056Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--------------------\n",
      "International Business Machines Corporation operates as an integrated technology and services company worldwide. Its Cognitive Solutions segment offers Watson, a computing platform that interacts in language, processes big data, and learns from interactions with people and computers. This segment also offers analytics and data management platforms, cloud data services, enterprise social software, and talent management and industry solutions; and transaction processing software for use in banking, airlines, and retail industries. The company's Global Business Services segment offers business consulting services; delivers system integration, application management, maintenance, and support services for packaged software applications; and finance, procurement, talent and engagement, and industry-specific business process outsourcing services. Its Technology Services & Cloud Platforms segment provides cloud, project-based, outsourcing, and other managed services for enterprise IT infrastructure environments. This segment also offers technical support, and software and solution support; and integration software solutions. The company's Systems segment offers servers for businesses, cloud service providers, and scientific computing organizations; data storage products and solutions; and z/OS, an enterprise operating system. Its Global Financing segment provides lease, installment payment plans, and loan financing services; short-term working capital financing to suppliers, distributors, and resellers; and remanufacturing and remarketing services. International Business Machines Corporation has a strategic partnership with Samsung Electronics to manufacture microprocessors; and collaboration with HCL Technologies in designing hybrid cloud of organizations. The company was formerly known as Computing-Tabulating-Recording Co. and changed its name to International Business Machines Corporation in 1924. The company was founded in 1911 and is headquartered in Armonk, New York.\n",
      "--------------------\n"
     ]
    }
   ],
   "source": [
    "summary_tag = ibm_soup.find(\"p\", class_=\"Mt(15px) Lh(1.6)\")\n",
    "print('--------------------')\n",
    "print(summary_tag.text)\n",
    "print('--------------------')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With this in mind, you can scrape almost any webpage of interest. Other formats such as <a href=\"http://www.json.org/\" target=\"_blank\">JSON</a> and <a href=\"https://www.w3.org/XML/\" target=\"_blank\">XML</a> do have high similarities and a few differences. They are not very difficult to know the basics! (We've talked about this in [the previous notebook](https://github.com/zhiyzuo/uiowa-isrc-python/blob/master/5-Getting-Data-Using-APIs.ipynb)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***But keep in mind that you should act politely, with propoer permission!! To find out whether specific paths/contents are allowed to be scraped, you can check their ___`robots.txt`___. For example, <a href=\"https://www.google.com/robots.txt\" target=\"_blank\">here's</a> the permission information set by Google.***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### More than one page"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Simple case: Pagination"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "There are situtaions where we need to scrape data that is in different pages. We will be using [this website](http://spidyquotes.herokuapp.com/) as an example."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After we clikc on `next page`, we can see that the address bar will become http://spidyquotes.herokuapp.com/page/2/, which clearly shows that we can jump to any page with a proper setting. Let's try!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:13:55.365875Z",
     "start_time": "2019-03-15T01:13:55.361651Z"
    }
   },
   "outputs": [],
   "source": [
    "quote_url = \"http://spidyquotes.herokuapp.com/page/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's use `2`:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:13:58.458960Z",
     "start_time": "2019-03-15T01:13:58.452937Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://spidyquotes.herokuapp.com/page/2'"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quote_2 = quote_url + \"2\"\n",
    "quote_2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:13:59.847868Z",
     "start_time": "2019-03-15T01:13:59.394612Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get(quote_2)\n",
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:13:59.978353Z",
     "start_time": "2019-03-15T01:13:59.958233Z"
    }
   },
   "outputs": [],
   "source": [
    "quote_2_soup = Soup(r.text, 'lxml')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "By checking the HTML structure using Chrome's `developer tools`, we find that all the quotes are `div` tags with class `quote`. Within each of the `quote div`, there are 3 elements: quote text, author, and tags:\n",
    "```html\n",
    "<div class=\"quote\" itemscope=\"\" itemtype=\"http://schema.org/CreativeWork\">\n",
    "        <span class=\"text\" itemprop=\"text\">“If you can't explain it to a six year old, you don't understand it yourself.”</span>\n",
    "        <span>by <small class=\"author\" itemprop=\"author\">Albert Einstein</small>\n",
    "        <a href=\"/author/Albert-Einstein\">(about)</a>\n",
    "        </span>\n",
    "        <div class=\"tags\">\n",
    "            Tags:\n",
    "            <meta class=\"keywords\" itemprop=\"keywords\" content=\"simplicity,understand\"> \n",
    "            \n",
    "            <a class=\"tag\" href=\"/tag/simplicity/page/1/\">simplicity</a>\n",
    "            \n",
    "            <a class=\"tag\" href=\"/tag/understand/page/1/\">understand</a>\n",
    "            \n",
    "        </div>\n",
    "    </div>\n",
    "```"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With all these in mind, we can build a very sinmple wrapper function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:03.829401Z",
     "start_time": "2019-03-15T01:14:02.667008Z"
    }
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:03.973455Z",
     "start_time": "2019-03-15T01:14:03.963678Z"
    }
   },
   "outputs": [],
   "source": [
    "def parse_quote(quote_div):\n",
    "    '''Each tag has 3 elements. We just retrieve text and author here'''\n",
    "    span_tags = quote_div.find_all('span')\n",
    "    quote_text = span_tags[0].text\n",
    "    quote_author = span_tags[1].find('small').text\n",
    "    return {'author': quote_author, 'quote': quote_text}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:04.066871Z",
     "start_time": "2019-03-15T01:14:04.062090Z"
    }
   },
   "outputs": [],
   "source": [
    "# Often, constants are stored in variables with all caps.\n",
    "QUOTE_URL = \"http://spidyquotes.herokuapp.com/page/\"\n",
    "def get_quote(page=1):\n",
    "    r = requests.get(QUOTE_URL+str(page))\n",
    "    soup = Soup(r.text, 'lxml')\n",
    "    quote_div_list = soup.find_all('div', class_='quote')\n",
    "    quote_df = pd.DataFrame([parse_quote(div) for div in quote_div_list])\n",
    "    return quote_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:04.840466Z",
     "start_time": "2019-03-15T01:14:04.506107Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>author</th>\n",
       "      <th>quote</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Marilyn Monroe</td>\n",
       "      <td>“This life is what you make it. No matter what...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>J.K. Rowling</td>\n",
       "      <td>“It takes a great deal of bravery to stand up ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Albert Einstein</td>\n",
       "      <td>“If you can't explain it to a six year old, yo...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Bob Marley</td>\n",
       "      <td>“You may not be her first, her last, or her on...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Dr. Seuss</td>\n",
       "      <td>“I like nonsense, it wakes up the brain cells....</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Douglas Adams</td>\n",
       "      <td>“I may not have gone where I intended to go, b...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Elie Wiesel</td>\n",
       "      <td>“The opposite of love is not hate, it's indiff...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Friedrich Nietzsche</td>\n",
       "      <td>“It is not a lack of love, but a lack of frien...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mark Twain</td>\n",
       "      <td>“Good friends, good books, and a sleepy consci...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Allen Saunders</td>\n",
       "      <td>“Life is what happens to us while we are makin...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                author                                              quote\n",
       "0       Marilyn Monroe  “This life is what you make it. No matter what...\n",
       "1         J.K. Rowling  “It takes a great deal of bravery to stand up ...\n",
       "2      Albert Einstein  “If you can't explain it to a six year old, yo...\n",
       "3           Bob Marley  “You may not be her first, her last, or her on...\n",
       "4            Dr. Seuss  “I like nonsense, it wakes up the brain cells....\n",
       "5        Douglas Adams  “I may not have gone where I intended to go, b...\n",
       "6          Elie Wiesel  “The opposite of love is not hate, it's indiff...\n",
       "7  Friedrich Nietzsche  “It is not a lack of love, but a lack of frien...\n",
       "8           Mark Twain  “Good friends, good books, and a sleepy consci...\n",
       "9       Allen Saunders  “Life is what happens to us while we are makin..."
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_quote(2)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can further come up with a function that takes in a list of pages"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:05.818309Z",
     "start_time": "2019-03-15T01:14:05.813729Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_quotes(page_list=[1]):\n",
    "    return pd.concat([get_quote(pg) for pg in page_list], \n",
    "                     axis=0, ignore_index=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:06.865938Z",
     "start_time": "2019-03-15T01:14:06.365568Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>author</th>\n",
       "      <th>quote</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Albert Einstein</td>\n",
       "      <td>“The world as we have created it is a process ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>J.K. Rowling</td>\n",
       "      <td>“It is our choices, Harry, that show what we t...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Albert Einstein</td>\n",
       "      <td>“There are only two ways to live your life. On...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>Jane Austen</td>\n",
       "      <td>“The person, be it gentleman or lady, who has ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>Marilyn Monroe</td>\n",
       "      <td>“Imperfection is beauty, madness is genius and...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>Albert Einstein</td>\n",
       "      <td>“Try not to become a man of success. Rather be...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>André Gide</td>\n",
       "      <td>“It is better to be hated for what you are tha...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Thomas A. Edison</td>\n",
       "      <td>“I have not failed. I've just found 10,000 way...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Eleanor Roosevelt</td>\n",
       "      <td>“A woman is like a tea bag; you never know how...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Steve Martin</td>\n",
       "      <td>“A day without sunshine is like, you know, nig...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              author                                              quote\n",
       "0    Albert Einstein  “The world as we have created it is a process ...\n",
       "1       J.K. Rowling  “It is our choices, Harry, that show what we t...\n",
       "2    Albert Einstein  “There are only two ways to live your life. On...\n",
       "3        Jane Austen  “The person, be it gentleman or lady, who has ...\n",
       "4     Marilyn Monroe  “Imperfection is beauty, madness is genius and...\n",
       "5    Albert Einstein  “Try not to become a man of success. Rather be...\n",
       "6         André Gide  “It is better to be hated for what you are tha...\n",
       "7   Thomas A. Edison  “I have not failed. I've just found 10,000 way...\n",
       "8  Eleanor Roosevelt  “A woman is like a tea bag; you never know how...\n",
       "9       Steve Martin  “A day without sunshine is like, you know, nig..."
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_quotes([1,2])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Inifinite scroll?"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In some cases, we may see infinite scroll pages such as Twitter/Facebook. The quote website also offers a great example for us to try scraping on this type of page: http://spidyquotes.herokuapp.com/scroll"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When we examine such pages, what we should first do is to look at the ___network___ tab in the `developer tool`:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![Imgur](https://i.imgur.com/Nmna5nc.gif)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can then see that when we scroll down, there are actually hidden API calls! Therefore, we can just call the APIS to ___simulate what the javascript's doing___ to get the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:13.331436Z",
     "start_time": "2019-03-15T01:14:13.327217Z"
    }
   },
   "outputs": [],
   "source": [
    "QUOTE_API = \"http://spidyquotes.herokuapp.com/api/quotes\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's test with page 2. Note that the request method is `GET` as shown in the `developer's tool`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:14.214717Z",
     "start_time": "2019-03-15T01:14:14.049594Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "200"
      ]
     },
     "execution_count": 31,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r = requests.get(QUOTE_API, params={'page': 2})\n",
    "r.status_code"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:14.637496Z",
     "start_time": "2019-03-15T01:14:14.633480Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'http://spidyquotes.herokuapp.com/api/quotes?page=2'"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "r.url"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:15.665982Z",
     "start_time": "2019-03-15T01:14:15.659839Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "dict_keys(['has_next', 'page', 'quotes', 'tag', 'top_ten_tags'])"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quote_json = r.json()\n",
    "quote_json.keys()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:16.121868Z",
     "start_time": "2019-03-15T01:14:16.111456Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[{'author': {'goodreads_link': '/author/show/82952.Marilyn_Monroe',\n",
       "   'name': 'Marilyn Monroe',\n",
       "   'slug': 'Marilyn-Monroe'},\n",
       "  'tags': ['friends',\n",
       "   'heartbreak',\n",
       "   'inspirational',\n",
       "   'life',\n",
       "   'love',\n",
       "   'sisters'],\n",
       "  'text': \"“This life is what you make it. No matter what, you're going to mess up sometimes, it's a universal truth. But the good part is you get to decide how you're going to mess it up. Girls will be your friends - they'll act like it anyway. But just remember, some come, some go. The ones that stay with you through everything - they're your true best friends. Don't let go of them. Also remember, sisters make the best friends in the world. As for lovers, well, they'll come and go too. And baby, I hate to say it, most of them - actually pretty much all of them are going to break your heart, but you can't give up because if you give up, you'll never find your soulmate. You'll never find that half who makes you whole and that goes for everything. Just because you fail once, doesn't mean you're gonna fail at everything. Keep trying, hold on, and always, always, always believe in yourself, because if you don't, then who will, sweetie? So keep your head high, keep your chin up, and most importantly, keep smiling, because life's a beautiful thing and there's so much to smile about.”\"},\n",
       " {'author': {'goodreads_link': '/author/show/1077326.J_K_Rowling',\n",
       "   'name': 'J.K. Rowling',\n",
       "   'slug': 'J-K-Rowling'},\n",
       "  'tags': ['courage', 'friends'],\n",
       "  'text': '“It takes a great deal of bravery to stand up to our enemies, but just as much to stand up to our friends.”'},\n",
       " {'author': {'goodreads_link': '/author/show/9810.Albert_Einstein',\n",
       "   'name': 'Albert Einstein',\n",
       "   'slug': 'Albert-Einstein'},\n",
       "  'tags': ['simplicity', 'understand'],\n",
       "  'text': \"“If you can't explain it to a six year old, you don't understand it yourself.”\"},\n",
       " {'author': {'goodreads_link': '/author/show/25241.Bob_Marley',\n",
       "   'name': 'Bob Marley',\n",
       "   'slug': 'Bob-Marley'},\n",
       "  'tags': ['love'],\n",
       "  'text': \"“You may not be her first, her last, or her only. She loved before she may love again. But if she loves you now, what else matters? She's not perfect—you aren't either, and the two of you may never be perfect together but if she can make you laugh, cause you to think twice, and admit to being human and making mistakes, hold onto her and give her the most you can. She may not be thinking about you every second of the day, but she will give you a part of her that she knows you can break—her heart. So don't hurt her, don't change her, don't analyze and don't expect more than she can give. Smile when she makes you happy, let her know when she makes you mad, and miss her when she's not there.”\"},\n",
       " {'author': {'goodreads_link': '/author/show/61105.Dr_Seuss',\n",
       "   'name': 'Dr. Seuss',\n",
       "   'slug': 'Dr-Seuss'},\n",
       "  'tags': ['fantasy'],\n",
       "  'text': '“I like nonsense, it wakes up the brain cells. Fantasy is a necessary ingredient in living.”'},\n",
       " {'author': {'goodreads_link': '/author/show/4.Douglas_Adams',\n",
       "   'name': 'Douglas Adams',\n",
       "   'slug': 'Douglas-Adams'},\n",
       "  'tags': ['life', 'navigation'],\n",
       "  'text': '“I may not have gone where I intended to go, but I think I have ended up where I needed to be.”'},\n",
       " {'author': {'goodreads_link': '/author/show/1049.Elie_Wiesel',\n",
       "   'name': 'Elie Wiesel',\n",
       "   'slug': 'Elie-Wiesel'},\n",
       "  'tags': ['activism',\n",
       "   'apathy',\n",
       "   'hate',\n",
       "   'indifference',\n",
       "   'inspirational',\n",
       "   'love',\n",
       "   'opposite',\n",
       "   'philosophy'],\n",
       "  'text': \"“The opposite of love is not hate, it's indifference. The opposite of art is not ugliness, it's indifference. The opposite of faith is not heresy, it's indifference. And the opposite of life is not death, it's indifference.”\"},\n",
       " {'author': {'goodreads_link': '/author/show/1938.Friedrich_Nietzsche',\n",
       "   'name': 'Friedrich Nietzsche',\n",
       "   'slug': 'Friedrich-Nietzsche'},\n",
       "  'tags': ['friendship',\n",
       "   'lack-of-friendship',\n",
       "   'lack-of-love',\n",
       "   'love',\n",
       "   'marriage',\n",
       "   'unhappy-marriage'],\n",
       "  'text': '“It is not a lack of love, but a lack of friendship that makes unhappy marriages.”'},\n",
       " {'author': {'goodreads_link': '/author/show/1244.Mark_Twain',\n",
       "   'name': 'Mark Twain',\n",
       "   'slug': 'Mark-Twain'},\n",
       "  'tags': ['books', 'contentment', 'friends', 'friendship', 'life'],\n",
       "  'text': '“Good friends, good books, and a sleepy conscience: this is the ideal life.”'},\n",
       " {'author': {'goodreads_link': '/author/show/276029.Allen_Saunders',\n",
       "   'name': 'Allen Saunders',\n",
       "   'slug': 'Allen-Saunders'},\n",
       "  'tags': ['fate', 'life', 'misattributed-john-lennon', 'planning', 'plans'],\n",
       "  'text': '“Life is what happens to us while we are making other plans.”'}]"
      ]
     },
     "execution_count": 34,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "quote_json['quotes']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Therefore, for each page, we can directly parse the resulting JSON instead of parsing the HTML manually!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:26.178152Z",
     "start_time": "2019-03-15T01:14:26.173695Z"
    }
   },
   "outputs": [],
   "source": [
    "def get_quote_api(page):\n",
    "    r = requests.get(QUOTE_API, params={'page': page})\n",
    "    if r.status_code != 200:\n",
    "        print('Request failed: %s'%r.text)\n",
    "        return None\n",
    "    quotes = r.json()['quotes']\n",
    "    return pd.DataFrame([{'quote': q['text'], 'author': q['author']['name']} for q in quotes])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2019-03-15T01:14:26.677013Z",
     "start_time": "2019-03-15T01:14:26.528925Z"
    }
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>author</th>\n",
       "      <th>quote</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>Charles Bukowski</td>\n",
       "      <td>“That's the problem with drinking, I thought, ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>Suzanne Collins</td>\n",
       "      <td>“You don’t forget the face of the person who w...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>Suzanne Collins</td>\n",
       "      <td>“Remember, we're madly in love, so it's all ri...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>C.S. Lewis</td>\n",
       "      <td>“To love at all is to be vulnerable. Love anyt...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>J.R.R. Tolkien</td>\n",
       "      <td>“Not all those who wander are lost.”</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>J.K. Rowling</td>\n",
       "      <td>“Do not pity the dead, Harry. Pity the living,...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>Ernest Hemingway</td>\n",
       "      <td>“There is nothing to writing. All you do is si...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>Ralph Waldo Emerson</td>\n",
       "      <td>“Finish each day and be done with it. You have...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>Mark Twain</td>\n",
       "      <td>“I have never let my schooling interfere with ...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>Dr. Seuss</td>\n",
       "      <td>“I have heard there are troubles of more than ...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                author                                              quote\n",
       "0     Charles Bukowski  “That's the problem with drinking, I thought, ...\n",
       "1      Suzanne Collins  “You don’t forget the face of the person who w...\n",
       "2      Suzanne Collins  “Remember, we're madly in love, so it's all ri...\n",
       "3           C.S. Lewis  “To love at all is to be vulnerable. Love anyt...\n",
       "4       J.R.R. Tolkien               “Not all those who wander are lost.”\n",
       "5         J.K. Rowling  “Do not pity the dead, Harry. Pity the living,...\n",
       "6     Ernest Hemingway  “There is nothing to writing. All you do is si...\n",
       "7  Ralph Waldo Emerson  “Finish each day and be done with it. You have...\n",
       "8           Mark Twain  “I have never let my schooling interfere with ...\n",
       "9            Dr. Seuss  “I have heard there are troubles of more than ..."
      ]
     },
     "execution_count": 36,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "get_quote_api(7)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Done!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Conclusion"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see, it is not that difficult to parse HTML pages. By manually parsing the pages, we in turn gain more control on how we want the results to be shaped. However, anti-scraping deign of websites may bring obstables in finding ways to retrieve valuable informaiton in automated ways, just as those infinite scroll design. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Note that the examples we are using here are relatively simple. There are cases that we cannot access the pagination/scoll simply by `requests` alone. In those cases, [Selenium](http://selenium-python.readthedocs.io/) will save our lifes by ___simulating Browsers___!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Some more tutorials/tools:\n",
    "\n",
    "- https://scrapy.org/\n",
    "- https://www.dataquest.io/blog/web-scraping-tutorial-python/\n",
    "- https://www.quora.com/Python-programming-language-1/How-is-BeautifulSoup-different-from-Scrapy"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.2"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {
    "height": "133px",
    "width": "254px"
   },
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": "block",
   "toc_window_display": true
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}