Chapter04_CrawlingModels.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "def getPage(url):\n",
    "    \"\"\"\n",
    "    Utilty function used to get a Beautiful Soup object from a given URL\n",
    "    \"\"\"\n",
    "\n",
    "    session = requests.Session()\n",
    "    headers = {'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/39.0.2171.95 Safari/537.36',\n",
    "               'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8'}\n",
    "    try:\n",
    "        req = session.get(url, headers=headers)\n",
    "    except requests.exceptions.RequestException:\n",
    "        return None\n",
    "    bs = BeautifulSoup(req.text, 'html.parser')\n",
    "    return bs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Dealing with different website layouts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: Delivering inclusive urban access: 3 uncomfortable truths\n",
      "URL: https://www.brookings.edu/blog/future-development/2018/01/26/delivering-inclusive-urban-access-3-uncomfortable-truths/\n",
      "\n",
      "\n",
      "The past few decades have been filled with a deep optimism about the role of cities and suburbs across the world. These engines of economic growth host a majority of world population, are major drivers of economic innovation, and have created pathways to opportunities for untold amounts of people.\t\n",
      "Authors\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Jeffrey Gutman\n",
      "Senior Fellow - Global Economy and Development\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Adie Tomer\n",
      "Fellow - Metropolitan Policy Program\n",
      "\n",
      " Twitter\n",
      "AdieTomer\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "But all is not well within our so-called Urban Century. Rapid urbanization, rising gentrification, concentrated poverty, and shortages of basic infrastructure have combined to create spatial inequity in cities and suburbs across the globe. The challenges of housing, moving, and employing so many people have led to longer travel times, rising housing costs, and unsustainable public spending. Moreover, policymakers are questioning traditional policies and approaches.\n",
      "The past couple years, we’ve led a project at Brookings—Moving to Access—that responds to these spatial challenges by promoting the idea of connecting people to opportunities as a new foundational principle for 21st century urban development. This principle of accessibility is meant to be a corollary to the natural questions we ask ourselves everyday about the communities where we live: Is this the best location to access employment? Are there nearby schools and health services? Is there a market in the neighborhood? How can I get from here to there? Such choices are valid for those with sufficient income. But what about those with more limited resources and thus choices in terms of affordable housing and affordable transport?\n",
      "While economists, planners, and engineers have promoted accessibility for decades, the concept is more often found in textbooks than formal urban policies. In the first stage of this project, we worked with a team of experts to determine what has stalled practical implementation of appropriate policies and practices? “Delivering Inclusive Access,” a report of this initial work, offers a synthesis of what we found and where we believe researchers, policymakers, and practitioners can take this work next. The paper found three central challenges.\n",
      "The fallacy of the single indicator\n",
      "The current transport regime’s approach to measurement is one of outward elegance: The dominant pursuit is speed, and the primary way to measure it is congestion (or what slows us down). Many have come to label this approach a pursuit of “mobility.” It is seen through different, but often singular, measures of how congestion affects a specific roadway. Such singular measures are easily interpreted by policymakers and civil society and can be translated directly into economic analysis of related investments through timesavings. They also conveniently serve such purposes as the internationally agreed-upon Sustainable Development Goals. Yet they actually don’t answer the fundamental question of who can reach where, in how much time, and at what cost.\n",
      "\n",
      "\n",
      "Related Content\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Report\n",
      "Delivering inclusive access\n",
      "\n",
      "Jeffrey Gutman, Adie Tomer, Joseph Kane, Nirav Patel, and Ranjitha Shivaram\n",
      "August 2017\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Report\n",
      "Measuring performance: Accessibility metrics in metropolitan regions around the world\n",
      "\n",
      "Geneviève Boisjoly and Ahmed El-Geneidy\n",
      "August 2017\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Report\n",
      "Is better access key to inclusive cities?\n",
      "\n",
      "Jeffrey Gutman and Nirav Patel\n",
      "Wednesday, October 5, 2016\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Accessibility measures can answer those questions, but not through any one measure. First, the variable social, economic, and political contexts related to access mean searching for a single magical indicator is counterintuitive. For example, a wealthy, automobile-centric region like Dallas, Texas, may have very different measurable goals than a denser, poorer region like Dar es Salaam, Tanzania. Second, academic literature is now rife with such complex measures that it could be difficult to communicate their methodology and results with practitioners. The development of a suite of indicators could offer a menu for policymakers and practitioners to judge accessibility based on local objectives, local conditions, and local capacity.\n",
      "The danger of excessive localization\n",
      "Decentralization and empowering local communities is fast becoming a mantra of governance experts across the world, from development practitioners at institutions like the World Bank to city-focused theorists. And for good reason: delegating policy design and fiscal authority directly to the local level helps ensure policies and practices respond to local needs and desires. Yet as urban areas spillover into contiguous and often numerous municipalities, local independence can introduce certain challenges, especially relating to social and environmental externalities. When it comes to transportation and land development, interests of one municipality are often different from its neighbors. And these divergent development goals can exacerbate accessibility challenges within growing regions, spreading people, housing jobs, and other activities further from one another.\n",
      "Addressing spatial inequities in land use and real estate markets require a broader approach to horizontal governance. While there are examples of metropolitan transport authorities, there is less willingness to consider metropolitan or horizontal governance of land use and fiscal policies. For example, should housing be coordinated across an entire region?\n",
      "Countries with a more centralized top down approach to governance, such as France and Germany, have greater ability to formulate metropolitan governance than more decentralized countries such as the U.S. This is not to say there is a one-size-fits-all approach, but there is an opportunity to test different solutions within different governance contexts, comparing how effective each model is to promote spatial inclusivity.\n",
      "The finance community is missing in action\n",
      "Financing is a central topic in infrastructure circles. As maintenance bills from the automobile era come due, populations continue to grow, and fiscal budgets are tight, how can urban areas afford to build enough infrastructure to support future economic growth? In response, new approaches are evolving in fiscal instruments, such as value capture and private-public partnerships. Missing in these discussions, however, are the implications for inclusive access.\n",
      "We conducted a multi-decade review of past academic literature on access and found that there is no clear substantive discussion of accessibility from a fiscal perspective. While urban transport and land use professionals clearly recognize their interrelationship in achieving inclusive accessibility, at least in theory, the fiscal and finance professionals generally ignore the implications of their instruments with regard to inclusivity. The multilateral development banks and their economic evaluations have ignored the distributive impacts until very recently. And the efforts of some countries to incorporate measures through multi-criteria analysis have had limited impact.\n",
      "This gap must be resolved in any effort toward inclusive urban development. There is little doubt that fiscal approaches must carefully assess who ultimately pays and that alternative finance instruments should be adapted to foster access for all.\n",
      "Going forward\n",
      "Our research confirms that there are enormous opportunities to advance accessibility theory into practice. At this point, what is desperately needed is to launch a range of case studies that deal with these issues and challenges under different geographic, governance, and economic contexts. The good news is that many initiatives are already underway, and more robust communication channels and technology can support such efforts. In Chicago, researchers created an online platform to visually explore accessibility by location. In Bogota, researchers evaluated how affordability is a key principle of access. And in Cairo and Kigali, researchers used open tools to achieve new insights for accessibility. Sharing the results of these case studies could lead to a new level of cross-disciplinary approaches to improve accessibility and lessen the effects of spatial inequity.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title: The Men Who Want to Live Forever\n",
      "URL: https://www.nytimes.com/2018/01/25/opinion/sunday/silicon-valley-immortality.html\n",
      "\n",
      "Would you like to live forever? Some billionaires, already invincible in every other way, have decided that they also deserve not to die. Today several biotech companies, fueled by Silicon Valley fortunes, are devoted to “life extension” — or as some put it, to solving “the problem of death.”\n",
      "It’s a cause championed by the tech billionaire Peter Thiel, the TED Talk darling Aubrey de Gray, Google’s billion-dollar Calico longevity lab and investment by Amazon’s Jeff Bezos. The National Academy of Medicine, an independent group, recently dedicated funding to “end aging forever.”\n",
      "As the longevity entrepreneur Arram Sabeti told The New Yorker: “The proposition that we can live forever is obvious. It doesn’t violate the laws of physics, so we can achieve it.” Of all the slightly creepy aspects to this trend, the strangest is the least noticed: The people publicly championing life extension are mainly men.\n",
      "Not all of them, of course. In 2009, Elizabeth Blackburn received the Nobel Prize for her work on telomeres, protein caps on chromosomes that may be a key to understanding aging. Cynthia Kenyon, the vice president for aging research at Calico, studied life extension long before it was cool; her former protégée, Laura Deming, now runs a venture capital fund for the cause. But these women are focused on curbing age-related pathology, a concept about as controversial as cancer research. They do not appear thirsty for the Fountain of Youth.\n",
      "Professor Blackburn’s new book on telomeres couldn’t be clearer. “Does our research show that by maintaining your telomeres you will live into your hundreds?” it says. “No. Everyone’s cells become old and eventually we die.” Ms. Kenyon once described her research’s goal as “to just have a healthy life and then turn out the lights.” Even Ms. Deming, a 23-year-old prodigy who worked in Ms. Kenyon’s lab at age 12, points out that “aging is innately important to us.”\n",
      "Few of these experts come close to matching the gaudy statements of the longevity investor and “biohacker” Dave Asprey, who has told journalists, “I decided that I was just not going to die.” Or those of Brian Hanley, a microbiologist who has tested an anti-aging gene therapy he developed on himself, who claimed: “There’s a bunch of things that will need to be done to achieve life spans into at least hundreds of years. But we’ll get there.” Or of the 74-year-old fashion mogul Peter Nygard, who during a promotional clip receives injections of his own stem cells to reverse his aging while declaring: “Ponce de León had the right idea. He was just too early. That was then. This is now.”\n",
      "I came across Mr. Nygard’s ode to human endurance three years ago while beginning research on a novel about a woman who can’t die, and watching that video allowed me to experience something close to life extension. As Mr. Nygard compared himself to Leonardo da Vinci and Benjamin Franklin while dancing with a bevy of models — or as a voice-over explained, “living a life most can only dream of” — nine minutes of YouTube expanded into a vapid eternity, where time melted into a vortex of solipsism.\n",
      "At that time I was immersed in caring for my four young children, and this paean to everlasting youth seemed especially stupid. I recall thinking that if this was eternal life, death didn’t seem that bad.\n",
      "But now, as powerful men have begun falling like dominoes under accusations of sexual assault, that video with its young women clustered around an elderly multimillionaire has haunted me anew. As I recall my discomfort with the proclamations of longevity-driven men who hope to achieve “escape velocity,” I think of the astonishing hubris of the Harvey Weinsteins of the world, those who saw young women’s bodies as theirs for the taking.\n",
      "Much has been said about why we allowed such behavior to go unchecked. What has remained unsaid, because it is so obvious, is what would make someone so shameless in the first place: These people believed they were invincible. They saw their own bodies as entirely theirs and other people’s bodies as at their disposal; apparently nothing in their lives led them to believe otherwise.\n",
      "Historically, this is a mistake that few women would make, because until very recently, the physical experience of being a woman entailed exactly the opposite — and not only because women have to hold their keys in self-defense while walking through parking lots at night. It’s only very recently that women have widely participated in public life, but it’s even more recently that men have been welcome, or even expected, to provide physical care for vulnerable people.\n",
      "Only for a nanosecond of human history have men even slightly shared what was once exclusively a woman’s burden: the relentless daily labor of caring for another person’s body, the life-preserving work of cleaning feces and vomit, the constant cycle of cooking and feeding and blanketing and bathing, whether for the young, the ill or the old. For nearly as long as there have been humans, being a female human has meant a daily nonoptional immersion in the fragility of human life and the endless effort required to sustain it.\n",
      "Obviously not everyone who provides care for others is a saint. But engaging in that daily devotion, or even living with its expectation, has enormous potential to change a person. It forces one to constantly imagine the world from someone else’s point of view: Is he hungry? Maybe she’s tired. Is his back hurting him? What is she trying to say?\n",
      "The most obvious cure for today’s gender inequities is to put more women in power. But if we really hope to create an equal society, we will also need more men to care for the powerless — more women in the boardroom, but also more men at the nurses’ station and the changing table, immersed in daily physical empathy. If that sounds like an evolutionary impossibility, well, it doesn’t violate the laws of physics, so we can achieve it. It is surely worth at least as much investment as defeating death.\n",
      "Perhaps it takes the promise of immortality to inspire the self-absorbed to invest in unsexy work like Alzheimer’s research. If so, we may all one day bless the inane death-defiance as a means to a worthy end.\n",
      "But men who hope to live forever might pause on their eternal journey to consider the frightening void at invincibility’s core. Death is the ultimate vulnerability. It is the moment when all of us must confront exactly what so many women have known all too well: You are a body, only a body, and nothing more.\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "class Content:\n",
    "    def __init__(self, url, title, body):\n",
    "        self.url = url\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "\n",
    "\n",
    "def getPage(url):\n",
    "    req = requests.get(url)\n",
    "    return BeautifulSoup(req.text, 'html.parser')\n",
    "\n",
    "\n",
    "def scrapeNYTimes(url):\n",
    "    bs = getPage(url)\n",
    "    title = bs.find('h1').text\n",
    "    lines = bs.select('div.StoryBodyCompanionColumn div p')\n",
    "    body = '\\n'.join([line.text for line in lines])\n",
    "    return Content(url, title, body)\n",
    "\n",
    "def scrapeBrookings(url):\n",
    "    bs = getPage(url)\n",
    "    title = bs.find('h1').text\n",
    "    body = bs.find('div', {'class', 'post-body'}).text\n",
    "    return Content(url, title, body)\n",
    "\n",
    "\n",
    "url = 'https://www.brookings.edu/blog/future-development/2018/01/26/delivering-inclusive-urban-access-3-uncomfortable-truths/'\n",
    "content = scrapeBrookings(url)\n",
    "print('Title: {}'.format(content.title))\n",
    "print('URL: {}\\n'.format(content.url))\n",
    "print(content.body)\n",
    "\n",
    "url = 'https://www.nytimes.com/2018/01/25/opinion/sunday/silicon-valley-immortality.html'\n",
    "content = scrapeNYTimes(url)\n",
    "print('Title: {}'.format(content.title))\n",
    "print('URL: {}\\n'.format(content.url))\n",
    "print(content.body)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Content:\n",
    "    \"\"\"\n",
    "    Common base class for all articles/pages\n",
    "    \"\"\"\n",
    "    def __init__(self, url, title, body):\n",
    "        self.url = url\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "\n",
    "    def print(self):\n",
    "        \"\"\"\n",
    "        Flexible printing function controls output\n",
    "        \"\"\"\n",
    "        print('URL: {}'.format(self.url))\n",
    "        print('TITLE: {}'.format(self.title))\n",
    "        print('BODY:\\n{}'.format(self.body))\n",
    "\n",
    "class Website:\n",
    "    \"\"\" \n",
    "    Contains information about website structure\n",
    "    \"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "\n",
    "class Crawler:\n",
    "\n",
    "    def getPage(self, url):\n",
    "        try:\n",
    "            req = requests.get(url)\n",
    "        except requests.exceptions.RequestException:\n",
    "            return None\n",
    "        return BeautifulSoup(req.text, 'html.parser')\n",
    "\n",
    "    def safeGet(self, pageObj, selector):\n",
    "        \"\"\"\n",
    "        Utilty function used to get a content string from a Beautiful Soup\n",
    "        object and a selector. Returns an empty string if no object\n",
    "        is found for the given selector\n",
    "        \"\"\"\n",
    "        selectedElems = pageObj.select(selector)\n",
    "        if selectedElems is not None and len(selectedElems) > 0:\n",
    "            return '\\n'.join([elem.get_text() for elem in selectedElems])\n",
    "        return ''\n",
    "\n",
    "    def parse(self, site, url):\n",
    "        \"\"\"\n",
    "        Extract content from a given page URL\n",
    "        \"\"\"\n",
    "        bs = self.getPage(url)\n",
    "        if bs is not None:\n",
    "            title = self.safeGet(bs, site.titleTag)\n",
    "            body = self.safeGet(bs, site.bodyTag)\n",
    "            if title != '' and body != '':\n",
    "                content = Content(url, title, body)\n",
    "                content.print()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title is:\n",
      "Learning Python, 5th Edition \n",
      "Body is:\n",
      "\n",
      "Get a comprehensive, in-depth introduction to the core Python language with this hands-on book. Based on author Mark Lutz’s popular training course, this updated fifth edition will help you quickly write efficient, high-quality code with Python. It’s an ideal way to begin, whether you’re new to programming or a professional developer versed in other languages. \n",
      "\n",
      "Complete with quizzes, exercises, and helpful illustrations,  this easy-to-follow, self-paced tutorial gets you started with both Python 2.7 and 3.3— the latest releases in the 3.X  and 2.X lines—plus all other releases in common use today. You’ll also learn some advanced language features that recently have become more common in Python code.\n",
      "\n",
      "Explore Python’s major built-in object types such as numbers, lists, and dictionaries \n",
      "Create and process objects with Python statements, and learn Python’s general syntax model\n",
      "Use functions to avoid code redundancy and package code for reuse\n",
      "Organize statements, functions, and other tools into larger components with modules \n",
      "Dive into classes: Python’s object-oriented programming tool for structuring code\n",
      "Write large programs with Python’s exception-handling model and development tools\n",
      "Learn advanced Python tools, including decorators, descriptors, metaclasses, and Unicode processing\n",
      "\n",
      "\n",
      "URL: http://shop.oreilly.com/product/0636920028154.do\n",
      "TITLE: Learning Python, 5th Edition \n",
      "BODY:\n",
      "\n",
      "Get a comprehensive, in-depth introduction to the core Python language with this hands-on book. Based on author Mark Lutz’s popular training course, this updated fifth edition will help you quickly write efficient, high-quality code with Python. It’s an ideal way to begin, whether you’re new to programming or a professional developer versed in other languages. \n",
      "\n",
      "Complete with quizzes, exercises, and helpful illustrations,  this easy-to-follow, self-paced tutorial gets you started with both Python 2.7 and 3.3— the latest releases in the 3.X  and 2.X lines—plus all other releases in common use today. You’ll also learn some advanced language features that recently have become more common in Python code.\n",
      "\n",
      "Explore Python’s major built-in object types such as numbers, lists, and dictionaries \n",
      "Create and process objects with Python statements, and learn Python’s general syntax model\n",
      "Use functions to avoid code redundancy and package code for reuse\n",
      "Organize statements, functions, and other tools into larger components with modules \n",
      "Dive into classes: Python’s object-oriented programming tool for structuring code\n",
      "Write large programs with Python’s exception-handling model and development tools\n",
      "Learn advanced Python tools, including decorators, descriptors, metaclasses, and Unicode processing\n",
      "\n",
      "\n",
      "title is:\n",
      "EPA chief wants scientists to debate climate on TV\n",
      "Body is:\n",
      "WASHINGTON (Reuters) - The U.S. Environmental Protection Agency is in the early stages of launching a debate about climate change that could air on television – challenging scientists to prove the widespread view that global warming is a serious threat, the head of the agency said. The move comes as the administration of President Donald Trump seeks to roll back a slew of Obama-era regulations limiting carbon dioxide emissions from fossil fuels, and begins a withdrawal from the Paris Climate Agreement - a global pact to stem planetary warming through emissions cuts. “There are lots of questions that have not been asked and answered (about climate change),” EPA Administrator Scott Pruitt told Reuters in an interview late on Monday.  “Who better to do that than a group of scientists... getting together and having a robust discussion for all the world to see,” he added without explaining how the scientists would be chosen. Asked if he thought the debate should be televised, Pruitt said: “I think so. I think so. I mean, I don’t know yet, but you want this to be open to the world. You want this to be on full display. I think the American people would be very interested in consuming that. I think they deserve it.” Pruitt, one of the most controversial figures in the Trump administration, has repeatedly expressed doubts about climate change – one of the main points of contention in his narrow confirmation by the Senate.  While acknowledging the planet is warming, Pruitt says he questions the gravity of the problem and the need for regulations that require companies to take costly measures to reduce their carbon footprint. “It is a question about how much we contribute to it. How do we measure that with precision? And by the way, are we on an unsustainable path? And is it causing an existential threat?” he said in the interview. Since taking up his role at EPA, he has emerged as one of the more prolific Trump cabinet appointees, taking steps to undo more than two dozen regulations, and influencing Trump’s decision to pull the United States from the Paris climate change deal, agreed by nearly 200 countries in 2015. Pruitt rejected global criticism of the United States for pulling out of the climate deal, which Trump has said would have cost America trillions of dollars without benefit. Environmental Protection Agency Administrator Scott Pruitt waves after an interview for Reuters at his office in Washington, U.S., July 10, 2017. REUTERS/Yuri Gripas“We have nothing to be apologetic about,” Pruitt said. “It was absolutely a decision of courage and fortitude and truly represented an America First strategy with respect to how we are leading on this issue.” Pruitt said the United States had already cut its carbon output to the lowest levels in nearly 25 years without mandates, thanks mainly to increased use of natural gas - which burns cleaner than coal. “RED TEAM, BLUE TEAM” TACTICS Slideshow (4 Images)Pruitt said his desire for the agency to host an ongoing climate change debate was inspired by two articles published in April – one in the Wall Street Journal by theoretical physicist Steve Koonin, who served as undersecretary of energy under Obama – and one by conservative columnist Brett Stephens in the New York Times.  Koonin’s article made the case that climate science should use the “red team-blue team” methodology used by the national security community to test assumptions. Stephens’ article criticized claims of complete certainty in climate science, saying that it “traduces the spirit of science.” Pruitt said scientists should not scoff at the idea of participating in these debates. Related CoverageTranscript of Reuters interview with EPA Administrator Scott Pruitt“If you’re going to win and if you’re so certain about it, come and do your deal. They shouldn’t be scared of the debate and discussion,” he said. Pruitt said debate is not necessarily aimed at undermining the 2009 “endangerment finding,” the scientific determination that carbon dioxide harms human health that formed the basis for the Democratic Obama administration’s regulation of greenhouse gases. He said there may be a legal basis to challenge the finding but would prefer Congress weigh in on the matter. In the interview, Pruitt added that he intended to deal “very aggressively” with automakers that use devices to cheat emissions tests, and would also seek to boost accountability for companies to clean up polluted sites under the Superfund program.  He said EPA was also not ready to decide yet on a change proposed by Trump’s special adviser Carl Icahn to the U.S. biofuels program, that would shift the burden of blending biofuels like ethanol into gasoline away from refiners to companies further down the supply chain. Editing by Richard Valdmanis and Marguerita ChoyOur Standards:The Thomson Reuters Trust Principles.\n",
      "URL: http://www.reuters.com/article/us-usa-epa-pruitt-idUSKBN19W2D0\n",
      "TITLE: EPA chief wants scientists to debate climate on TV\n",
      "BODY:\n",
      "WASHINGTON (Reuters) - The U.S. Environmental Protection Agency is in the early stages of launching a debate about climate change that could air on television – challenging scientists to prove the widespread view that global warming is a serious threat, the head of the agency said. The move comes as the administration of President Donald Trump seeks to roll back a slew of Obama-era regulations limiting carbon dioxide emissions from fossil fuels, and begins a withdrawal from the Paris Climate Agreement - a global pact to stem planetary warming through emissions cuts. “There are lots of questions that have not been asked and answered (about climate change),” EPA Administrator Scott Pruitt told Reuters in an interview late on Monday.  “Who better to do that than a group of scientists... getting together and having a robust discussion for all the world to see,” he added without explaining how the scientists would be chosen. Asked if he thought the debate should be televised, Pruitt said: “I think so. I think so. I mean, I don’t know yet, but you want this to be open to the world. You want this to be on full display. I think the American people would be very interested in consuming that. I think they deserve it.” Pruitt, one of the most controversial figures in the Trump administration, has repeatedly expressed doubts about climate change – one of the main points of contention in his narrow confirmation by the Senate.  While acknowledging the planet is warming, Pruitt says he questions the gravity of the problem and the need for regulations that require companies to take costly measures to reduce their carbon footprint. “It is a question about how much we contribute to it. How do we measure that with precision? And by the way, are we on an unsustainable path? And is it causing an existential threat?” he said in the interview. Since taking up his role at EPA, he has emerged as one of the more prolific Trump cabinet appointees, taking steps to undo more than two dozen regulations, and influencing Trump’s decision to pull the United States from the Paris climate change deal, agreed by nearly 200 countries in 2015. Pruitt rejected global criticism of the United States for pulling out of the climate deal, which Trump has said would have cost America trillions of dollars without benefit. Environmental Protection Agency Administrator Scott Pruitt waves after an interview for Reuters at his office in Washington, U.S., July 10, 2017. REUTERS/Yuri Gripas“We have nothing to be apologetic about,” Pruitt said. “It was absolutely a decision of courage and fortitude and truly represented an America First strategy with respect to how we are leading on this issue.” Pruitt said the United States had already cut its carbon output to the lowest levels in nearly 25 years without mandates, thanks mainly to increased use of natural gas - which burns cleaner than coal. “RED TEAM, BLUE TEAM” TACTICS Slideshow (4 Images)Pruitt said his desire for the agency to host an ongoing climate change debate was inspired by two articles published in April – one in the Wall Street Journal by theoretical physicist Steve Koonin, who served as undersecretary of energy under Obama – and one by conservative columnist Brett Stephens in the New York Times.  Koonin’s article made the case that climate science should use the “red team-blue team” methodology used by the national security community to test assumptions. Stephens’ article criticized claims of complete certainty in climate science, saying that it “traduces the spirit of science.” Pruitt said scientists should not scoff at the idea of participating in these debates. Related CoverageTranscript of Reuters interview with EPA Administrator Scott Pruitt“If you’re going to win and if you’re so certain about it, come and do your deal. They shouldn’t be scared of the debate and discussion,” he said. Pruitt said debate is not necessarily aimed at undermining the 2009 “endangerment finding,” the scientific determination that carbon dioxide harms human health that formed the basis for the Democratic Obama administration’s regulation of greenhouse gases. He said there may be a legal basis to challenge the finding but would prefer Congress weigh in on the matter. In the interview, Pruitt added that he intended to deal “very aggressively” with automakers that use devices to cheat emissions tests, and would also seek to boost accountability for companies to clean up polluted sites under the Superfund program.  He said EPA was also not ready to decide yet on a change proposed by Trump’s special adviser Carl Icahn to the U.S. biofuels program, that would shift the burden of blending biofuels like ethanol into gasoline away from refiners to companies further down the supply chain. Editing by Richard Valdmanis and Marguerita ChoyOur Standards:The Thomson Reuters Trust Principles.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title is:\n",
      "Idea to Retire: Old methods of policy education\n",
      "Idea to Retire: Old methods of policy education\n",
      "Body is:\n",
      "\n",
      "Public policy and public affairs schools aim to train competent creators and implementers of government policy. While drawing on the principles that gird our economic and political systems to provide a well-rounded education, like law schools and business schools, policy schools provide professional training. They are quite distinct from graduate programs in political science or economics which aim to train the next generation of academics. As professional training programs, they add value by imparting both the skills which are relevant to current employers, and skills which we know will be relevant as organizations and societies evolve. \n",
      "The relevance of the skills that policy programs impart to address problems of today and tomorrow bears further discussion. We are living through an era in which societies are increasingly interconnected. The wide-scale adoption of devices such as the smartphone is having a profound impact on our culture, communities, and economy. The use of social and digital media and associated means of communication enabled by mobile devices is changing the tone, content, and geographic scope of our conversations, modifying how information is generated and consumed, and changing the very nature of citizen engagement. \n",
      "Information technology-based platforms provisioned by private providers such as Facebook, Google, Uber, and Lyft maintain information about millions of citizens and enable services such as transportation that were mediated in the past solely by the public sector. Surveillance for purposes of public safety via large-scale deployment of sensors also raises fundamental questions about information privacy. From technology-enabled global delivery of work to displacement and replacement of categories of work, some studies estimate that up to 47 percent of U.S. employment might be at risk of computerization with an attendant rise in income inequality. These technology-induced changes will affect every policy domain. How should policy programs best prepare students to address societal challenges in this world that is being transformed by technology? We believe the answer lies in educating students to be “men and women of intelligent action.” \n",
      "A model of policy education\n",
      "We begin with a skills-based model of policy education. These four essential skills address the general problems policy practitioners frequently face:\n",
      "\n",
      "Design skills to craft policy ideas \n",
      "Analytical skills to make smart ex ante decisions \n",
      "Interpersonal experience to manage policy implementation  \n",
      "Evaluative skills to assess outcomes ex post and correct course if necessary\n",
      "\n",
      "These skills make up the policy analysis toolkit required to be data driven practitioner of “intelligent action” in any policy domain. This toolkit needs to be supplemented by an understanding of how technology is transforming societal challenges, enabling new solutions, or disrupting existing regulatory regimes. This understanding is essential to policy formulation and implementation. \n",
      "Pillar 1: Design skills\n",
      "As with engineering, where design precedes analysis, this first pillar seeks to educate students in thinking creatively about problems in order to devise and develop policy ideas. Using ideas derived from design, divergent and convergent thinking principles are employed to generate, explore, and arrive at a candidate set of solutions. Using Uber as an example, an approach to identify and explore the key policy issues such as convenience, costs, driver working hours, and insurance would involve interviewing and observing both incumbent taxi drivers and Uber drivers. This in turn would lead to a set of alternatives that deserve further and careful consideration.  Using these skills, candidate designs and choices that are generated can be evaluated using the policy analytic toolkit. \n",
      "Pillar 2: Analytical skills\n",
      "\n",
      "\n",
      "Related\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "The Blockchain: What It Is and Why It Matters\n",
      "\n",
      "Mohit Kaushal and Sheel Tyle\n",
      "Tuesday, January 13, 2015\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "How technology is changing manufacturing\n",
      "\n",
      "Darrell M. West\n",
      "Thursday, June 2, 2016\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "Rural and urban America divided by broadband access\n",
      "\n",
      "Darrell M. West and Jack Karsten\n",
      "Monday, July 18, 2016\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "At Carnegie Mellon, we are often cited in media and interrogated by peers on our approach to analytical and technology skills education. Curiosity about which skills are the “right” skills to teach policy practitioners are common, but we believe this is the wrong approach. We instead begin from the premise that policy or management decisions should be grounded in evidence.  We then determine the skills required to assemble the types of evidence that will likely be available to policy makers in the future.  In increasingly instrumented environments where citizens and infrastructure produce continuous streams of data, making sense of it all will require a somewhat different set of skills. We believe that a grounding in micro-economics, operations research, statistics, and program evaluation (aka causal inference) to be an essential core to policy programs. \n",
      "New coursework will teach students to work with multi-variable data and machine learning with an emphasis on prediction. This material ought to be part of the required coursework in statistics given the importance of prediction in many policy implementation settings. Along the same lines, the ability to work with unstructured data (especially text) and data visualization will become increasingly relevant to all students, not just those students who want to specialize in data analytics. Finally, knowledge of data manipulation and analysis languages such as Python and R for analytic work will be important because data often has to be massaged and cleansed prior to analysis. An important task for programs will be to determine the competencies expected of graduates. \n",
      "Pillar 3: Interpersonal experiences\n",
      "The third pillar of the skills-based model is interpersonal experience, where the practiced habits of good communication and steady negotiation developed with a sound understanding of organizations, their design and their behaviors. We label these purposely as experiences rather than skills because we believe they are best practiced either in the real-world or in simulated real-world settings. It is also in this pillar where practitioners learn the knowledge necessary to become credible experts in their domain. We believe that in addition to core coursework in the area, a supplementary curriculum which provides students with opportunities to gain these experiences is an essential component of our educational model.\n",
      "Pillar 4: Evaluative skills\n",
      "\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Constitution 3.0\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tEdited by Jeffrey Rosen and Benjamin Wittes \n",
      "2013\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The Need for Speed\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Robert E. Litan and Hal J. Singer \n",
      "2013\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "After the Breakup\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Robert W. Crandall \n",
      "2010\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The ability to carefully diagnose the effectiveness of policy or management interventions is the fourth pillar of our model. It is insufficient to create and execute policy without measurement, and this is where both careful thought to the fundamental issues of measurement and evaluation become important. The ability to make objective judgments on the benefits, liabilities, and unintended consequences of prior policies is the goal of this set of skills. Here, sound statistical and econometric training with an understanding of the principles of causal inference is essential. In addition, program evaluation skills such as cost-benefit and financial analysis help practitioners round out their evaluation skills by considering both non-monetary and economic impacts.\n",
      "What should be retired?\n",
      "A skills-based approach might replace certain aspects of existing policy training.  This depends on a number of factors specific to each institution, but three generally applicable observations are clear. First, real-world experiences are a powerful way to encode domain learning as well as project management skills. Through project-based work, students can learn about institutional contexts in specific policy domains and political processes such as budgeting. Second, team-based projects allow students to learn and apply principles of management and organizational behavior. At Carnegie Mellon, we refer to these as “systems synthesis” projects, since they require students to adopt a systemic point of view and to synthesize a number of skills in their policy analysis toolkit. Third, interpersonal skills training can be practiced through activities such as weekend negotiation exercises, hackathons, and speaker series. These activities can be highly intentional and fashioned to reinforce skills rather than as a recess from the “real work” of classroom training. Since students complete graduate programs in such a short time, counseling them to focus on outcomes from day one will allow them to choose a reinforcing set of coursework and real-world experiences. \n",
      "In summary, we argue for a model of policy education that views practitioners as future problem solvers. Good policy education must consider the ways in which problems will present themselves, and the ways in which answers will obscure themselves. Rigorous training grounded in the analysis of available evidence and buoyed by real-world interpersonal experiences is a sound approach to relevant, durable policy training.\n",
      " \n",
      "Authors\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "R\n",
      "\n",
      "\n",
      "\n",
      "Ramayya Krishnan\n",
      "Ramayya Krishnan is the dean of H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University where he is the W.W. Cooper and Ruth F. Cooper Professor of Management Science and Information Systems.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "J\n",
      "\n",
      "\n",
      "\n",
      "Jon Nehlsen\n",
      "Jon Nehlsen is senior director of external relations at H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Read other essays in the Ideas to Retire blog series here.\n",
      "\n",
      "URL: https://www.brookings.edu/blog/techtank/2016/03/01/idea-to-retire-old-methods-of-policy-education/\n",
      "TITLE: Idea to Retire: Old methods of policy education\n",
      "Idea to Retire: Old methods of policy education\n",
      "BODY:\n",
      "\n",
      "Public policy and public affairs schools aim to train competent creators and implementers of government policy. While drawing on the principles that gird our economic and political systems to provide a well-rounded education, like law schools and business schools, policy schools provide professional training. They are quite distinct from graduate programs in political science or economics which aim to train the next generation of academics. As professional training programs, they add value by imparting both the skills which are relevant to current employers, and skills which we know will be relevant as organizations and societies evolve. \n",
      "The relevance of the skills that policy programs impart to address problems of today and tomorrow bears further discussion. We are living through an era in which societies are increasingly interconnected. The wide-scale adoption of devices such as the smartphone is having a profound impact on our culture, communities, and economy. The use of social and digital media and associated means of communication enabled by mobile devices is changing the tone, content, and geographic scope of our conversations, modifying how information is generated and consumed, and changing the very nature of citizen engagement. \n",
      "Information technology-based platforms provisioned by private providers such as Facebook, Google, Uber, and Lyft maintain information about millions of citizens and enable services such as transportation that were mediated in the past solely by the public sector. Surveillance for purposes of public safety via large-scale deployment of sensors also raises fundamental questions about information privacy. From technology-enabled global delivery of work to displacement and replacement of categories of work, some studies estimate that up to 47 percent of U.S. employment might be at risk of computerization with an attendant rise in income inequality. These technology-induced changes will affect every policy domain. How should policy programs best prepare students to address societal challenges in this world that is being transformed by technology? We believe the answer lies in educating students to be “men and women of intelligent action.” \n",
      "A model of policy education\n",
      "We begin with a skills-based model of policy education. These four essential skills address the general problems policy practitioners frequently face:\n",
      "\n",
      "Design skills to craft policy ideas \n",
      "Analytical skills to make smart ex ante decisions \n",
      "Interpersonal experience to manage policy implementation  \n",
      "Evaluative skills to assess outcomes ex post and correct course if necessary\n",
      "\n",
      "These skills make up the policy analysis toolkit required to be data driven practitioner of “intelligent action” in any policy domain. This toolkit needs to be supplemented by an understanding of how technology is transforming societal challenges, enabling new solutions, or disrupting existing regulatory regimes. This understanding is essential to policy formulation and implementation. \n",
      "Pillar 1: Design skills\n",
      "As with engineering, where design precedes analysis, this first pillar seeks to educate students in thinking creatively about problems in order to devise and develop policy ideas. Using ideas derived from design, divergent and convergent thinking principles are employed to generate, explore, and arrive at a candidate set of solutions. Using Uber as an example, an approach to identify and explore the key policy issues such as convenience, costs, driver working hours, and insurance would involve interviewing and observing both incumbent taxi drivers and Uber drivers. This in turn would lead to a set of alternatives that deserve further and careful consideration.  Using these skills, candidate designs and choices that are generated can be evaluated using the policy analytic toolkit. \n",
      "Pillar 2: Analytical skills\n",
      "\n",
      "\n",
      "Related\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "The Blockchain: What It Is and Why It Matters\n",
      "\n",
      "Mohit Kaushal and Sheel Tyle\n",
      "Tuesday, January 13, 2015\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "How technology is changing manufacturing\n",
      "\n",
      "Darrell M. West\n",
      "Thursday, June 2, 2016\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "Rural and urban America divided by broadband access\n",
      "\n",
      "Darrell M. West and Jack Karsten\n",
      "Monday, July 18, 2016\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "At Carnegie Mellon, we are often cited in media and interrogated by peers on our approach to analytical and technology skills education. Curiosity about which skills are the “right” skills to teach policy practitioners are common, but we believe this is the wrong approach. We instead begin from the premise that policy or management decisions should be grounded in evidence.  We then determine the skills required to assemble the types of evidence that will likely be available to policy makers in the future.  In increasingly instrumented environments where citizens and infrastructure produce continuous streams of data, making sense of it all will require a somewhat different set of skills. We believe that a grounding in micro-economics, operations research, statistics, and program evaluation (aka causal inference) to be an essential core to policy programs. \n",
      "New coursework will teach students to work with multi-variable data and machine learning with an emphasis on prediction. This material ought to be part of the required coursework in statistics given the importance of prediction in many policy implementation settings. Along the same lines, the ability to work with unstructured data (especially text) and data visualization will become increasingly relevant to all students, not just those students who want to specialize in data analytics. Finally, knowledge of data manipulation and analysis languages such as Python and R for analytic work will be important because data often has to be massaged and cleansed prior to analysis. An important task for programs will be to determine the competencies expected of graduates. \n",
      "Pillar 3: Interpersonal experiences\n",
      "The third pillar of the skills-based model is interpersonal experience, where the practiced habits of good communication and steady negotiation developed with a sound understanding of organizations, their design and their behaviors. We label these purposely as experiences rather than skills because we believe they are best practiced either in the real-world or in simulated real-world settings. It is also in this pillar where practitioners learn the knowledge necessary to become credible experts in their domain. We believe that in addition to core coursework in the area, a supplementary curriculum which provides students with opportunities to gain these experiences is an essential component of our educational model.\n",
      "Pillar 4: Evaluative skills\n",
      "\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Constitution 3.0\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tEdited by Jeffrey Rosen and Benjamin Wittes \n",
      "2013\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The Need for Speed\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Robert E. Litan and Hal J. Singer \n",
      "2013\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "After the Breakup\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Robert W. Crandall \n",
      "2010\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The ability to carefully diagnose the effectiveness of policy or management interventions is the fourth pillar of our model. It is insufficient to create and execute policy without measurement, and this is where both careful thought to the fundamental issues of measurement and evaluation become important. The ability to make objective judgments on the benefits, liabilities, and unintended consequences of prior policies is the goal of this set of skills. Here, sound statistical and econometric training with an understanding of the principles of causal inference is essential. In addition, program evaluation skills such as cost-benefit and financial analysis help practitioners round out their evaluation skills by considering both non-monetary and economic impacts.\n",
      "What should be retired?\n",
      "A skills-based approach might replace certain aspects of existing policy training.  This depends on a number of factors specific to each institution, but three generally applicable observations are clear. First, real-world experiences are a powerful way to encode domain learning as well as project management skills. Through project-based work, students can learn about institutional contexts in specific policy domains and political processes such as budgeting. Second, team-based projects allow students to learn and apply principles of management and organizational behavior. At Carnegie Mellon, we refer to these as “systems synthesis” projects, since they require students to adopt a systemic point of view and to synthesize a number of skills in their policy analysis toolkit. Third, interpersonal skills training can be practiced through activities such as weekend negotiation exercises, hackathons, and speaker series. These activities can be highly intentional and fashioned to reinforce skills rather than as a recess from the “real work” of classroom training. Since students complete graduate programs in such a short time, counseling them to focus on outcomes from day one will allow them to choose a reinforcing set of coursework and real-world experiences. \n",
      "In summary, we argue for a model of policy education that views practitioners as future problem solvers. Good policy education must consider the ways in which problems will present themselves, and the ways in which answers will obscure themselves. Rigorous training grounded in the analysis of available evidence and buoyed by real-world interpersonal experiences is a sound approach to relevant, durable policy training.\n",
      " \n",
      "Authors\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "R\n",
      "\n",
      "\n",
      "\n",
      "Ramayya Krishnan\n",
      "Ramayya Krishnan is the dean of H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University where he is the W.W. Cooper and Ruth F. Cooper Professor of Management Science and Information Systems.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "J\n",
      "\n",
      "\n",
      "\n",
      "Jon Nehlsen\n",
      "Jon Nehlsen is senior director of external relations at H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Read other essays in the Ideas to Retire blog series here.\n",
      "\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "title is:\n",
      "Oil Boom Gives the U.S. a New Edge in Energy and Diplomacy\n",
      "Body is:\n",
      "HOUSTON — A substantial rise in oil prices in recent months has led to a resurgence in American oil production, enabling the country to challenge the dominance of Saudi Arabia and dampen price pressures at the pump.\n",
      "The success has come in the face of efforts by Saudi Arabia and its oil allies to undercut the shale drilling spree in the United States. Those strategies backfired and ultimately ended up benefiting the oil industry.\n",
      "Overcoming three years of slumping prices proved the resiliency of the shale boom. Energy companies and their financial backers were able to weather market turmoil — and the maneuvers of the global oil cartel — by adjusting exploration and extraction techniques.\n",
      "After a painful shakeout in the industry that included scores of bankruptcies and a significant loss of jobs, a steadier shale-drilling industry is arising, anchored by better-financed companies.\n",
      "With the price of West Texas intermediate crude above $65 a barrel, a level not seen in almost three years, the United States is becoming a dominant producer. It is able to outflank competitors in supplying growing global markets, particularly China and India, while slashing imports from the Middle East and North Africa.\n",
      "This year, the United States is expected to surpass Saudi Arabia and to rival Russia as the world’s leader, with record output of over 10 million barrels a day, according to the International Energy Agency.\n",
      "“This is a 180-degree turn for the United States and the impacts are being felt around the world,” said Daniel Yergin, the economic historian and author of “The Prize: The Epic Quest for Oil, Money and Power.” “This not only contributes to U.S. energy security but also contributes to world energy security by bringing new supplies to the world.”\n",
      "At the same time, the United States is becoming a major exporter of natural gas, another outgrowth of the shale revolution, undercutting Russian energy dominance over Eastern Europe.\n",
      "The improving energy picture comes as the Trump administration is attempting to increase offshore drilling and loosen other regulations on fossil fuel development. But just as the surge in oil and gas production in shale fields during the Barack Obama administration had little to do with Washington, the current rise is the result of private companies responding to global markets.\n",
      "Shale fields can be developed relatively quickly and at modest costs relative to the giant projects, whether on land or offshore, that were once favored by big oil companies. That makes it easier to turn investment spigots on or off to adjust to market fluctuations. Companies like Exxon Mobil and Chevron are putting increasing amounts of capital in shale fields, particularly in West Texas and New Mexico.\n",
      "The results go far beyond the economic, offering Washington strategic weapons once unthinkable. The United States and its allies now have a supply cushion at a time when political turmoil in Venezuela, Libya and Nigeria is threatening to interrupt flows to markets.\n",
      "Only a few years ago, such threats — along with a recent pipeline failure in the North Sea and storms in the Gulf of Mexico — would have sent the price of crude soaring. Instead, the rise has been muted, and gasoline at the pump remains below $2.60 a gallon across most of the United States.\n",
      "The new energy power also relieves pressure on Washington to act militarily if tensions between Iran and Saudi Arabia break out into war. And it gives Washington the leeway to apply sanctions on other producers — as it has in Russia, and may in Iran or Venezuela — with far less risk to the global economy.\n",
      "It is a striking contrast to the 1970s, when Arab oil boycotts forced motorists to line up for blocks to fill their tanks and the economy went into a tailspin. Even more recently, during the presidency of George W. Bush, domestic oil output was declining so rapidly that the country set a course to replace oil with biofuels like ethanol.\n",
      "Many environmentalists argue that by increasing oil and gas supplies and lowering prices for consumers, shale drilling is extending the life of fossil fuels to the detriment of the environment and the development of cleaner energy.\n",
      "The shale drilling revolution has remade the global energy market, with imports from members of the Organization of the Petroleum Exporting Countries plunging by 20 percent from late 2016 to late 2017. At the same time, exports rose by hundreds of thousands of barrels a day.\n",
      "Nothing like the current situation was foreseen in late 2014, when rising domestic production began weighing on global oil prices.\n",
      "In response, Saudi Arabia led OPEC in a new direction. Instead of throttling back to support prices as the cartel had done so often, it left the market alone and even increased production for a time.\n",
      "Prices fell below $40 a barrel, as the Saudis and their allies hoped to drive American operations out of business by making shale drilling uneconomical. American exploration quickly dropped, but the price squeeze made companies more innovative in the use of drilling technologies, robotics and sensors to maximize output and reduce costs.\n",
      "While scores of smaller companies went out of business, the survivors lengthened horizontal wells to yield more oil, and used clever hedging and drilling strategies to maximize profits even when prices slumped.\n",
      "The response surprised the global oil community. OPEC, Russia and allied producing countries changed course and began cutting back again in 2016.\n",
      "“OPEC missed the point,” said René Ortiz, a former OPEC secretary general and former Ecuadorean energy minister. “They thought they could recover the U.S. market by bringing the prices down. Now the U.S. has gained the leading position in the world oil market regardless of what OPEC does.”\n",
      "“This displacement of Saudi oil, Nigerian oil, Libyan oil and Venezuelan oil,” Mr. Ortiz concluded, “was never anticipated.”\n",
      "A week ago, OPEC leaders met in Oman to discuss a probable extension of production cuts into 2019 to support prices. Their biggest obstacle is the United States.\n",
      "Technological advances unlocking oil from tight rocks like shale has led to a drilling frenzy enabling a doubling of output in a decade, transforming unlikely places like North Dakota and New Mexico into world class petroleum hubs. Pipelines are being built across Texas to serve ports where oil can be pumped onto tankers headed for China, India and other markets.\n",
      "Domestic production last year averaged 9.3 million barrels a day, and the Energy Department projects that the figure will climb to 10.3 million barrels a day this year, surpassing the record set in 1970. In the meantime, since a 40-year export ban was lifted in 2015, exports of American oil have risen to roughly two million barrels a day — more than many OPEC members.\n",
      "The department projects an additional increase in domestic production of 500,000 barrels a day in 2019.\n",
      "Concerns over climate change as well as the growing popularity of electric cars and the eventual aging of the best shale fields will probably curb production and demand over the next few decades. But in the short term, the boom has changed the landscape.\n",
      "The Energy Department projects that the recent surge will hold the price of Brent crude, the global benchmark, to $60 a barrel in 2018 and $61 a barrel in 2019 — a modest increase from $54 last year. (The Brent price rose above $70 a barrel this month, but few analysts see a return to $100-a-barrel oil.)\n",
      "The emerging order in the energy realm is a stable balance of power. Saudi Arabia, which essentially runs OPEC, has put a floor under the oil price — probably around $50 a barrel — with its limits on output and exports over the last four years. But now the United States, by the sheer force of its production, the supremacy of its technology, and an unmatched pipeline, refinery and storage structure, has put a ceiling to the price.\n",
      "Experts note that when oil climbs to $60 a barrel and higher, as it has lately, a drilling rush commences — the national rig count has climbed by over a third in the last year — promising to refill domestic and even global energy inventories. Only a major war or other disruption is likely to send prices soaring.\n",
      "“We have all suffered these depressed prices over the last two years and we are excited to see the new prices and we will respond accordingly,” said Harald Jordan, vice president for engineering at Peak Energy, a Colorado-based producer. “You will see rig activity continue to increase.”\n",
      "URL: https://www.nytimes.com/2018/01/28/business/energy-environment/oil-boom.html\n",
      "TITLE: Oil Boom Gives the U.S. a New Edge in Energy and Diplomacy\n",
      "BODY:\n",
      "HOUSTON — A substantial rise in oil prices in recent months has led to a resurgence in American oil production, enabling the country to challenge the dominance of Saudi Arabia and dampen price pressures at the pump.\n",
      "The success has come in the face of efforts by Saudi Arabia and its oil allies to undercut the shale drilling spree in the United States. Those strategies backfired and ultimately ended up benefiting the oil industry.\n",
      "Overcoming three years of slumping prices proved the resiliency of the shale boom. Energy companies and their financial backers were able to weather market turmoil — and the maneuvers of the global oil cartel — by adjusting exploration and extraction techniques.\n",
      "After a painful shakeout in the industry that included scores of bankruptcies and a significant loss of jobs, a steadier shale-drilling industry is arising, anchored by better-financed companies.\n",
      "With the price of West Texas intermediate crude above $65 a barrel, a level not seen in almost three years, the United States is becoming a dominant producer. It is able to outflank competitors in supplying growing global markets, particularly China and India, while slashing imports from the Middle East and North Africa.\n",
      "This year, the United States is expected to surpass Saudi Arabia and to rival Russia as the world’s leader, with record output of over 10 million barrels a day, according to the International Energy Agency.\n",
      "“This is a 180-degree turn for the United States and the impacts are being felt around the world,” said Daniel Yergin, the economic historian and author of “The Prize: The Epic Quest for Oil, Money and Power.” “This not only contributes to U.S. energy security but also contributes to world energy security by bringing new supplies to the world.”\n",
      "At the same time, the United States is becoming a major exporter of natural gas, another outgrowth of the shale revolution, undercutting Russian energy dominance over Eastern Europe.\n",
      "The improving energy picture comes as the Trump administration is attempting to increase offshore drilling and loosen other regulations on fossil fuel development. But just as the surge in oil and gas production in shale fields during the Barack Obama administration had little to do with Washington, the current rise is the result of private companies responding to global markets.\n",
      "Shale fields can be developed relatively quickly and at modest costs relative to the giant projects, whether on land or offshore, that were once favored by big oil companies. That makes it easier to turn investment spigots on or off to adjust to market fluctuations. Companies like Exxon Mobil and Chevron are putting increasing amounts of capital in shale fields, particularly in West Texas and New Mexico.\n",
      "The results go far beyond the economic, offering Washington strategic weapons once unthinkable. The United States and its allies now have a supply cushion at a time when political turmoil in Venezuela, Libya and Nigeria is threatening to interrupt flows to markets.\n",
      "Only a few years ago, such threats — along with a recent pipeline failure in the North Sea and storms in the Gulf of Mexico — would have sent the price of crude soaring. Instead, the rise has been muted, and gasoline at the pump remains below $2.60 a gallon across most of the United States.\n",
      "The new energy power also relieves pressure on Washington to act militarily if tensions between Iran and Saudi Arabia break out into war. And it gives Washington the leeway to apply sanctions on other producers — as it has in Russia, and may in Iran or Venezuela — with far less risk to the global economy.\n",
      "It is a striking contrast to the 1970s, when Arab oil boycotts forced motorists to line up for blocks to fill their tanks and the economy went into a tailspin. Even more recently, during the presidency of George W. Bush, domestic oil output was declining so rapidly that the country set a course to replace oil with biofuels like ethanol.\n",
      "Many environmentalists argue that by increasing oil and gas supplies and lowering prices for consumers, shale drilling is extending the life of fossil fuels to the detriment of the environment and the development of cleaner energy.\n",
      "The shale drilling revolution has remade the global energy market, with imports from members of the Organization of the Petroleum Exporting Countries plunging by 20 percent from late 2016 to late 2017. At the same time, exports rose by hundreds of thousands of barrels a day.\n",
      "Nothing like the current situation was foreseen in late 2014, when rising domestic production began weighing on global oil prices.\n",
      "In response, Saudi Arabia led OPEC in a new direction. Instead of throttling back to support prices as the cartel had done so often, it left the market alone and even increased production for a time.\n",
      "Prices fell below $40 a barrel, as the Saudis and their allies hoped to drive American operations out of business by making shale drilling uneconomical. American exploration quickly dropped, but the price squeeze made companies more innovative in the use of drilling technologies, robotics and sensors to maximize output and reduce costs.\n",
      "While scores of smaller companies went out of business, the survivors lengthened horizontal wells to yield more oil, and used clever hedging and drilling strategies to maximize profits even when prices slumped.\n",
      "The response surprised the global oil community. OPEC, Russia and allied producing countries changed course and began cutting back again in 2016.\n",
      "“OPEC missed the point,” said René Ortiz, a former OPEC secretary general and former Ecuadorean energy minister. “They thought they could recover the U.S. market by bringing the prices down. Now the U.S. has gained the leading position in the world oil market regardless of what OPEC does.”\n",
      "“This displacement of Saudi oil, Nigerian oil, Libyan oil and Venezuelan oil,” Mr. Ortiz concluded, “was never anticipated.”\n",
      "A week ago, OPEC leaders met in Oman to discuss a probable extension of production cuts into 2019 to support prices. Their biggest obstacle is the United States.\n",
      "Technological advances unlocking oil from tight rocks like shale has led to a drilling frenzy enabling a doubling of output in a decade, transforming unlikely places like North Dakota and New Mexico into world class petroleum hubs. Pipelines are being built across Texas to serve ports where oil can be pumped onto tankers headed for China, India and other markets.\n",
      "Domestic production last year averaged 9.3 million barrels a day, and the Energy Department projects that the figure will climb to 10.3 million barrels a day this year, surpassing the record set in 1970. In the meantime, since a 40-year export ban was lifted in 2015, exports of American oil have risen to roughly two million barrels a day — more than many OPEC members.\n",
      "The department projects an additional increase in domestic production of 500,000 barrels a day in 2019.\n",
      "Concerns over climate change as well as the growing popularity of electric cars and the eventual aging of the best shale fields will probably curb production and demand over the next few decades. But in the short term, the boom has changed the landscape.\n",
      "The Energy Department projects that the recent surge will hold the price of Brent crude, the global benchmark, to $60 a barrel in 2018 and $61 a barrel in 2019 — a modest increase from $54 last year. (The Brent price rose above $70 a barrel this month, but few analysts see a return to $100-a-barrel oil.)\n",
      "The emerging order in the energy realm is a stable balance of power. Saudi Arabia, which essentially runs OPEC, has put a floor under the oil price — probably around $50 a barrel — with its limits on output and exports over the last four years. But now the United States, by the sheer force of its production, the supremacy of its technology, and an unmatched pipeline, refinery and storage structure, has put a ceiling to the price.\n",
      "Experts note that when oil climbs to $60 a barrel and higher, as it has lately, a drilling rush commences — the national rig count has climbed by over a third in the last year — promising to refill domestic and even global energy inventories. Only a major war or other disruption is likely to send prices soaring.\n",
      "“We have all suffered these depressed prices over the last two years and we are excited to see the new prices and we will respond accordingly,” said Harald Jordan, vice president for engineering at Peak Energy, a Colorado-based producer. “You will see rig activity continue to increase.”\n"
     ]
    }
   ],
   "source": [
    "crawler = Crawler()\n",
    "\n",
    "siteData = [\n",
    "    ['O\\'Reilly Media', 'http://oreilly.com', 'h1', 'section#product-description'],\n",
    "    ['Reuters', 'http://reuters.com', 'h1', 'div.StandardArticleBody_body_1gnLA'],\n",
    "    ['Brookings', 'http://www.brookings.edu', 'h1', 'div.post-body'],\n",
    "    ['New York Times', 'http://nytimes.com', 'h1', 'div.StoryBodyCompanionColumn div p']\n",
    "]\n",
    "websites = []\n",
    "for row in siteData:\n",
    "    websites.append(Website(row[0], row[1], row[2], row[3]))\n",
    "\n",
    "crawler.parse(websites[0], 'http://shop.oreilly.com/product/0636920028154.do')\n",
    "crawler.parse(\n",
    "    websites[1], 'http://www.reuters.com/article/us-usa-epa-pruitt-idUSKBN19W2D0')\n",
    "crawler.parse(\n",
    "    websites[2],\n",
    "    'https://www.brookings.edu/blog/techtank/2016/03/01/idea-to-retire-old-methods-of-policy-education/')\n",
    "crawler.parse(\n",
    "    websites[3], \n",
    "    'https://www.nytimes.com/2018/01/28/business/energy-environment/oil-boom.html')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawling through sites with search"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Content:\n",
    "    \"\"\"Common base class for all articles/pages\"\"\"\n",
    "\n",
    "    def __init__(self, topic, url, title, body):\n",
    "        self.topic = topic\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "        self.url = url\n",
    "\n",
    "    def print(self):\n",
    "        \"\"\"\n",
    "        Flexible printing function controls output\n",
    "        \"\"\"\n",
    "        print('New article found for topic: {}'.format(self.topic))\n",
    "        print('URL: {}'.format(self.url))\n",
    "        print('TITLE: {}'.format(self.title))\n",
    "        print('BODY:\\n{}'.format(self.body))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Website:\n",
    "    \"\"\"Contains information about website structure\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, searchUrl, resultListing, resultUrl, absoluteUrl, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.searchUrl = searchUrl\n",
    "        self.resultListing = resultListing\n",
    "        self.resultUrl = resultUrl\n",
    "        self.absoluteUrl = absoluteUrl\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GETTING INFO ABOUT: python\n",
      "New article found for topic: python\n",
      "URL: Learning Python, 5th Edition \n",
      "TITLE: \n",
      "Get a comprehensive, in-depth introduction to the core Python language with this hands-on book. Based on author Mark Lutz’s popular training course, this updated fifth edition will help you quickly write efficient, high-quality code with Python. It’s an ideal way to begin, whether you’re new to programming or a professional developer versed in other languages. \n",
      "\n",
      "Complete with quizzes, exercises, and helpful illustrations,  this easy-to-follow, self-paced tutorial gets you started with both Python 2.7 and 3.3— the latest releases in the 3.X  and 2.X lines—plus all other releases in common use today. You’ll also learn some advanced language features that recently have become more common in Python code.\n",
      "\n",
      "Explore Python’s major built-in object types such as numbers, lists, and dictionaries \n",
      "Create and process objects with Python statements, and learn Python’s general syntax model\n",
      "Use functions to avoid code redundancy and package code for reuse\n",
      "Organize statements, functions, and other tools into larger components with modules \n",
      "Dive into classes: Python’s object-oriented programming tool for structuring code\n",
      "Write large programs with Python’s exception-handling model and development tools\n",
      "Learn advanced Python tools, including decorators, descriptors, metaclasses, and Unicode processing\n",
      "\n",
      "\n",
      "BODY:\n",
      "http://shop.oreilly.com/product/0636920028154.do\n",
      "New article found for topic: python\n",
      "URL: The Hutchins Center Explains: Budgeting for aging America\n",
      "TITLE: \n",
      "\n",
      "\n",
      "For decades, we have been hearing that the baby-boom generation was like a pig moving through a python–bigger than the generations before and after. \n",
      "That’s true. But that’s also a very misleading metaphor for understanding the demographic forces that are driving up federal spending: They aren’t temporary. The generation born between 1946 and 1964 is the beginning of a demographic transition that will persist for decades after the baby boomers die, the consequence of lengthening lifespans and declining fertility. Putting the federal budget on a sustainable course requires long-lasting fixes, not short-lived tweaks.  \n",
      "First, a few demographic facts.\n",
      "As the chart below illustrates, there was a surge in births in the U.S. at the end of World War II, a subsequent decline, and then an uptick as baby boomers began having children.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Although the population has been rising, the number of births in the U.S. the past few years has been below the peak baby-boom levels, possibly because many couples chose not to have children during bad economic times. More significant, fertility rates–roughly the number of babies born per woman during her lifetime–have fallen well below pre-baby-boom levels.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Meanwhile, Americans are living longer. In 1950, a man who made it to age 65 could expect to live until 78 and a woman until 81. Social Security’s actuaries project that a man who lived to age 65 in 2010 will reach 84 and a woman age 86.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Put all this together, and it’s clear that a growing fraction of the U.S. population will be 65 or older.   \n",
      "The combination of longer life spans and lower fertility rates means the ratio of elderly (over 65) to working-age population (ages 20 to 64) is rising. As the chart below illustrates, the ratio will rise steadily as more baby boomers reach retirement age–and then it levels off.  \n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Diversity Explosion\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy William H. Frey \n",
      "2018\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Society at a Glance 2014\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Organization for Economic Cooperation and Development OECD \n",
      "2014\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Brookings-Wharton Papers on Urban Affairs: 2002\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tEdited by William G. Gale and Janet Rothenberg Pack \n",
      "2010\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Simply put, this doesn’t look like a pig in a python.  \n",
      "So what do these demographic facts portend for the federal budget?  In simple dollars and cents, the federal government spends more on the old than the young. More older Americans means more federal spending on Social Security and Medicare, the health insurance program for the elderly. On top of that, health care spending per person is likely to continue to grow faster than the overall economy.\n",
      "The net result: 85 percent of the increase in federal spending that the Congressional Budget Office projects for the next 10 years, based on current policies, will go toward Social Security, Medicare and other major federal health programs, and interest on the national debt.\n",
      " \n",
      "Authors\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "David Wessel\n",
      "Director - The Hutchins Center on Fiscal and Monetary Policy Senior Fellow - Economic Studies\n",
      "\n",
      " Twitter\n",
      "davidmwessel\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Louise Sheiner\n",
      "The Robert S. Kerr Senior Fellow - Economic Studies Policy Director - The Hutchins Center on Fiscal and Monetary Policy\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Restraining future deficits and the size of the federal debt mean restraining spending on these programs or raising taxes–and probably both. One-time savings or minor tweaks won’t suffice. Nor will limiting the belt-tightening to annually appropriated spending.\n",
      "The fundamental fiscal problem is not coping with the retirement of the baby boomers and then going back to budgets that resemble those of the past. The fundamental fiscal problem is that retirement of the baby boomers marks a major demographic transition for the nation, one that will require long-lived changes to benefit programs and taxes.\n",
      "Editor’s Note: This post originally appeared on \n",
      "The Wall Street Journal’s Washington Wire\n",
      " on December 18, 2015.\n",
      "BODY:\n",
      "https://www.brookings.edu/blog/up-front/2015/12/21/the-hutchins-center-explains-budgeting-for-aging-america/\n",
      "New article found for topic: python\n",
      "URL: Inside the Pentagon’s Secret Afghan Spy Machine\n",
      "TITLE: \n",
      "The Pentagon’s top researchers have rushed a classified and controversial intelligence program into Afghanistan. Known as “Nexus 7,” and previously undisclosed as a war-zone surveillance effort, it ties together everything from spy radars to fruit prices in order to glean clues about Afghan instability.\n",
      "\n",
      "BODY:\n",
      "https://www.brookings.edu/opinions/inside-the-pentagons-secret-afghan-spy-machine/\n",
      "New article found for topic: python\n",
      "URL: The Silicon Valley Wage Premium\n",
      "TITLE: \n",
      "\n",
      "\n",
      "Software application developers earn large salaries in the United States, $96,260 a year on average. But in metropolitan San Jose they earn $131,270, the highest in the country. There are many partial explanations for this—local cost of living, differences in education levels, experience, and industry—but none of them quite account for it. It turns out that developers living in San Jose have acquired the specific skills most valued by employers.\n",
      " \n",
      "As the map below shows, there is a huge amount of variation in earnings for software application developers across regional labor markets. In large metropolitan areas like New York, they earn $105,000, but in Louisville, they earn just $72,000.\n",
      " \n",
      "\n",
      "Average Salary of Software Application Developers by Metropolitan Area, 2013\n",
      "\n",
      " \n",
      "\n",
      "\n",
      " \n",
      "Similar patterns could be shown for other occupations, of course; for even within the same job title, people vary by education and experience, and regions vary by company and industry mix, productivity and export orientation, which all affect salaries and regional housing prices.\n",
      " \n",
      "The surprising thing, when it comes to software developers and other skilled occupations too, is that none of these factors can fully account for the San Jose premium. Software developers in San Jose are typically slightly less experienced, and while their levels of education are higher—including their likelihood of having majored in engineering or computer science—the difference is not enough to explain their elevated earnings. Likewise, the cost of living in San Jose is remarkably high, but comparable to other major cities. \n",
      " \n",
      "\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "The New Localism\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Bruce Katz and Jeremy Nowak \n",
      "2018\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "The Public Wealth of Cities\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Dag Detter and Stefan Fölster \n",
      "2017\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Challenges in the Process of China’s Urbanization\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tEdited by Karen Eggleston, Jean C. Oi, and Yiming Wang \n",
      "2017\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "So what distinguishes San Jose software developers? \n",
      " \n",
      "To figure this out, I analyzed a database of 29 million job vacancies advertised online during 2013 as compiled by the analytics firm Burning Glass. Of these, roughly 1.4 million were for software application developers, making it the most in-demand occupation. In total, 3 million ads also contained salary information, which I could use to estimate the average value of each distinct skill advertised. \n",
      " \n",
      "Author\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "J\n",
      "\n",
      "\n",
      "\n",
      "Jonathan Rothwell\n",
      "Former Brookings Expert\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "In San Jose, the skills advertised for software developers are particularly valuable. The average vacancy requires higher value skills in San Jose than almost any other metropolitan area, even using national rather than local salary values.  \n",
      " \n",
      "For example, 8.4 percent of ads for software developers in San Jose requested Java, a widely used programming language, associated with an average salary of $98,000 across all U.S. ads mentioning both it and a salary requirement. Yet, for the United States as a whole, just 5.7 percent of software developer ads required Java. In New York City, the share was 6.7, and it was 4.7 in Louisville.\n",
      " \n",
      "Other high-value programming languages and skills were disproportionately advertised in San Jose, such as Linux, C++, Python and the term “software engineering.”  These skills were much less commonly required for jobs in Louisville and even New York. Only 0.2 percent of software jobs required Python in Louisville and 1.7 in New York City, compared to 2.8 percent in San Jose. It is valued at $100,345.\n",
      " \n",
      "These and other skills contribute to the high premium enjoyed by Silicon Valley computer workers, but they could be profitably learned by a much larger swath of people, as online educators like Treehouse, Udacity, and Code Fellows aim to demonstrate.\n",
      "\n",
      "BODY:\n",
      "https://www.brookings.edu/blog/the-avenue/2014/08/06/the-silicon-valley-wage-premium/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: python\n",
      "URL: Leveraging the disruptive power of artificial intelligence for fairer opportunities\n",
      "TITLE: \n",
      "According to President Obama’s Council of Economic Advisers (CEA), approximately 3.1 million jobs will be rendered obsolete or permanently altered as a consequence of artificial intelligence technologies. Artificial intelligence (AI) will, for the foreseeable future, have a significant disruptive impact on jobs. That said, this disruption can create new opportunities if policymakers choose to harness them—including some with the potential to help address long-standing social inequities. Investing in quality training programs that deliver premium skills, such as computational analysis and cognitive thinking, provides a real opportunity to leverage AI’s disruptive power.\n",
      "\n",
      "Author\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Makada Henry-Nickie\n",
      "David M. Rubenstein Fellow - Governance Studies, Race, Prosperity, and Inclusion Initiative\n",
      "\n",
      " Twitter\n",
      "mhnickie\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "AI’s disruption presents a clear challenge: competition to traditional skilled workers arising from the cross-relevance of data scientists and code engineers, who can adapt quickly to new contexts. Data analytics has become an indispensable feature of successful companies across all industries. This reality dictates that companies invest heavily in data analytics to remain competitive and profitable. Consequently, unlikely industries such as retail, banking, finance, and even agricultural firms are aggressively competing for talent with specific computational data science and programming skills. A recent IBM report expertly quantifies the scope and breadth of employers’ hiring demands, noting that “[d]emand for data-driven decision makers, such as data-enabled marketing managers, will comprise one-third of the data savvy professional job market, with a projected increase of 110,000 positions by 2020.” Herein lies a window of opportunity: the rapidly growing technical skills gap.\n",
      "Investing in high-quality education and training programs is one way that policymakers proactively attempt to address the workforce challenges presented by artificial intelligence. It is essential that we make affirmative, inclusive choices to ensure that marginalized communities participate equitably in these opportunities.\n",
      "Policymakers should prioritize understanding the demographics of those most likely to lose jobs in the short-run. As opposed to obsessively assembling case studies, we need to proactively identify policy entrepreneurs who can conceive of training policies that equip workers with technical skills of “long-game” relevance. As IBM points out, “[d]ata democratization impacts every career path, so academia must strive to make data literacy an option, if not a requirement, for every student in any field of study.”\n",
      "Machines are an equal opportunity displacer, blind to color and socioeconomic status.\n",
      "Machines are an equal opportunity displacer, blind to color and socioeconomic status. Effective policy responses require collaborative data collection and coordination among key stakeholders—policymakers, employers, and educational institutions—to  identify at-risk worker groups and to inform workforce development strategies. Machine substitution is purely an efficiency game in which workers overwhelmingly lose. Nevertheless, we can blunt these effects by identifying critical leverage points.\n",
      "Investing in innovative education and training is an excellent place to start. Bill Gates’ recent $1.7 billion investment in U.S. public schools is a sign of the way forward, which offers two compelling messages for policymakers. First, innovate and experiment until we identify the right policies. Second, prioritize high-needs schools in poor neighborhoods; they deserve distinct attention to close their opportunity gaps and prepare them to be competitive in the future workforce.\n",
      "Policymakers can choose to harness AI’s disruptive power to address workforce challenges and redesign fair access to opportunity simultaneously. We should train our collective energies on identifying practical policies that update our current agrarian-based education model, which unfairly disadvantages children from economically segregated neighborhoods. Evidence from a Harvard and New York University research study suggests attending a high-quality high school increases a student’s chances of attending a four-year college; which by extension improves their future income earning potential.\n",
      "Let me ask a bold question: how much do we lose if we experiment with substituting an entry-level data science class for machine shop or a vocational carpentry program in urban high schools and community colleges? A 2010 pilot partnership between the University of California, Los Angeles and the National Science Foundation is an encouraging sign; the pilot focuses on redesigning computer science curricula in urban high schools to include newer mobile technologies and computational analysis.\n",
      "\n",
      "Related\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "Will robots and AI take your job? The economic and political consequences of automation\n",
      "\n",
      "Darrell M. West\n",
      "Wednesday, April 18, 2018\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "The state of self-driving car laws across the U.S.\n",
      "\n",
      "Jack Karsten and Darrell West\n",
      "Tuesday, May 1, 2018\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "TechTank\n",
      "How blockchain could improve election transparency\n",
      "\n",
      "Kevin C. Desouza and Kiran Kabtta Somvanshi\n",
      "Wednesday, May 30, 2018\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Data science is an applied computational technology best suited to inquisitive minds, making it appropriate for young students. Google’s TensorFlow is an open source machine-learning platform; its free price tag makes the platform an accessible and scalable training resource for schools with constrained budgets. Introducing a data science program into urban schools would be a major paradigm shift for these students. An applied data science program teaching gateway coding skills such as Python, R, SQL, and computational analysis would boost employment possibilities and create meaningful pathways to economic mobility.\n",
      "I am suggesting that we leverage AI’s transformative power to disrupt diminishing possibilities for marginalized groups, like young men of color, who often do not feature in innovative-themed discussions outside of the social justice arena. Open Source groups such as Code.org and StudentRND exemplify the kinds of transformational approaches that democratize access and opportunity.\n",
      "Producing a diverse pipeline of tech-savvy workers for Google and Amazon, even if only at the entry level, is a more attainable dream for most cities than competing in a stacked race for Amazon’s HQ2. Broadening adoption of artificial intelligence technologies poses significant workforce challenges, but it also offers the chance to blunt these effects and create opportunities for marginalized groups if we act preemptively.\n",
      "Google is a donor to the Brookings Institution. The findings, interpretations, and conclusions posted in this piece are solely those of the authors and not influenced by any donation.\n",
      "\n",
      "BODY:\n",
      "https://www.brookings.edu/blog/techtank/2017/11/16/leveraging-the-disruptive-power-of-artificial-intelligence-for-fairer-opportunities/\n",
      "New article found for topic: python\n",
      "URL: Idea to Retire: Old methods of policy education\n",
      "TITLE: \n",
      "Public policy and public affairs schools aim to train competent creators and implementers of government policy. While drawing on the principles that gird our economic and political systems to provide a well-rounded education, like law schools and business schools, policy schools provide professional training. They are quite distinct from graduate programs in political science or economics which aim to train the next generation of academics. As professional training programs, they add value by imparting both the skills which are relevant to current employers, and skills which we know will be relevant as organizations and societies evolve. \n",
      "The relevance of the skills that policy programs impart to address problems of today and tomorrow bears further discussion. We are living through an era in which societies are increasingly interconnected. The wide-scale adoption of devices such as the smartphone is having a profound impact on our culture, communities, and economy. The use of social and digital media and associated means of communication enabled by mobile devices is changing the tone, content, and geographic scope of our conversations, modifying how information is generated and consumed, and changing the very nature of citizen engagement. \n",
      "Information technology-based platforms provisioned by private providers such as Facebook, Google, Uber, and Lyft maintain information about millions of citizens and enable services such as transportation that were mediated in the past solely by the public sector. Surveillance for purposes of public safety via large-scale deployment of sensors also raises fundamental questions about information privacy. From technology-enabled global delivery of work to displacement and replacement of categories of work, some studies estimate that up to 47 percent of U.S. employment might be at risk of computerization with an attendant rise in income inequality. These technology-induced changes will affect every policy domain. How should policy programs best prepare students to address societal challenges in this world that is being transformed by technology? We believe the answer lies in educating students to be “men and women of intelligent action.” \n",
      "A model of policy education\n",
      "We begin with a skills-based model of policy education. These four essential skills address the general problems policy practitioners frequently face:\n",
      "\n",
      "Design skills to craft policy ideas \n",
      "Analytical skills to make smart ex ante decisions \n",
      "Interpersonal experience to manage policy implementation  \n",
      "Evaluative skills to assess outcomes ex post and correct course if necessary\n",
      "\n",
      "These skills make up the policy analysis toolkit required to be data driven practitioner of “intelligent action” in any policy domain. This toolkit needs to be supplemented by an understanding of how technology is transforming societal challenges, enabling new solutions, or disrupting existing regulatory regimes. This understanding is essential to policy formulation and implementation. \n",
      "Pillar 1: Design skills\n",
      "As with engineering, where design precedes analysis, this first pillar seeks to educate students in thinking creatively about problems in order to devise and develop policy ideas. Using ideas derived from design, divergent and convergent thinking principles are employed to generate, explore, and arrive at a candidate set of solutions. Using Uber as an example, an approach to identify and explore the key policy issues such as convenience, costs, driver working hours, and insurance would involve interviewing and observing both incumbent taxi drivers and Uber drivers. This in turn would lead to a set of alternatives that deserve further and careful consideration.  Using these skills, candidate designs and choices that are generated can be evaluated using the policy analytic toolkit. \n",
      "Pillar 2: Analytical skills\n",
      "At Carnegie Mellon, we are often cited in media and interrogated by peers on our approach to analytical and technology skills education. Curiosity about which skills are the “right” skills to teach policy practitioners are common, but we believe this is the wrong approach. We instead begin from the premise that policy or management decisions should be grounded in evidence.  We then determine the skills required to assemble the types of evidence that will likely be available to policy makers in the future.  In increasingly instrumented environments where citizens and infrastructure produce continuous streams of data, making sense of it all will require a somewhat different set of skills. We believe that a grounding in micro-economics, operations research, statistics, and program evaluation (aka causal inference) to be an essential core to policy programs. \n",
      "New coursework will teach students to work with multi-variable data and machine learning with an emphasis on prediction. This material ought to be part of the required coursework in statistics given the importance of prediction in many policy implementation settings. Along the same lines, the ability to work with unstructured data (especially text) and data visualization will become increasingly relevant to all students, not just those students who want to specialize in data analytics. Finally, knowledge of data manipulation and analysis languages such as Python and R for analytic work will be important because data often has to be massaged and cleansed prior to analysis. An important task for programs will be to determine the competencies expected of graduates. \n",
      "Pillar 3: Interpersonal experiences\n",
      "The third pillar of the skills-based model is interpersonal experience, where the practiced habits of good communication and steady negotiation developed with a sound understanding of organizations, their design and their behaviors. We label these purposely as experiences rather than skills because we believe they are best practiced either in the real-world or in simulated real-world settings. It is also in this pillar where practitioners learn the knowledge necessary to become credible experts in their domain. We believe that in addition to core coursework in the area, a supplementary curriculum which provides students with opportunities to gain these experiences is an essential component of our educational model.\n",
      "Pillar 4: Evaluative skills\n",
      "\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Constitution 3.0\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tEdited by Jeffrey Rosen and Benjamin Wittes \n",
      "2013\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "The Need for Speed\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Robert E. Litan and Hal J. Singer \n",
      "2013\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "After the Breakup\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Robert W. Crandall \n",
      "2010\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "The ability to carefully diagnose the effectiveness of policy or management interventions is the fourth pillar of our model. It is insufficient to create and execute policy without measurement, and this is where both careful thought to the fundamental issues of measurement and evaluation become important. The ability to make objective judgments on the benefits, liabilities, and unintended consequences of prior policies is the goal of this set of skills. Here, sound statistical and econometric training with an understanding of the principles of causal inference is essential. In addition, program evaluation skills such as cost-benefit and financial analysis help practitioners round out their evaluation skills by considering both non-monetary and economic impacts.\n",
      "What should be retired?\n",
      "A skills-based approach might replace certain aspects of existing policy training.  This depends on a number of factors specific to each institution, but three generally applicable observations are clear. First, real-world experiences are a powerful way to encode domain learning as well as project management skills. Through project-based work, students can learn about institutional contexts in specific policy domains and political processes such as budgeting. Second, team-based projects allow students to learn and apply principles of management and organizational behavior. At Carnegie Mellon, we refer to these as “systems synthesis” projects, since they require students to adopt a systemic point of view and to synthesize a number of skills in their policy analysis toolkit. Third, interpersonal skills training can be practiced through activities such as weekend negotiation exercises, hackathons, and speaker series. These activities can be highly intentional and fashioned to reinforce skills rather than as a recess from the “real work” of classroom training. Since students complete graduate programs in such a short time, counseling them to focus on outcomes from day one will allow them to choose a reinforcing set of coursework and real-world experiences. \n",
      "In summary, we argue for a model of policy education that views practitioners as future problem solvers. Good policy education must consider the ways in which problems will present themselves, and the ways in which answers will obscure themselves. Rigorous training grounded in the analysis of available evidence and buoyed by real-world interpersonal experiences is a sound approach to relevant, durable policy training.\n",
      " \n",
      "Authors\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "R\n",
      "\n",
      "\n",
      "\n",
      "Ramayya Krishnan\n",
      "Ramayya Krishnan is the dean of H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University where he is the W.W. Cooper and Ruth F. Cooper Professor of Management Science and Information Systems.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "J\n",
      "\n",
      "\n",
      "\n",
      "Jon Nehlsen\n",
      "Jon Nehlsen is senior director of external relations at H. John Heinz III College of Information Systems and Public Policy at Carnegie Mellon University.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Read other essays in the Ideas to Retire blog series here.\n",
      "\n",
      "BODY:\n",
      "https://www.brookings.edu/blog/techtank/2016/03/01/idea-to-retire-old-methods-of-policy-education/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: python\n",
      "URL: Skills, success, and why your choice of college matters\n",
      "TITLE: \n",
      "\n",
      "\n",
      "Amidst growing frustration with the cost of higher education, complaints also abound about its quality. One critique, launched in the book Academically Adrift by two sociologists, finds little evidence that college students score better on measures of critical thinking, writing, and reasoning after attending college. This is something of a paradox, since strong evidence shows that attending college tends to raise earnings power, even for students who start with mediocre preparation. \n",
      "Our recent study uses a different approach to assess the value of a college education. We find that the particular skills listed by a college’s alumni on their resumes predict how well graduates from those schools perform in terms of earning a living, meeting debt obligations, and working for high-paying or innovative companies. Since jobs requiring more valuable skills typically require at least some college education, this finding suggests many students are gaining valuable skills from college. But the variation in alumni skills across schools is wide, even after considering the aptitude of the students in terms of their pre-admission test scores. This variation implies that what one studies and where have big effects on economic outcomes.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Skills versus degrees\n",
      "\n",
      "It is widely known that education raises individual earnings, but education—measured in years of study or level of degree—is a very rough measure of learning. Thus, it is not surprising that studies consistently find that skills are an important predictor of economic outcomes. People with higher test scores—another measure of learning—earn higher wages, even with the same level of education. Likewise, graduates with technical degrees earn more, as do workers in occupations requiring more STEM skills. At the international scale, performance on standardized exams has a much stronger statistical relationship with economic outcomes than do years of education, according to a new OECD study.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "How we valued skills by college\n",
      "\n",
      "Using data from the company Burning Glass, we calculated the average salary listed for distinct skills based on 3 million job vacancy ads. To match these skills to colleges, we used data from LinkedIn’s college profile pages, which show the 25 most common skills (e.g., customer service, Microsoft Excel, Python) listed by alumni from each college. For the average college, we observed 1,150 profiles per skill. (A great advantage of using LinkedIn data is the large sample size.) We obtained data for 2,164 colleges representing profiles for 2.5 million U.S. residents who attended college. By comparison, Academically Adrift surveyed 2,300 college graduates.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Alumni with more valuable skills earn higher salaries\n",
      "\n",
      "Measured at mid-career (meaning at least 10 years of working), salaries tend to be much higher for alumni who list high-value skills on their resumes. Earnings go up by an average of $2,600 for every decile of skill. Our more detailed empirical work shows that skills predict higher earnings even after controlling for math test scores on the ACT and SAT, as well as other student characteristics like family income.\n",
      "Cal Tech graduates list the highest-value skills (e.g., Matlab, Python, C++, algorithms, and machine learning) and typically earn $126,000 at mid-career. Other four-year schools with high-value skills and high salaries include Harvey Mudd, MIT, the Polytechnic Institute of New York University, and the Air Force Academy. \n",
      "Earnings data from two-year colleges are not as widely available, and the correlation with alumni skills is weaker, but alumni from those schools also seem to benefit from higher skills training. Top schools include the Pittsburg Institute of Aeronautics, Spartan College of Aeronautics and Technology (Tulsa, Okla.), Coleman University (San Diego), Hondros College (Columbus, Ohio), and SUNY College of Technology at Alfred.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Alumni with more valuable skills have higher loan repayment rates\n",
      "\n",
      "\n",
      "\n",
      "Related Books\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Leapfrogging Inequality\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Rebecca Winthrop; With Adam Barton and Eileen McGivney \n",
      "2018\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "The Transformation of Title IX\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy R. Shep Melnick \n",
      "2018\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " \n",
      "\n",
      "\n",
      "\n",
      "Making College Work\n",
      "\n",
      "\n",
      "\t\t\t\t\t\tBy Harry J. Holzer and Sandy Baum \n",
      "2017\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "As an alternative to mid-career earnings, we also analyzed how skills predict the ability to make student loan payments immediately after graduation. Here too, more valuable skills translate into labor market success. For example, not a single Harvey Mudd attendee between 2009 and 2011 defaulted on his or her federal loans within three years of leaving. Repayment rates average 95 percent for four-year colleges in the top 10 percent for alumni skills but 87 percent for those in the bottom 10 percent. For two-year colleges, repayment rates are uniformly lower, but colleges offering higher-value skills still have significantly higher repayment rates than those that do not.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Alumni with more valuable skills are more likely to work for top organizations\n",
      "\n",
      "Another outcome measure is whether alumni work for a desirable company or organization. LinkedIn lists the 25 enterprises that employ the most alumni from each school. To quantify the value of working for a given entity, employers were coded for desirability with data from a 2014 survey of 46,000 U.S. college students in 329 universities, developed by Universum, a corporate marketing intelligence company. A total of 212 employers, including government agencies, made it onto a top 100 list for at least one group of student majors. The most desirable employers across majors were Google, Disney, Apple, Microsoft, the FBI, Nike, NASA, the Environmental Protection Agency, the Peace Corps, and Facebook. \n",
      "For the top 10 percent of four-year colleges on alumni skills, half of LinkedIn alumni profiles indicate employment at one of the 212 top-rated companies, compared to just one in four for schools in the bottom 10 percent. For two-year schools, nearly two in five alumni (37 percent) of top-tier schools by skill worked for a top company, versus one in five alumni (21 percent) of bottom-tier schools.\n",
      "For placement at Google specifically, Harvey Mudd has the highest rate—2 percent of all alumni—followed by Stanford, Carnegie Mellon, Caltech, and MIT. Almost all of the colleges with the highest placement rates at Google are in the top 20 percent of alumni skills, including liberal arts colleges like Swarthmore, Pomona, Claremont, McKenna, and Williams.\n",
      " \n",
      "Author\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "J\n",
      "\n",
      "\n",
      "\n",
      "Jonathan Rothwell\n",
      "Former Brookings Expert\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Alumni with more valuable skills are more likely to work for innovative organizations\n",
      "\n",
      "Workers who contribute to the creation and development of new, valuable products can lift the living standards of people around the world. Companies that patent are more likely to be creating these sorts of advanced industry products, and 843 entities, including universities and government agencies, own at least 40 patents granted by the U.S. Patent and Trademark Office in 2014. \n",
      "Four-colleges that graduate alumni in the top 10 percent by skill are twice as likely to have graduates working at a top patenting organization than are colleges in the bottom 10 percent (3.3 versus 1.6 percent). Likewise, graduates from two-year colleges are about twice as likely to be working for a patenting entity if their school is in the top 10 percent compared to the bottom (1.9 versus 0.9 percent).\n",
      "Schools with high placement rates at patenting entities include those listed above, as well as less the U.S. Naval Academy, Lawrence Technological University, the Stevens Institute of Technology, Santa Clara University, Brazosport College, Mount Mercy University, University of Texas-Dallas, the Missouri University of Science and Technology, and San Jose State University.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "How to judge colleges\n",
      "\n",
      "Earnings and other economic outcomes should not be equated with social value, and there are plenty of jobs and professions—child care, teaching, social work—that pay modestly but are nevertheless highly valuable to society. Colleges that specialize in this training or instill even moderately valuable skills in the least academically prepared students may be socially important institutions even if their alumni frequently are less affluent.\n",
      "Nonetheless, earnings clearly matter privately and socially, as does work that supports innovation and highly productive advanced industries. Many colleges offer programs of study in fields that appear to have almost no market value—nor even any social value since the knowledge acquired is never put to use, at least through paid employment. In this sense, how well colleges instill highly valuable skills and prepare students to contribute productively to the economy should be an important consideration when evaluating schools. Colleges that do this for the students least likely to otherwise succeed are offering an even more beneficial service, as we have discussed in our value-added college research.\n",
      "\n",
      "\n",
      "Correction: A previous version of this post showed graphs which reversed the label on 2- and 4-year colleges. The graphs have been corrected.\n",
      "\n",
      "BODY:\n",
      "https://www.brookings.edu/opinions/skills-success-and-why-your-choice-of-college-matters/\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "New article found for topic: python\n",
      "URL: Modeling with Data: Tools and Techniques for Scientific Computing\n",
      "TITLE: \n",
      "\n",
      "PREFACE\n",
      "\n",
      "\n",
      "BODY:\n",
      "https://www.brookings.edu/articles/modeling-with-data-tools-and-techniques-for-scientific-computing/\n",
      "New article found for topic: python\n",
      "URL: Forum: Debating Bush’s Wars\n",
      "TITLE: \n",
      "\n",
      "In the \n",
      "\n",
      "Winter 2007–08 issue \n",
      "of Survival, Philip Gordon argued that America’s strategy against terror is failing ‘because the Bush administration chose to wage the wrong war’. Survival invited former Bush speechwriter and Deputy Assistant to the President Peter Wehner and Kishore Mahbubani, Dean and Professor at the Lee Kuan Yew School of Public Policy in Singapore, to reflect on Gordon’s arguments. Their \n",
      "comments are available in the above PDF and Philip Gordon’s response is below.\n",
      "\n",
      "BODY:\n",
      "https://www.brookings.edu/articles/forum-debating-bushs-wars/\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-13-2cc6e9570188>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     59\u001b[0m     \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'GETTING INFO ABOUT: '\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0mtopic\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     60\u001b[0m     \u001b[0;32mfor\u001b[0m \u001b[0mtargetSite\u001b[0m \u001b[0;32min\u001b[0m \u001b[0msites\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 61\u001b[0;31m         \u001b[0mcrawler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msearch\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtopic\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtargetSite\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-13-2cc6e9570188>\u001b[0m in \u001b[0;36msearch\u001b[0;34m(self, topic, site)\u001b[0m\n\u001b[1;32m     27\u001b[0m             \u001b[0;31m# Check to see whether it's a relative or an absolute URL\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m             \u001b[0;32mif\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mabsoluteUrl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m                 \u001b[0mbs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetPage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     30\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m                 \u001b[0mbs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetPage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murl\u001b[0m \u001b[0;34m+\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-13-2cc6e9570188>\u001b[0m in \u001b[0;36mgetPage\u001b[0;34m(self, url)\u001b[0m\n\u001b[1;32m      6\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mgetPage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m      7\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m----> 8\u001b[0;31m             \u001b[0mreq\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m      9\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRequestException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(url, params, **kwargs)\u001b[0m\n\u001b[1;32m     70\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     71\u001b[0m     \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetdefault\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'allow_redirects'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 72\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'get'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     73\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     74\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(method, url, **kwargs)\u001b[0m\n\u001b[1;32m     56\u001b[0m     \u001b[0;31m# cases, and look like a memory leak in others.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     57\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0msessions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 58\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     59\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     60\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(self, method, url, params, data, headers, cookies, files, auth, timeout, allow_redirects, proxies, hooks, stream, verify, cert, json)\u001b[0m\n\u001b[1;32m    506\u001b[0m         }\n\u001b[1;32m    507\u001b[0m         \u001b[0msend_kwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mupdate\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0msettings\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 508\u001b[0;31m         \u001b[0mresp\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msend\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mprep\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0msend_kwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    509\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    510\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mresp\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36msend\u001b[0;34m(self, request, **kwargs)\u001b[0m\n\u001b[1;32m    656\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    657\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mstream\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 658\u001b[0;31m             \u001b[0mr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcontent\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    659\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    660\u001b[0m         \u001b[0;32mreturn\u001b[0m \u001b[0mr\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/models.py\u001b[0m in \u001b[0;36mcontent\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    821\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_content\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    822\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 823\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_content\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mbytes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mjoin\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0miter_content\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mCONTENT_CHUNK_SIZE\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mor\u001b[0m \u001b[0mbytes\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    824\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    825\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_content_consumed\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/models.py\u001b[0m in \u001b[0;36mgenerate\u001b[0;34m()\u001b[0m\n\u001b[1;32m    743\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mhasattr\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'stream'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    744\u001b[0m                 \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 745\u001b[0;31m                     \u001b[0;32mfor\u001b[0m \u001b[0mchunk\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraw\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstream\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mchunk_size\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecode_content\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    746\u001b[0m                         \u001b[0;32myield\u001b[0m \u001b[0mchunk\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    747\u001b[0m                 \u001b[0;32mexcept\u001b[0m \u001b[0mProtocolError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/response.py\u001b[0m in \u001b[0;36mstream\u001b[0;34m(self, amt, decode_content)\u001b[0m\n\u001b[1;32m    430\u001b[0m         \"\"\"\n\u001b[1;32m    431\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchunked\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msupports_chunked_reads\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 432\u001b[0;31m             \u001b[0;32mfor\u001b[0m \u001b[0mline\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mread_chunked\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mamt\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mdecode_content\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mdecode_content\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    433\u001b[0m                 \u001b[0;32myield\u001b[0m \u001b[0mline\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    434\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/response.py\u001b[0m in \u001b[0;36mread_chunked\u001b[0;34m(self, amt, decode_content)\u001b[0m\n\u001b[1;32m    596\u001b[0m         \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_error_catcher\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    597\u001b[0m             \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 598\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_update_chunk_length\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    599\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchunk_left\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    600\u001b[0m                     \u001b[0;32mbreak\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/response.py\u001b[0m in \u001b[0;36m_update_chunk_length\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    538\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mchunk_left\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    539\u001b[0m             \u001b[0;32mreturn\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 540\u001b[0;31m         \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_fp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mfp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreadline\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    541\u001b[0m         \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mline\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msplit\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34mb';'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;36m1\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m[\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m]\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    542\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/socket.py\u001b[0m in \u001b[0;36mreadinto\u001b[0;34m(self, b)\u001b[0m\n\u001b[1;32m    584\u001b[0m         \u001b[0;32mwhile\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    585\u001b[0m             \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 586\u001b[0;31m                 \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_sock\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mb\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    587\u001b[0m             \u001b[0;32mexcept\u001b[0m \u001b[0mtimeout\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    588\u001b[0m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_timeout_occurred\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/urllib3/contrib/pyopenssl.py\u001b[0m in \u001b[0;36mrecv_into\u001b[0;34m(self, *args, **kwargs)\u001b[0m\n\u001b[1;32m    278\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    279\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 280\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnection\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrecv_into\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    281\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mOpenSSL\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSSL\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSysCallError\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    282\u001b[0m             \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msuppress_ragged_eofs\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0me\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0margs\u001b[0m \u001b[0;34m==\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0;36m1\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'Unexpected EOF'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/OpenSSL/SSL.py\u001b[0m in \u001b[0;36mrecv_into\u001b[0;34m(self, buffer, nbytes, flags)\u001b[0m\n\u001b[1;32m   1544\u001b[0m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_lib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSSL_peek\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ssl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnbytes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1545\u001b[0m         \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 1546\u001b[0;31m             \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0m_lib\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSSL_read\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ssl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mbuf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mnbytes\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   1547\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_raise_ssl_error\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0m_ssl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   1548\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "class Crawler:\n",
    "\n",
    "    def getPage(self, url):\n",
    "        try:\n",
    "            req = requests.get(url)\n",
    "        except requests.exceptions.RequestException:\n",
    "            return None\n",
    "        return BeautifulSoup(req.text, 'html.parser')\n",
    "\n",
    "    def safeGet(self, pageObj, selector):\n",
    "        childObj = pageObj.select(selector)\n",
    "        if childObj is not None and len(childObj) > 0:\n",
    "            return childObj[0].get_text()\n",
    "        return ''\n",
    "\n",
    "    def search(self, topic, site):\n",
    "        \"\"\"\n",
    "        Searches a given website for a given topic and records all pages found\n",
    "        \"\"\"\n",
    "        bs = self.getPage(site.searchUrl + topic)\n",
    "        searchResults = bs.select(site.resultListing)\n",
    "        for result in searchResults:\n",
    "            url = result.select(site.resultUrl)[0].attrs['href']\n",
    "            # Check to see whether it's a relative or an absolute URL\n",
    "            if(site.absoluteUrl):\n",
    "                bs = self.getPage(url)\n",
    "            else:\n",
    "                bs = self.getPage(site.url + url)\n",
    "            if bs is None:\n",
    "                print('Something was wrong with that page or URL. Skipping!')\n",
    "                return\n",
    "            title = self.safeGet(bs, site.titleTag)\n",
    "            body = self.safeGet(bs, site.bodyTag)\n",
    "            if title != '' and body != '':\n",
    "                content = Content(topic, title, body, url)\n",
    "                content.print()\n",
    "\n",
    "\n",
    "crawler = Crawler()\n",
    "\n",
    "siteData = [\n",
    "    ['O\\'Reilly Media', 'http://oreilly.com', 'https://ssearch.oreilly.com/?q=',\n",
    "        'article.product-result', 'p.title a', True, 'h1', 'section#product-description'],\n",
    "    ['Reuters', 'http://reuters.com', 'http://www.reuters.com/search/news?blob=', 'div.search-result-content',\n",
    "        'h3.search-result-title a', False, 'h1', 'div.StandardArticleBody_body_1gnLA'],\n",
    "    ['Brookings', 'http://www.brookings.edu', 'https://www.brookings.edu/search/?s=',\n",
    "        'div.list-content article', 'h4.title a', True, 'h1', 'div.post-body']\n",
    "]\n",
    "sites = []\n",
    "for row in siteData:\n",
    "    sites.append(Website(row[0], row[1], row[2],\n",
    "                         row[3], row[4], row[5], row[6], row[7]))\n",
    "\n",
    "topics = ['python', 'data science']\n",
    "for topic in topics:\n",
    "    print('GETTING INFO ABOUT: ' + topic)\n",
    "    for targetSite in sites:\n",
    "        crawler.search(topic, targetSite)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawling Sites through Links"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Website:\n",
    "\n",
    "    def __init__(self, name, url, targetPattern, absoluteUrl, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.targetPattern = targetPattern\n",
    "        self.absoluteUrl = absoluteUrl\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag\n",
    "\n",
    "\n",
    "class Content:\n",
    "\n",
    "    def __init__(self, url, title, body):\n",
    "        self.url = url\n",
    "        self.title = title\n",
    "        self.body = body\n",
    "\n",
    "    def print(self):\n",
    "        print('URL: {}'.format(self.url))\n",
    "        print('TITLE: {}'.format(self.title))\n",
    "        print('BODY:\\n{}'.format(self.body))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GETTING https://www.reuters.com\n",
      "GETTING https://www.reuters.com/article/us-usa-trump-5g/trump-national-security-team-sees-building-5g-network-as-option-idUSKBN1FH103\n",
      "URL: https://www.reuters.com/article/us-usa-trump-5g/trump-national-security-team-sees-building-5g-network-as-option-idUSKBN1FH103\n",
      "TITLE: Trump security team sees building U.S. 5G network as option\n",
      "BODY:\n",
      "WASHINGTON (Reuters) - President Donald Trump’s national security team is looking at options to counter the threat of China spying on U.S. phone calls that include the government building a super-fast 5G wireless network, a senior administration official said on Sunday. The official, confirming the gist of a report from Axios.com, said the option was being debated at a low level in the administration and was six to eight months away from being considered by the president himself. The 5G network concept is aimed at addressing what officials see as China’s threat to U.S. cyber security and economic security. The Trump administration has taken a harder line on policies initiated by predecessor Barack Obama on issues ranging from Beijing’s role in restraining North Korea to Chinese efforts to acquire U.S. strategic industries. Earlier this month, AT&T; was forced to scrap a plan to offer its customers handsets built by China’s Huawei after some members of Congress lobbied against the idea with federal regulators, sources told Reuters.  In 2012, Huawei and ZTE Corp were the subject of a U.S. investigation into whether their equipment provided an opportunity for foreign espionage and threatened critical U.S. infrastructure.  Some members of the House intelligence committee remain troubled by security threats posed by Huawei and ZTE, according to a congressional aide. Issues raised in a 2012 committee report about the Chinese firms have “never subsided,” the aide said, adding that there was newer classified intelligence that recently resurfaced those concerns. “We want to build a network so the Chinese can’t listen to your calls,” the senior official told Reuters. “We have to have a secure network that doesn’t allow bad actors to get in. We also have to ensure the Chinese don’t take over the market and put every non-5G network out of business.” Major wireless carriers have spent billions of dollars buying spectrum to launch 5G networks, and it is unclear if the U.S. government would have enough spectrum to build its own 5G network. Furthermore, Accenture has estimated that wireless operators will invest as much as $275 billion in the United States over seven years as they build out 5G. Last year, T-Mobile US Inc spent $8 billion and Dish Network Corp $6.2 billion to win the bulk of broadcast airwaves spectrum for sale in a government auction.  An AT&T; spokesman said they could not comment on something they have not seen, and added: “Thanks to multi-billion dollar investments made by American companies, the work to launch 5G service in the United States is already well down the road.” Later this year, AT&T; is set to be the first to launch mobile 5G service in 12 U.S. locations, the spokesman said. A Verizon spokesman declined to comment. Representatives for Sprint and T-Mobile did not immediately respond to requests for comment.     Another option includes having a 5G network built by a consortium of wireless carriers, the U.S. official said. “We want to build a secure 5G network and we have to work with industry to figure out the best way to do it,” the official said, speaking on condition of anonymity. Axios published documents it said were from a presentation from a National Security Council official. If the government built the network, it would rent access to carriers, Axios said. A looming concern laid out in the presentation was China’s growing presence in the manufacture and operation of wireless networks. A concerted government push could help the U.S. compete on that front, according to the presentation. A 5G network is expected to offer significantly faster speeds, more capacity and shorter response times, which could be utilized for new technologies ranging from self-driving cars to remote surgeries. Telecom companies and their suppliers consider it to be a multibillion-dollar revenue opportunity. Reporting by Steve Holland and Pete Schroeder; Additional reporting by Duston Volz, Suzanne Barlyn and David Shepardson; Editing by Chris Sanders, Peter Cooney and Cynthia OstermanOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-trump-effect-nafta-farmers/u-s-farmers-have-much-to-lose-if-nafta-deal-collapses-idUSKBN1FH0O0\n",
      "URL: https://www.reuters.com/article/us-trump-effect-nafta-farmers/u-s-farmers-have-much-to-lose-if-nafta-deal-collapses-idUSKBN1FH0O0\n",
      "TITLE: U.S. farmers have much to lose if NAFTA deal collapses\n",
      "BODY:\n",
      "CHICAGO/MONTREAL (Reuters) - A collapse of the North American Free Trade Agreement (NAFTA), which U.S. President Donald Trump has threatened to scrap, could create the most profound disruption for U.S. farmers who produce grains, meats and dairy products sold to Canada and Mexico. Blake Erwin, a third-generation American who raises cattle, corn and soybeans in Dixon, Nebraska, said on Saturday that he is not closely monitoring the negotiations, but that he hopes the outcome will support U.S. farmers who are struggling to make a living due to low commodities prices, rising healthcare costs and high property taxes.  “A trade agreement has to be fair for the United States, but we also want to keep those exports going for the farmer,” said Erwin, 34. “We don’t want to mess up any good things we got going.”  Erwin spoke to Reuters over the weekend as U.S., Canadian and Mexican negotiators met in Montreal for the sixth of seven planned rounds of talks to revamp the 1994 pact. U.S. farmers and exporters are fighting to preserve their exports at a time when Canada is finding customers in new markets. They also face strained relations between the United States and Mexico, a major buyer of U.S. corn, wheat, beef, pork and dairy products. “The U.S. is behaving so badly it’s going to create opportunities for Canadian agriculture,” Iowa State University economist Dermot Hayes said last week during a visit to Winnipeg.  Trade flows have already begun to shift. The United States remains the dominant grain supplier to Mexico. Yet Mexico imported 583,000 metric tonnes of corn from Brazil in 2017, a 980 percent jump from the previous year, according to Mexican government trade data. Mexican imports of U.S. soybean meal, used to feed chickens and livestock, fell 29 percent in the first 11 months of 2017, compared with the same period the previous year, according to the U.S. Department of Agriculture.  ‘MORE IMPORTANT THAN PRICE’  Slideshow (17 Images)Trump’s animosity toward Mexico and complaints over trade imbalances have pushed longtime buyers to work with new suppliers and expand existing relationships in South America, the European Union and other regions, trade experts said. “You get partners who build a bond and get real comfortable working together. We’re starting to see that bond becoming more important than price for where countries are buying grains,” said Karl Setzer, risk management team leader for MaxYield Cooperative. Case in point: A rare 30,000-tonne shipment of Brazilian corn steamed its way in November to grain terminals in the state of Veracruz, Mexico, operated by agribusiness heavyweights Cargill Inc [CARG.UL] and Archer Daniels Midland Co (ADM.N).  Despite a steep decline in U.S. corn prices, with stocks sitting at a historic high, the buyer paid a premium for the Brazilian grain - as much as $2 more per tonne, according to trade sources.  A Cargill spokeswoman said the company had no immediate comment. ADM did not respond to requests for comment. Canada last week agreed to join the new version of the Trans-Pacific Partnership, part of a broad effort to court new trade partners. “The tough NAFTA negotiations have convinced Canada that we have to have a number of trading partners, not just one,” said Ron Bonnett, a beef farmer and president of the Canadian Federation of Agriculture.  The revised TPP, known as the Comprehensive and Progressive Agreement for Trans-Pacific Partnership, will reduce tariffs on Canadian pork, beef and wheat to Japan and other markets, in some cases eliminating duties altogether. Darci Vetter, former U.S. chief agriculture negotiator, said if the talks dragged on past March they might not end until next year, making it more challenging to sell American farm products. “Other trade agreements will be implemented, buyers of U.S. products in Mexico and Canada won’t be sure that we are a good long-term bet, and so we’re likely to see our clients react accordingly,” she told a panel on NAFTA in Montreal on Friday. Reporting by P.J. Huffstutter in Chicago and David Ljunggren in Montreal; Writing and additional reporting by Rod Nickel in Winnipeg, Manitoba; Additional reporting by Lucas Jackson in Dixon, Nebraska; Editing by Jim Finkle and Daniel WallisOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-afghanistan-blast/militants-attack-afghan-army-post-near-military-academy-in-capital-idUSKBN1FI07M\n",
      "URL: https://www.reuters.com/article/us-afghanistan-blast/militants-attack-afghan-army-post-near-military-academy-in-capital-idUSKBN1FI07M\n",
      "TITLE: Militants attack Afghan army post near military academy in capital\n",
      "BODY:\n",
      "KABUL (Reuters) - At least four militants attacked an army outpost near one of Afghanistan’s main military academies on Monday and at least one soldier was killed and three wounded, a defense ministry official said. The attack in the western outskirts of the capital, Kabul, came two days after an ambulance bomb in the center of the city killed more than 100 people and just over a week after another attack on the Hotel Intercontinental killed more than 20. Both of those attacks were claimed by the Taliban. Ministry of Defence officials said the militants attacked the outpost near the well-defended Marshal Fahim military academy just before dawn. One of the attackers blew himself up, one had been killed and two were still fighting. One soldier had been killed and three were wounded, said one official, who declined to be identified. Ministry spokesman Dawlat Waziri said three soldiers had been wounded and the clash was still going on. Earlier, resident Mohammad Ehsan said he heard a series of explosions coming from the area beginning at around 5 a.m. and lasting for at least an hour.  Smaller blasts could still be heard at less frequent intervals. In October, a suicide attacker rammed a car full of explosives into a bus carrying cadets from the defense university, which is home to one of Afghanistan’s main officer training schools, killing 15 of them. Reporting by Omar Sobhani; Writing by James Mackenzie; Editing by Robert Birsel and Paul TaitOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-usa-immigration-manchin/democratic-senator-criticizes-pelosis-immigration-comment-idUSKBN1FH0RC\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.reuters.com/article/us-usa-immigration-manchin/democratic-senator-criticizes-pelosis-immigration-comment-idUSKBN1FH0RC\n",
      "TITLE: Democratic senator criticizes Pelosi's immigration comment\n",
      "BODY:\n",
      "WASHINGTON (Reuters) - U.S. Senator Joe Manchin, a moderate Democrat, said on Sunday he thought a new White House immigration plan was a good starting point, and he criticized House Democratic leader Nancy Pelosi for dismissing it as a way to “make America white again.” “We don’t need that type of rhetoric on either side, from Nancy, (Republican House Speaker) Paul Ryan or anybody else,” said Manchin, a West Virginian and a leader of a bipartisan Senate group working on immigration. He spoke on CNN’s “State of the Union” program. Manchin’s comments highlighted differences among Democrats ahead of a Feb. 8 deadline for the U.S. Congress to pass another spending bill and try to reach an immigration agreement that would also protect up to 1.8 million illegal immigrants brought to the United States as children, a group known as “Dreamers,” from deportation. Senior White House officials outlined an immigration plan on Thursday that would offer Dreamers a path to citizenship. The proposal also would curb some legal immigration programs and build a border wall with Mexico. The White House described the language on Dreamers as a major concession to Democrats, but leading Democrats quickly dismissed the plan as a non-starter. FILE PHOTO - House Minority Leader Nancy Pelosi (D-CA) speaks during a news conference after President Donald Trump and the U.S. Congress failed to reach a deal on funding for federal agencies on Capitol Hill in Washington, U.S., January 20, 2018.  REUTERS/Joshua Roberts Pelosi said it held Dreamers “hostage to a hateful anti-immigrant scheme” and accused the administration of Republican President Donald Trump of a campaign “to make America white again.” In a separate interview on Sunday on NBC’s “Meet the Press” program, Manchin said he thought the White House plan was “a good starting point.”   Manchin said the bipartisan group that he and moderate Republican Senator Susan Collins lead would meet on Monday evening to examine the White House immigration outline. “I think we can find a pathway forward; I really do,” Manchin told NBC. Collins, speaking to CBS’s “Face the Nation” on Sunday, said the group hoped to influence a proposal that the Senate’s two whips, Republican John Cornyn and Democrat Dick Durbin, are trying to assemble. “If (Cornyn and Durbin) agree, I have a feeling that that will be a bill that can go all the way to the president’s desk, and that’s our goal,” Collins said.  The group of more than 20 senators from both parties, which has been dubbed the “Common Sense Coalition,” helped to end a three-day U.S. government shutdown last week. Reporting by Susan Cornwell; Editing by Jeffrey Benkoe and Lisa Von AhnOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-usa-trump-russia/republican-senators-urge-congress-to-revisit-bill-to-protect-mueller-idUSKBN1FH0SG\n",
      "URL: https://www.reuters.com/article/us-usa-trump-russia/republican-senators-urge-congress-to-revisit-bill-to-protect-mueller-idUSKBN1FH0SG\n",
      "TITLE: Republican senators urge Congress to revisit bill to protect Mueller\n",
      "BODY:\n",
      "WASHINGTON (Reuters) - The U.S. Congress should revisit proposed legislation to protect special counsel Robert Mueller after President Donald Trump tried to fire him last year while he investigated the Trump campaign’s ties with Russia, two Republican senators said on Sunday. In separate television interviews, Senators Susan Collins and Lindsey Graham expressed dismay at reports the Republican president had told the top White House lawyer to order U.S. Justice Department officials to fire Mueller. “I’ve got legislation protecting Mr. Mueller, and I’d be glad to pass it tomorrow,” Graham told the ABC News “This Week” program.  On CNN’s “State of the Union,” Collins said: “It certainly wouldn’t hurt to put that extra safeguard in place, given the latest stories.”  Tensions over Mueller’s probe are hovering over Trump’s year-old presidency as he prepares to give his first State of the Union Address on Tuesday. The New York Times reported on Thursday that Trump ordered White House counsel Donald McGahn to fire Mueller in June but backed down after McGahn threatened to resign rather than carry out the order. McGahn was “fed up” after Trump’s order, a person familiar with the matter told Reuters. He did not issue an ultimatum directly to the president but told then-White House chief of staff Reince Priebus and then-chief strategist Steve Bannon that he wanted to quit, the source said. FILE PHOTO: Special Counsel Robert Mueller (R) departs after briefing members of the U.S. Senate on his investigation into potential collusion between Russia and the Trump campaign on Capitol Hill in Washington, U.S., June 21, 2017.   REUTERS/Joshua Roberts/File PhotoGraham and three Democratic senators introduced legislation last August that would protect special counsels, including Mueller, by requiring that a panel of federal judges review any action to remove them. The likelihood that such a bill would become law have seemed remote. Republicans in the U.S. House of Representatives have shown little enthusiasm for the idea. “I don’t think there’s a need for legislation right now to protect Mueller,” House Republican leader Kevin McCarthy said on NBC’s “Meet the Press” program. “The president and his team have fully cooperated” with the special counsel, he said. U.S. President Donald Trump looks on during the World Economic Forum (WEF) annual meeting in Davos, Switzerland January 26, 2018. REUTERS/Carlos Barria Republicans hold the majority in both the House and Senate. Mueller is investigating whether Trump associates and the Kremlin colluded during the 2016 presidential election. Russia denies such collusion, and Trump frequently denounces the probe as a “witch hunt.” Both Collins and Graham said they saw no sign that Trump is currently trying to fire Mueller. “I think what happened here is the president had a bad idea,” Collins said. “He talked with his counsel, who explained to an angry and frustrated president why it was a bad idea.” Graham said: “It’s pretty clear to me everyone in the White House knows it’d be the end of President Trump’s presidency if he tried to fire Mr. Mueller.” Reporting by Yasmeen Abutaled and Caren Bohan Writing by Warren Strobel Editing by Lisa Von AhnOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-china-economy-growth/china-eyes-black-swans-gray-rhinos-as-2018-growth-seen-slowing-to-6-5-6-8-percent-media-idUSKBN1FI0AY?il=0\n",
      "URL: https://www.reuters.com/article/us-china-economy-growth/china-eyes-black-swans-gray-rhinos-as-2018-growth-seen-slowing-to-6-5-6-8-percent-media-idUSKBN1FI0AY?il=0\n",
      "TITLE: China eyes black swans, gray rhinos as 2018 growth seen slowing to 6.5-6.8 percent - media\n",
      "BODY:\n",
      "BEIJING (Reuters) - China’s economic growth will likely slow to 6.5-6.8 percent this year, a senior official at the country’s top economic planner wrote in the Beijing Daily on Monday, while warning about the risks of “Black Swan” and “Gray Rhino” events.  Black swans, or unforeseen occurrences, and gray rhinos, or highly possible yet ignored threats, are likely to occur this year with adverse consequences, Fan Hengshan, vice secretary general of the National Development and Reform Commission (NDRC), wrote in a commentary in the state-controlled newspaper.  China’s economy grew 6.9 percent in 2017, the first annual acceleration since 2010. That pace easily beat the government’s 2017 target of around 6.5 percent, welcome news for policymakers looking to curb financial risks and cut corporate debt. “My personal opinion is that economic growth this year is very likely to exceed 6.5 percent, roughly between 6.5 and 6.8 percent,” Fan said.  Analysts polled by Reuters earlier this month also predicted a slowdown to 6.5 percent this year as government-led crackdowns on debt risks and factory pollution drag on overall activity. China will focus on fending off risks this year, particularly risks that will impact regions and cause systemic fluctuations, Fan said.  “To this end, we must remain highly vigilant and enhance our sense of urgency,” Fan said.  Earlier this month, China’s banking regulator chief told the official People’s Daily in an interview that a black swan event could threaten the country’s financial stability, adding that risks, while still manageable, are “complex and serious.” Reporting by Stella Qiu and Ryan Woo; Editing by Kim CoghillOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-australia-defence/australia-to-spend-3-1-billion-to-increase-stake-in-global-arms-exports-idUSKBN1FI08I?il=0\n",
      "URL: https://www.reuters.com/article/us-australia-defence/australia-to-spend-3-1-billion-to-increase-stake-in-global-arms-exports-idUSKBN1FI08I?il=0\n",
      "TITLE: Australia to spend $3.1 billion to increase stake in global arms exports\n",
      "BODY:\n",
      "SYDNEY (Reuters) - Prime Minister Malcolm Turnbull said on Monday Australian military equipment manufacturers will be offered government-backed loans as part of a A$3.8 billion ($3.1 billion) package to become one of the world’s top 10 defense exporters. Australia said in 2016 it would boost defense spending by A$30 billion by 2021, purchasing frigates, armored personnel carriers, strike fighter jets, drones and a fleet of new submarines - many of which would be built at home. The defense industry has struggled to obtain finance from traditional lenders that have been unwilling to fund the arms industry, so Australia has created a A$3.8 billion loan scheme for companies seeking finance to export military equipment.  “Australia is around the 20th largest exporter. Given the size of our defense budget we should be higher up the scale than that,” Turnbull told reporters in Sydney. “The goal is to get into the top 10,” he said. Christopher Pyne, the minister for the defense industry, said Australia would target sales to the United States, Canada, Britain and New Zealand. Australia’s annual defense budget was worth A$34.6 billion this year. The scheme is also meant to arrest a slide in Australia’s manufacturing sector and provide some support for its economy, which has been hampered by record-low wage growth. Australia saw a record number of jobs created in 2017 but its manufacturing sector has shrunk significantly following the end of domestic car manufacturing. Employment in manufacturing peaked in mid-1989 at roughly 1.17 million, or 15 percent of the entire workforce. That shrank to 877,000, or 7 percent, late last year.   Australia has seen a wave of new jobs but companies are not keen on paying employees more, leaving wage growth near record lows in an unwelcome drag on consumer spending and inflation. Australia’s expansion plans come amid increased global demand for military hardware, led by China and Middle East nations, prompting criticism of Canberra from aid agencies who argue Australia could make human rights violations worse if weapons were sold to the wrong buyers. Analysts said Australia would need to significantly expand sales beyond its traditional partners to have any chance of fulfilling its ambition. “There are possibilities, but I doubt U.S. interest especially will go beyond niche capabilities,” said Euan Graham, director of the international security program at Australian think tank the Lowy Institute. Reporting by Colin PackhamOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-afghanistan-blast/militants-attack-afghan-army-post-near-military-academy-in-capital-idUSKBN1FI07M?il=0\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "URL: https://www.reuters.com/article/us-afghanistan-blast/militants-attack-afghan-army-post-near-military-academy-in-capital-idUSKBN1FI07M?il=0\n",
      "TITLE: Militants attack Afghan army post near military academy in capital\n",
      "BODY:\n",
      "KABUL (Reuters) - At least four militants attacked an army outpost near one of Afghanistan’s main military academies on Monday and at least one soldier was killed and three wounded, a defense ministry official said. The attack in the western outskirts of the capital, Kabul, came two days after an ambulance bomb in the center of the city killed more than 100 people and just over a week after another attack on the Hotel Intercontinental killed more than 20. Both of those attacks were claimed by the Taliban. Ministry of Defence officials said the militants attacked the outpost near the well-defended Marshal Fahim military academy just before dawn. One of the attackers blew himself up, one had been killed and two were still fighting. One soldier had been killed and three were wounded, said one official, who declined to be identified. Ministry spokesman Dawlat Waziri said three soldiers had been wounded and the clash was still going on. Earlier, resident Mohammad Ehsan said he heard a series of explosions coming from the area beginning at around 5 a.m. and lasting for at least an hour.  Smaller blasts could still be heard at less frequent intervals. In October, a suicide attacker rammed a car full of explosives into a bus carrying cadets from the defense university, which is home to one of Afghanistan’s main officer training schools, killing 15 of them. Reporting by Omar Sobhani; Writing by James Mackenzie; Editing by Robert Birsel and Paul TaitOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-usa-election-kochs/fearing-democratic-wave-koch-network-to-spend-big-on-u-s-midterm-elections-idUSKBN1FI07H?il=0\n",
      "URL: https://www.reuters.com/article/us-usa-election-kochs/fearing-democratic-wave-koch-network-to-spend-big-on-u-s-midterm-elections-idUSKBN1FI07H?il=0\n",
      "TITLE: Fearing Democratic wave, Koch network to spend big on U.S. midterm elections\n",
      "BODY:\n",
      "INDIAN WELLS, Calif. (Reuters) - The conservative Koch network spent its annual donor conference celebrating policy victories under President Donald Trump such as the tax overhaul, but the elation was tinged with anxiety over November’s congressional elections that could pose a risk to its agenda. To that end, the network plans to spend what would be unprecedented sums for the Kochs to maintain Republican majorities in the U.S. Senate and House of Representatives, while trying to sell voters on the benefits of the newly passed tax package, according to network officials who briefed reporters on their strategy during the conference this weekend in Indian Wells, California. Historically, the party in power loses seats in congressional elections after a new president’s election. This year, Republican angst is compounded by Trump, whose tumultuous presidency helped galvanize Democratic and independent voters to go to the polls in special and state elections last year. “It’s going to be a very challenging environment,” said Tim Phillips, president of Americans for Prosperity, a grassroots political group that is part of the Koch network. “The left is energized. There’s no question about that.” The network is prepared to spend up to $400 million on the congressional races - a 60 percent increase from its investment in the 2016 election, officials said. Energy magnates Charles and David Koch have long been outsized players in Republican politics, but they never warmed to Trump during the 2016 campaign. With Trump in the White House, however, they have seen several policy goals realized, including the tax legislation and cutting federal regulations. But the Koch network still diverges from the Republican president on issues such as immigration and trade. The Kochs strongly support legislation that would protect “Dreamers” - people brought illegally to the United States as children - from deportation. Two Koch operatives, Daniel Garza and Jorge Lima, were at the White House on Friday to try to help broker a deal with Congress over the Dreamers. The network issued a statement on Friday disagreeing with a proposal in Trump’s immigration blueprint that would set new limits on legal immigration. On trade, Phillips and other Koch operatives are deeply concerned about the administration’s moves to impose tariffs on some imports and ardently support the North American Free Trade Agreement with Canada and Mexico, which is being renegotiated and that Trump has threatened to abandon. ‘GET OUT AND DEFINE YOURSELF’ Senator John Cornyn, the No. 2 Republican in the Senate, was an attendee at the Koch event, a measure of the network’s deep ties to conservatives in Congress and an example of the leverage it would lose should Democrats take control next year. Of the $400 million the network is looking to spend, $20 million will go toward promoting the tax law, which passed Congress in December and included big cuts in corporate tax rates along with tax reductions for many individuals. Polls at the time of passage showed Americans divided about the bill’s merits, but Phillips said he believed that would  change as voters see increased paychecks. Democrats condemned the tax measure as favoring corporations and the rich. Americans for Prosperity is positioned to be a ground-level force in the congressional elections. Its largest presence is in Florida, where Democratic Senator Bill Nelson may face a fight from Republican Governor Rick Scott. The group also has offices in Arizona, Nevada, and Wisconsin, among other states, all of which are expected to see highly competitive Senate races. Phillips said his group would likely stay out of Republican primaries. One variable hanging over the Kochs’ effort is Trump. Republican losses in a special U.S. Senate election in Alabama last month and a governor’s race in Virginia in November were attributed in part to discontent with the president. In 2010, Republicans seized on voter worries about Democratic President Barack Obama’s signature healthcare law, known as Obamacare, to capture the House and thwart Obama’s policy goals. For the Koch network, the challenge will be to avoid having every local race become a referendum on the president, which could propel a Democratic wave. James Davis, vice president of Freedom Partners, another Koch-backed policy group, said strong candidates should be able to differentiate themselves from Trump. “Get out there and define yourself and where you stand on the issues,” Davis said.  During one event, a donor from Nebraska, Gail Werner-Robertson, stood up and addressed Charles Koch directly, urging attendees to contribute more to the midterm effort. “We can’t lose the progress you all have fought so hard for,” she said.  Reporting by James Oliphant; Editing by Caren Bohan and Peter CooneyOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/uk-global-markets/asia-shares-extend-bull-run-dollar-crawls-off-lows-idUSKBN1FI01D?il=0\n",
      "URL: https://www.reuters.com/article/uk-global-markets/asia-shares-extend-bull-run-dollar-crawls-off-lows-idUSKBN1FI01D?il=0\n",
      "TITLE: Asia shares extend bull run, dollar crawls off lows\n",
      "BODY:\n",
      "SYDNEY (Reuters) - Asian shares extended their bull run on Monday amid upbeat corporate earnings and strong global economic growth, while the dollar tried to bounce even as the White House continued to complain of “unfair” trade practices by competitors. MSCI’s broadest index of Asia-Pacific shares outside Japan added 0.4 percent, aiming for a 12th straight session of gains. It is up 8 percent for the year so far. Japan’s Nikkei rose 0.5 percent as the yen eased a little, while South Korea notched a record. Hong Kong’s Hang Seng also rose 0.5 percent. It has been the best performer for the year with a rise of more than 11 percent, followed by Shanghai blue chips with gains of nearly 9 percent, though the latter dipped on Monday. Wall Street has likewise been on a tear. Just last week, the Dow rose 2.08 percent, the S&P; 500 2.22 percent and the Nasdaq 2.31 percent. Quarterly earnings growth for the S&P; 500 is estimated at 13.2 percent, according to Thomson Reuters data, up from 12 percent at the start of the year. Of the 133 companies in the index that have reported, almost 80 percent beat forecasts. Another 36 percent of the S&P; 500 is due to report this week including heavy hitters Apple, Alphabet, Facebook, Microsoft and Amazon. The rush to equities combined with the risk of faster global inflation, has been a major negative for sovereign bonds with yields rising across much of the developed world. Yields on U.S. two-year Treasuries have risen steadily to their highest since 2008 and are fully priced for a rate hike by the Federal Reserve in March. Ten-year yields broke above the range of the last week or so to reach 2.69 percent on Monday, levels last visited in mid-2014. The Fed holds its next meeting on Wednesday, the last for Chair Janet Yellen, and analysts suspect the statement will only cement expectations for a March move. WORDS MATTER     The inexorable increase in Treasury yields has not, however, been enough to rescue the U.S. dollar which sank to three-year lows last week as U.S. officials welcomed a weaker currency. President Donald Trump did try and walk some of that back late in the week but by then the damage had been done. Indeed, in an interview shown on Sunday, Trump threatened to confront the European Union over what he calls “very unfair” trade policy toward the U.S.. “‘Words’ in the world of FX do matter,” said Deutsche Bank strategist George Saravelos. “The U.S. is reengaging with a weak dollar policy similarly to the 1994-95 period.” This was happening while the sum of trade and investment flows into the United States was shrinking. The opposite was happening in the euro zone, where the German export engine was powering an ever-expanding current account surplus. “We continue to target $1.30 in EUR/USD for this year,” Saravelos concluded. The euro did run into a little profit-taking in Asia on Monday which nudged it to back to $1.2393 and away from a three-year peak of $1.2538 last week. The dollar was a fraction firmer on the yen at 108.89, but not far from a four-month trough of 108.28. Against a basket of major currencies, it edged up 0.2 percent to 89.281 having been at the lowest since late 2014. The dollar faces a bevy of U.S. economic reports this week including releases on inflation, manufacturing and payrolls. The currency’s decline has been a boon for many commodities, with gold making a 17-month top last week and last trading at $1,348.10 an ounce. Oil prices had reached their highest in three years and Brent crude futures were holding atop $70 at $70.40 a barrel. U.S. crude futures were up 18 cents at $66.32.  Editing by Shri Navaratnam and Kim CoghillOur Standards:The Thomson Reuters Trust Principles.\n",
      "GETTING https://www.reuters.com/article/us-japan-cryptocurrency/japan-to-punish-hacked-cryptocurrency-exchange-coincheck-on-monday-idUSKBN1FI06S?il=0\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-69-4ebc9b5cac68>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     52\u001b[0m \u001b[0mreuters\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mWebsite\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'Reuters'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'https://www.reuters.com'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'^(/article/)'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mFalse\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'h1'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m'div.StandardArticleBody_body_1gnLA'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     53\u001b[0m \u001b[0mcrawler\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mCrawler\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mreuters\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 54\u001b[0;31m \u001b[0mcrawler\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mcrawl\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m<ipython-input-69-4ebc9b5cac68>\u001b[0m in \u001b[0;36mcrawl\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     47\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mabsoluteUrl\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     48\u001b[0m                     \u001b[0mtargetPage\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m'{}{}'\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mformat\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mtargetPage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 49\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mparse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mtargetPage\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     50\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     51\u001b[0m                 \u001b[0;31m# self, name, url, targetPattern, absoluteUrl, titleTag, bodyTag\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-69-4ebc9b5cac68>\u001b[0m in \u001b[0;36mparse\u001b[0;34m(self, url)\u001b[0m\n\u001b[1;32m     27\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     28\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mparse\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 29\u001b[0;31m         \u001b[0mbs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mgetPage\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     30\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mbs\u001b[0m \u001b[0;32mis\u001b[0m \u001b[0;32mnot\u001b[0m \u001b[0;32mNone\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     31\u001b[0m             \u001b[0mtitle\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msafeGet\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mbs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msite\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mtitleTag\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<ipython-input-69-4ebc9b5cac68>\u001b[0m in \u001b[0;36mgetPage\u001b[0;34m(self, url)\u001b[0m\n\u001b[1;32m      9\u001b[0m         \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"GETTING \"\u001b[0m\u001b[0;34m+\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     10\u001b[0m         \u001b[0;32mtry\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 11\u001b[0;31m             \u001b[0mreq\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     12\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mrequests\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mexceptions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mRequestException\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     13\u001b[0m             \u001b[0mprint\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m\"Did not get page\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py\u001b[0m in \u001b[0;36mget\u001b[0;34m(url, params, **kwargs)\u001b[0m\n\u001b[1;32m     65\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     66\u001b[0m     \u001b[0mkwargs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0msetdefault\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'allow_redirects'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;32mTrue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 67\u001b[0;31m     \u001b[0;32mreturn\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'get'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mparams\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mparams\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     68\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     69\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/api.py\u001b[0m in \u001b[0;36mrequest\u001b[0;34m(method, url, **kwargs)\u001b[0m\n\u001b[1;32m     51\u001b[0m     \u001b[0;31m# cases, and look like a memory leak in others.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     52\u001b[0m     \u001b[0;32mwith\u001b[0m \u001b[0msessions\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mSession\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m \u001b[0;32mas\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 53\u001b[0;31m         \u001b[0;32mreturn\u001b[0m \u001b[0msession\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrequest\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mmethod\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0murl\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0murl\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     54\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     55\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36m__exit__\u001b[0;34m(self, *args)\u001b[0m\n\u001b[1;32m    348\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    349\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__exit__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 350\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    351\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    352\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mprepare_request\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/sessions.py\u001b[0m in \u001b[0;36mclose\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    647\u001b[0m         \u001b[0;34m\"\"\"Closes all adapters and as such the session\"\"\"\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    648\u001b[0m         \u001b[0;32mfor\u001b[0m \u001b[0mv\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0madapters\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mvalues\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 649\u001b[0;31m             \u001b[0mv\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    650\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    651\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmount\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mprefix\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0madapter\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/adapters.py\u001b[0m in \u001b[0;36mclose\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    268\u001b[0m         \u001b[0mconnections\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    269\u001b[0m         \"\"\"\n\u001b[0;32m--> 270\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpoolmanager\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    271\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    272\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mrequest_url\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mrequest\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mproxies\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/poolmanager.py\u001b[0m in \u001b[0;36mclear\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     98\u001b[0m         \u001b[0mre\u001b[0m\u001b[0;34m-\u001b[0m\u001b[0mused\u001b[0m \u001b[0mafter\u001b[0m \u001b[0mcompletion\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     99\u001b[0m         \"\"\"\n\u001b[0;32m--> 100\u001b[0;31m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mpools\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclear\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    101\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    102\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mconnection_from_host\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mhost\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mport\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mNone\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mscheme\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;34m'http'\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/_collections.py\u001b[0m in \u001b[0;36mclear\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m     92\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispose_func\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     93\u001b[0m             \u001b[0;32mfor\u001b[0m \u001b[0mvalue\u001b[0m \u001b[0;32min\u001b[0m \u001b[0mvalues\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m---> 94\u001b[0;31m                 \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mdispose_func\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mvalue\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m     95\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     96\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mkeys\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/poolmanager.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(p)\u001b[0m\n\u001b[1;32m     64\u001b[0m         \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mconnection_pool_kw\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mconnection_pool_kw\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     65\u001b[0m         self.pools = RecentlyUsedContainer(num_pools,\n\u001b[0;32m---> 66\u001b[0;31m                                            dispose_func=lambda p: p.close())\n\u001b[0m\u001b[1;32m     67\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m     68\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m__enter__\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/requests/packages/urllib3/connectionpool.py\u001b[0m in \u001b[0;36mclose\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    410\u001b[0m                 \u001b[0mconn\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mold_pool\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mget\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mblock\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0;32mFalse\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    411\u001b[0m                 \u001b[0;32mif\u001b[0m \u001b[0mconn\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 412\u001b[0;31m                     \u001b[0mconn\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mclose\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    413\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    414\u001b[0m         \u001b[0;32mexcept\u001b[0m \u001b[0mEmpty\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "\n",
    "class Crawler:\n",
    "    def __init__(self, site):\n",
    "        self.site = site\n",
    "        self.visited = []\n",
    "\n",
    "    def getPage(self, url):\n",
    "        try:\n",
    "            req = requests.get(url)\n",
    "        except requests.exceptions.RequestException:\n",
    "            return None\n",
    "        return BeautifulSoup(req.text, 'html.parser')\n",
    "\n",
    "    def safeGet(self, pageObj, selector):\n",
    "        selectedElems = pageObj.select(selector)\n",
    "        if selectedElems is not None and len(selectedElems) > 0:\n",
    "            return '\\n'.join([elem.get_text() for elem in selectedElems])\n",
    "        return ''\n",
    "\n",
    "    def parse(self, url):\n",
    "        bs = self.getPage(url)\n",
    "        if bs is not None:\n",
    "            title = self.safeGet(bs, self.site.titleTag)\n",
    "            body = self.safeGet(bs, self.site.bodyTag)\n",
    "            if title != '' and body != '':\n",
    "                content = Content(url, title, body)\n",
    "                content.print()\n",
    "\n",
    "    def crawl(self):\n",
    "        \"\"\"\n",
    "        Get pages from website home page\n",
    "        \"\"\"\n",
    "        bs = self.getPage(self.site.url)\n",
    "        targetPages = bs.findAll('a', href=re.compile(self.site.targetPattern))\n",
    "        for targetPage in targetPages:\n",
    "            targetPage = targetPage.attrs['href']\n",
    "            if targetPage not in self.visited:\n",
    "                self.visited.append(targetPage)\n",
    "                if not self.site.absoluteUrl:\n",
    "                    targetPage = '{}{}'.format(self.site.url, targetPage)\n",
    "                self.parse(targetPage)\n",
    "\n",
    "\n",
    "reuters = Website('Reuters', 'https://www.reuters.com', '^(/article/)',\n",
    "                  False, 'h1', 'div.StandardArticleBody_body_1gnLA')\n",
    "crawler = Crawler(reuters)\n",
    "crawler.crawl()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Crawling multiple page types"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Website:\n",
    "    \"\"\"Common base class for all articles/pages\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, bodyTag):\n",
    "        self.name = name\n",
    "        self.url = url\n",
    "        self.titleTag = titleTag\n",
    "        self.bodyTag = bodyTag\n",
    "        "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class Product(Website):\n",
    "    \"\"\"Contains information for scraping a product page\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, productNumber, price):\n",
    "        Website.__init__(self, name, url, TitleTag)\n",
    "        self.productNumberTag = productNumberTag\n",
    "        self.priceTag = priceTag\n",
    "\n",
    "class Article(Website):\n",
    "    \"\"\"Contains information for scraping an article page\"\"\"\n",
    "\n",
    "    def __init__(self, name, url, titleTag, bodyTag, dateTag):\n",
    "        Website.__init__(self, name, url, titleTag)\n",
    "        self.bodyTag = bodyTag\n",
    "        self.dateTag = dateTag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "\n",
    "def parsePage(url):\n",
    "    \n",
    "    if '/ideas/' in url:\n",
    "        \n",
    "\n",
    "oreilly = Website('O\\'Reilly', 'https://oreilly.com', 'h1' '')        "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}