practing-scraping.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import numpy as np\n",
    "import pandas as pd\n",
    "import urllib\n",
    "import bs4"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to test if a given page is found or not on the server."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE HTML PUBLIC \"-//W3C//DTD HTML 4.0 Transitional//EN\"\n",
      "\"http://www.w3.org/TR/REC-html40/transitional.dtd\">\n",
      "<html>\n",
      "<head>\n",
      "<meta http-equiv=\"Content-Type\" content=\"text/html; charset=utf-8\">\n",
      "<title>Beautiful Soup: We called him Tortoise because he taught us.</title>\n",
      "<link rev=\"made\" href=\"mailto:leonardr@segfault.org\">\n",
      "<link rel=\"stylesheet\" type=\"text/css\" href=\"/nb/themes/Default/nb.css\">\n",
      "<meta name=\"Description\" content=\"Beautiful Soup: a library designed for screen-scraping HTML and XML.\">\n",
      "<meta name=\"generator\" content=\"Markov Approximation 1.4 (module: leonardr)\">\n",
      "<meta name=\"author\" content=\"Leonard Richardson\">\n",
      "</head>\n",
      "<body bgcolor=\"white\" text=\"black\" link=\"blue\" vlink=\"660066\" alink=\"red\">\n",
      "<img align=\"right\" src=\"10.1.jpg\" width=\"250\"><br />\n",
      "\n",
      "<p>You didn't write that awful page. You're just trying to get some\n",
      "data out of it. Beautiful Soup is here to help. Since 2004, it's been\n",
      "saving programmers hours or days of work on quick-turnaround\n",
      "screen scraping projects.</p>\n",
      "\n",
      "<div align=\"center\">\n",
      "\n",
      "<a href=\"bs4/download/\"><h1>Beautiful Soup</h1></a>\n",
      "\n",
      "<p>\"A tremendous boon.\" -- Python411 Podcast</p>\n",
      "\n",
      "<p>[ <a href=\"#Download\">Download</a> | <a\n",
      "href=\"bs4/doc/\">Documentation</a> | <a href=\"#HallOfFame\">Hall of Fame</a> | <a href=\"https://code.launchpad.net/beautifulsoup\">Source</a> | <a href=\"https://bazaar.launchpad.net/%7Eleonardr/beautifulsoup/bs4/view/head:/NEWS.txt\">Changelog</a> | <a href=\"https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup\">Discussion group</a>  | <a href=\"zine/\">Zine</a> ]</p>\n",
      "\n",
      "<small>If Beautiful Soup has saved you a lot of time and money, one way to pay me back is to read <a href=\"zine/\"><i>Tool Safety</i></a>, a short zine I wrote about what I learned about software development from working on Beautiful Soup. Thanks!</small> </div>\n",
      "\n",
      "<p><i>If you have questions, send them to <a\n",
      "href=\"https://groups.google.com/forum/?fromgroups#!forum/beautifulsoup\">the discussion\n",
      "group</a>. If you find a bug, <a href=\"https://bugs.launchpad.net/beautifulsoup/\">file it</a>.</i></p>\n",
      "\n",
      "<p>Beautiful Soup is a Python library designed for quick turnaround\n",
      "projects like screen-scraping. Three features make it powerful:\n",
      "\n",
      "<ol>\n",
      "\n",
      "<li>Beautiful Soup provides a few simple methods and Pythonic idioms\n",
      "for navigating, searching, and modifying a parse tree: a toolkit for\n",
      "dissecting a document and extracting what you need. It doesn't take\n",
      "much code to write an application\n",
      "\n",
      "<li>Beautiful Soup automatically converts incoming documents to\n",
      "Unicode and outgoing documents to UTF-8. You don't have to think\n",
      "about encodings, unless the document doesn't specify an encoding and\n",
      "Beautiful Soup can't detect one. Then you just have to specify the\n",
      "original encoding.\n",
      "\n",
      "<li>Beautiful Soup sits on top of popular Python parsers like <a\n",
      "href=\"http://lxml.de/\">lxml</a> and <a\n",
      "href=\"http://code.google.com/p/html5lib/\">html5lib</a>, allowing you\n",
      "to try out different parsing strategies or trade speed for\n",
      "flexibility.\n",
      "\n",
      "</ol>\n",
      "\n",
      "<p>Beautiful Soup parses anything you give it, and does the tree\n",
      "traversal stuff for you. You can tell it \"Find all the links\", or\n",
      "\"Find all the links of class <tt>externalLink</tt>\", or \"Find all the\n",
      "links whose urls match \"foo.com\", or \"Find the table heading that's\n",
      "got bold text, then give me that text.\"\n",
      "\n",
      "<p>Valuable data that was once locked up in poorly-designed websites\n",
      "is now within your reach. Projects that would have taken hours take\n",
      "only minutes with Beautiful Soup.\n",
      "\n",
      "<p>Interested? <a href=\"bs4/doc/\">Read more.</a>\n",
      "\n",
      "<a name=\"Download\"><h2>Download Beautiful Soup</h2></a>\n",
      "\n",
      "<p>The current release is <a href=\"bs4/download/\">Beautiful Soup\n",
      "4.6.3</a> (August 12, 2018). You can install Beautiful Soup 4 with\n",
      "<code>pip install beautifulsoup4</code>.\n",
      "\n",
      "<p>In Debian and Ubuntu, Beautiful Soup is available as the\n",
      "<code>python-bs4</code> package (for Python 2) or the\n",
      "<code>python3-bs4</code> package (for Python 3). In Fedora it's\n",
      "available as the <code>python-beautifulsoup4</code> package.\n",
      "\n",
      "<p>Beautiful Soup is licensed under the MIT license, so you can also\n",
      "download the tarball, drop the <code>bs4/</code> directory into almost\n",
      "any Python application (or into your library path) and start using it\n",
      "immediately. (If you want to do this under Python 3, you will need to\n",
      "manually convert the code using <code>2to3</code>.)\n",
      "\n",
      "<p>Beautiful Soup 4 works on both Python 2 (2.7+) and Python 3.\n",
      "\n",
      "<h3>Beautiful Soup 3</h3>\n",
      "\n",
      "<p>Beautiful Soup 3 was the official release line of Beautiful Soup\n",
      "from May 2006 to March 2012. It is considered stable, and only\n",
      "critical security bugs will be fixed. <a\n",
      "href=\"http://www.crummy.com/software/BeautifulSoup/bs3/documentation.html\">Here's\n",
      "the Beautiful Soup 3 documentation.</a>\n",
      "\n",
      "<p>Beautiful Soup 3 works only under Python 2.x. It is licensed under\n",
      "the same license as Python itself.\n",
      "\n",
      "<p>The current release of Beautiful Soup 3 is <a\n",
      "href=\"download/3.x/BeautifulSoup-3.2.1.tar.gz\">3.2.1</a> (February 16,\n",
      "2012). You can install Beautiful Soup 3 with <code>pip install\n",
      "BeautifulSoup</code>. It's also available as\n",
      "<code>python-beautifulsoup</code> in Debian and Ubuntu, and as\n",
      "<code>python-BeautifulSoup</code> in Fedora.\n",
      "\n",
      "<p>You can also download the tarball and use\n",
      "<code>BeautifulSoup.py</code> in your project directly.\n",
      "\n",
      "\n",
      "<a name=\"HallOfFame\"><h2>Hall of Fame</h2></a>\n",
      "\n",
      "<p>Over the years, Beautiful Soup has been used in hundreds of\n",
      "different projects. There's no way I can list them all, but I want to\n",
      "highlight a few high-profile projects. Beautiful Soup isn't what makes\n",
      "these projects interesting, but it did make their completion easier:\n",
      "\n",
      "<ul>\n",
      "\n",
      "<li><a\n",
      " href=\"http://www.nytimes.com/2007/10/25/arts/design/25vide.html\">\"Movable\n",
      " Type\"</a>, a work of digital art on display in the lobby of the New\n",
      " York Times building, uses Beautiful Soup to scrape news feeds.\n",
      "\n",
      "<li>Reddit uses Beautiful Soup to <a\n",
      "href=\"https://github.com/reddit/reddit/blob/85f9cff3e2ab9bb8f19b96acd8da4ebacc079f04/r2/r2/lib/media.py\">parse\n",
      "a page that's been linked to and find a representative image</a>.\n",
      "\n",
      "<li>Alexander Harrowell uses Beautiful Soup to <a\n",
      " href=\"http://www.harrowell.org.uk/viktormap.html\">track the business\n",
      " activities</a> of an arms merchant.\n",
      "\n",
      "<li>The developers of Python itself used Beautiful Soup to <a\n",
      "href=\"http://svn.python.org/view/tracker/importer/\">migrate the Python\n",
      "bug tracker from Sourceforge to Roundup</a>.\n",
      "\n",
      "<li>The <a href=\"http://www2.ljworld.com/\">Lawrence Journal-World</a>\n",
      "uses Beautiful Soup to <A\n",
      "href=\"http://www.b-list.org/weblog/2010/nov/02/news-done-broke/\">gather\n",
      "statewide election results</a>.\n",
      "\n",
      "<li>The <a href=\"http://esrl.noaa.gov/gsd/fab/\">NOAA's Forecast\n",
      "Applications Branch</a> uses Beautiful Soup in <a\n",
      "href=\"http://laps.noaa.gov/topograbber/\">TopoGrabber</a>, a script for\n",
      "downloading \"high resolution USGS datasets.\"\n",
      "\n",
      "</ul>\n",
      "\n",
      "<p>If you've used Beautiful Soup in a project you'd like me to know\n",
      "about, please do send email to me or <a\n",
      "href=\"http://groups.google.com/group/beautifulsoup/\">the discussion\n",
      "group</a>.\n",
      "\n",
      "<h2>Development</h2>\n",
      "\n",
      "<p>Development happens at <a\n",
      "href=\"https://launchpad.net/beautifulsoup\">Launchpad</a>. You can <a\n",
      "href=\"https://code.launchpad.net/beautifulsoup/\">get the source\n",
      "code</a> or <a href=\"https://bugs.launchpad.net/beautifulsoup/\">file\n",
      "bugs</a>.<hr><table><tr><td valign=\"top\">\n",
      "<p>This document (<a href=\"/source/software/BeautifulSoup/index.bhtml\">source</a>) is part of Crummy, the webspace of <a href=\"/self/\">Leonard Richardson</a> (<a href=\"/self/contact.html\">contact information</a>). It was last modified on Sunday, August 12 2018, 16:48:15 Nowhere Standard Time and last built on Wednesday, October 03 2018, 01:00:01 Nowhere Standard Time.</p><p><table class=\"licenseText\"><tr><td><a href=\"http://creativecommons.org/licenses/by-sa/2.0/\"><img border=\"0\" src=\"/nb//resources/img/somerights20.jpg\"></a></td><td valign=\"top\">Crummy is &copy; 1996-2018 Leonard Richardson. Unless otherwise noted, all text licensed under a <a href=\"http://creativecommons.org/licenses/by-sa/2.0/\">Creative Commons License</a>.</td></tr></table></span><!--<rdf:RDF xmlns=\"http://web.resource.org/cc/\" xmlns:dc=\"http://purl.org/dc/elements/1.1/\" xmlns:rdf=\"http://www.w3.org/1999/02/22-rdf-syntax-ns#\"><Work rdf:about=\"http://www.crummy.com/\"><dc:title>Crummy: The Site</dc:title><dc:rights><Agent><dc:title>Crummy: the Site</dc:title></Agent></dc:rights><dc:format>text/html</dc:format><license rdf:resource=http://creativecommons.org/licenses/by-sa/2.0//></Work><License rdf:about=\"http://creativecommons.org/licenses/by-sa/2.0/\"></License></rdf:RDF>--></p></td><td valign=top><p><b>Document tree:</b>\n",
      "<dl><dd><a href=\"http://www.crummy.com/\">http://www.crummy.com/</a><dl><dd><a href=\"http://www.crummy.com/software/\">software/</a><dl><dd><a href=\"http://www.crummy.com/software/BeautifulSoup/\">BeautifulSoup/</a></dl>\n",
      "</dl>\n",
      "</dl>\n",
      "\n",
      "\n",
      "Site Search:\n",
      "\n",
      "<form method=\"get\" action=\"/search/\">\n",
      "        <input type=\"text\" name=\"q\" maxlength=\"255\" value=\"\"></input>\n",
      "        </form>\n",
      "        </td>\n",
      "\n",
      "</tr>\n",
      "\n",
      "</table>\n",
      "</body>\n",
      "</html>\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# found\n",
    "url = \"https://www.crummy.com/software/BeautifulSoup/\"\n",
    "\n",
    "# not found\n",
    "#url = \"http://abcxyz.com/\"\n",
    "\n",
    "try:\n",
    "    page = urllib.request.urlopen(url).read().decode('utf-8')\n",
    "except urllib.error.HTTPError as e:\n",
    "    print(\"HTTP error\")\n",
    "except urllib.error.URLError as e:\n",
    "    print(\"URL error\")\n",
    "else:\n",
    "    print(page)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to download and display the content of robot.txt for en.wikipedia.org"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "<html><body><p># robots.txt for http://www.wikipedia.org/ and friends\n",
       "#\n",
       "# Please note: There are a lot of pages on this site, and there are\n",
       "# some misbehaved spiders out there that go _way_ too fast. If you're\n",
       "# irresponsible, your access to the site may be blocked.\n",
       "#\n",
       "\n",
       "# Observed spamming large amounts of https://en.wikipedia.org/?curid=NNNNNN\n",
       "# and ignoring 429 ratelimit responses, claims to respect robots:\n",
       "# http://mj12bot.com/\n",
       "User-agent: MJ12bot\n",
       "Disallow: /\n",
       "\n",
       "# advertising-related bots:\n",
       "User-agent: Mediapartners-Google*\n",
       "Disallow: /\n",
       "\n",
       "# Wikipedia work bots:\n",
       "User-agent: IsraBot\n",
       "Disallow:\n",
       "\n",
       "User-agent: Orthogaffe\n",
       "Disallow:\n",
       "\n",
       "# Crawlers that are kind enough to obey, but which we'd rather not have\n",
       "# unless they're feeding search engines.\n",
       "User-agent: UbiCrawler\n",
       "Disallow: /\n",
       "\n",
       "User-agent: DOC\n",
       "Disallow: /\n",
       "\n",
       "User-agent: Zao\n",
       "Disallow: /\n",
       "\n",
       "# Some bots are known to be trouble, particularly those designed to copy\n",
       "# entire sites. Please obey robots.txt.\n",
       "User-agent: sitecheck.internetseer.com\n",
       "Disallow: /\n",
       "\n",
       "User-agent: Zealbot\n",
       "Disallow: /\n",
       "\n",
       "User-agent: MSIECrawler\n",
       "Disallow: /\n",
       "\n",
       "User-agent: SiteSnagger\n",
       "Disallow: /\n",
       "\n",
       "User-agent: WebStripper\n",
       "Disallow: /\n",
       "\n",
       "User-agent: WebCopier\n",
       "Disallow: /\n",
       "\n",
       "User-agent: Fetch\n",
       "Disallow: /\n",
       "\n",
       "User-agent: Offline Explorer\n",
       "Disallow: /\n",
       "\n",
       "User-agent: Teleport\n",
       "Disallow: /\n",
       "\n",
       "User-agent: TeleportPro\n",
       "Disallow: /\n",
       "\n",
       "User-agent: WebZIP\n",
       "Disallow: /\n",
       "\n",
       "User-agent: linko\n",
       "Disallow: /\n",
       "\n",
       "User-agent: HTTrack\n",
       "Disallow: /\n",
       "\n",
       "User-agent: Microsoft.URL.Control\n",
       "Disallow: /\n",
       "\n",
       "User-agent: Xenu\n",
       "Disallow: /\n",
       "\n",
       "User-agent: larbin\n",
       "Disallow: /\n",
       "\n",
       "User-agent: libwww\n",
       "Disallow: /\n",
       "\n",
       "User-agent: ZyBORG\n",
       "Disallow: /\n",
       "\n",
       "User-agent: Download Ninja\n",
       "Disallow: /\n",
       "\n",
       "# Misbehaving: requests much too fast:\n",
       "User-agent: fast\n",
       "Disallow: /\n",
       "\n",
       "#\n",
       "# Sorry, wget in its recursive mode is a frequent problem.\n",
       "# Please read the man page and use it properly; there is a\n",
       "# --wait option you can use to set the delay between hits,\n",
       "# for instance.\n",
       "#\n",
       "User-agent: wget\n",
       "Disallow: /\n",
       "\n",
       "#\n",
       "# The 'grub' distributed client has been *very* poorly behaved.\n",
       "#\n",
       "User-agent: grub-client\n",
       "Disallow: /\n",
       "\n",
       "#\n",
       "# Doesn't follow robots.txt anyway, but...\n",
       "#\n",
       "User-agent: k2spider\n",
       "Disallow: /\n",
       "\n",
       "#\n",
       "# Hits many times per second, not acceptable\n",
       "# http://www.nameprotect.com/botinfo.html\n",
       "User-agent: NPBot\n",
       "Disallow: /\n",
       "\n",
       "# A capture bot, downloads gazillions of pages with no public benefit\n",
       "# http://www.webreaper.net/\n",
       "User-agent: WebReaper\n",
       "Disallow: /\n",
       "\n",
       "# Wayback Machine: defaults and whether to index user-pages\n",
       "# FIXME: Complete the removal of this block, per T7582.\n",
       "# User-agent: archive.org_bot\n",
       "# Allow: /\n",
       "\n",
       "\n",
       "#\n",
       "# Friendly, low-speed bots are welcome viewing article pages, but not\n",
       "# dynamically-generated pages please.\n",
       "#\n",
       "# Inktomi's \"Slurp\" can read a minimum delay between hits; if your\n",
       "# bot supports such a thing using the 'Crawl-delay' or another\n",
       "# instruction, please let us know.\n",
       "#\n",
       "# There is a special exception for API mobileview to allow dynamic\n",
       "# mobile web &amp; app views to load section content.\n",
       "# These views aren't HTTP-cached but use parser cache aggressively\n",
       "# and don't expose special: pages etc.\n",
       "#\n",
       "# Another exception is for REST API documentation, located at\n",
       "# /api/rest_v1/?doc.\n",
       "#\n",
       "User-agent: *\n",
       "Allow: /w/api.php?action=mobileview&amp;\n",
       "Allow: /w/load.php?\n",
       "Allow: /api/rest_v1/?doc\n",
       "Disallow: /w/\n",
       "Disallow: /api/\n",
       "Disallow: /trap/\n",
       "Disallow: /wiki/Special:\n",
       "Disallow: /wiki/Spezial:\n",
       "Disallow: /wiki/Spesial:\n",
       "Disallow: /wiki/Special%3A\n",
       "Disallow: /wiki/Spezial%3A\n",
       "Disallow: /wiki/Spesial%3A\n",
       "\n",
       "#\n",
       "# ar:\n",
       "Disallow: /wiki/%D8%AE%D8%A7%D8%B5:Search\n",
       "Disallow: /wiki/%D8%AE%D8%A7%D8%B5%3ASearch\n",
       "#\n",
       "# dewiki:\n",
       "# T6937\n",
       "# sensible deletion and meta user discussion pages:\n",
       "Disallow: /wiki/Wikipedia:L%C3%B6schkandidaten/\n",
       "Disallow: /wiki/Wikipedia:Löschkandidaten/\n",
       "Disallow: /wiki/Wikipedia:Vandalensperrung/\n",
       "Disallow: /wiki/Wikipedia:Benutzersperrung/\n",
       "Disallow: /wiki/Wikipedia:Vermittlungsausschuss/\n",
       "Disallow: /wiki/Wikipedia:Administratoren/Probleme/\n",
       "Disallow: /wiki/Wikipedia:Adminkandidaturen/\n",
       "Disallow: /wiki/Wikipedia:Qualitätssicherung/\n",
       "Disallow: /wiki/Wikipedia:Qualit%C3%A4tssicherung/\n",
       "# 4937#5\n",
       "Disallow: /wiki/Wikipedia:Vandalismusmeldung/\n",
       "Disallow: /wiki/Wikipedia:Gesperrte_Lemmata/\n",
       "Disallow: /wiki/Wikipedia:Löschprüfung/\n",
       "Disallow: /wiki/Wikipedia:L%C3%B6schprüfung/\n",
       "Disallow: /wiki/Wikipedia:Administratoren/Notizen/\n",
       "Disallow: /wiki/Wikipedia:Schiedsgericht/Anfragen/\n",
       "Disallow: /wiki/Wikipedia:L%C3%B6schpr%C3%BCfung/\n",
       "# T14111\n",
       "Disallow: /wiki/Wikipedia:Checkuser/\n",
       "Disallow: /wiki/Wikipedia_Diskussion:Checkuser/\n",
       "Disallow: /wiki/Wikipedia_Diskussion:Adminkandidaturen/\n",
       "# T15961\n",
       "Disallow: /wiki/Wikipedia:Spam-Blacklist-Log\n",
       "Disallow: /wiki/Wikipedia%3ASpam-Blacklist-Log\n",
       "Disallow: /wiki/Wikipedia_Diskussion:Spam-Blacklist-Log\n",
       "Disallow: /wiki/Wikipedia_Diskussion%3ASpam-Blacklist-Log\n",
       "#\n",
       "# enwiki:\n",
       "# Folks get annoyed when VfD discussions end up the number 1 google hit for\n",
       "# their name. See T6776\n",
       "Disallow: /wiki/Wikipedia:Articles_for_deletion/\n",
       "Disallow: /wiki/Wikipedia%3AArticles_for_deletion/\n",
       "Disallow: /wiki/Wikipedia:Votes_for_deletion/\n",
       "Disallow: /wiki/Wikipedia%3AVotes_for_deletion/\n",
       "Disallow: /wiki/Wikipedia:Pages_for_deletion/\n",
       "Disallow: /wiki/Wikipedia%3APages_for_deletion/\n",
       "Disallow: /wiki/Wikipedia:Miscellany_for_deletion/\n",
       "Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion/\n",
       "Disallow: /wiki/Wikipedia:Miscellaneous_deletion/\n",
       "Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion/\n",
       "Disallow: /wiki/Wikipedia:Copyright_problems\n",
       "Disallow: /wiki/Wikipedia%3ACopyright_problems\n",
       "Disallow: /wiki/Wikipedia:Protected_titles/\n",
       "Disallow: /wiki/Wikipedia%3AProtected_titles/\n",
       "# T15398\n",
       "Disallow: /wiki/Wikipedia:WikiProject_Spam/\n",
       "Disallow: /wiki/Wikipedia%3AWikiProject_Spam/\n",
       "# T16075\n",
       "Disallow: /wiki/MediaWiki:Spam-blacklist\n",
       "Disallow: /wiki/MediaWiki%3ASpam-blacklist\n",
       "Disallow: /wiki/MediaWiki_talk:Spam-blacklist\n",
       "Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist\n",
       "# T13261\n",
       "Disallow: /wiki/Wikipedia:Requests_for_arbitration/\n",
       "Disallow: /wiki/Wikipedia%3ARequests_for_arbitration/\n",
       "Disallow: /wiki/Wikipedia:Requests_for_comment/\n",
       "Disallow: /wiki/Wikipedia%3ARequests_for_comment/\n",
       "Disallow: /wiki/Wikipedia:Requests_for_adminship/\n",
       "Disallow: /wiki/Wikipedia%3ARequests_for_adminship/\n",
       "# T12288\n",
       "Disallow: /wiki/Wikipedia_talk:Articles_for_deletion/\n",
       "Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion/\n",
       "Disallow: /wiki/Wikipedia_talk:Votes_for_deletion/\n",
       "Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion/\n",
       "Disallow: /wiki/Wikipedia_talk:Pages_for_deletion/\n",
       "Disallow: /wiki/Wikipedia_talk%3APages_for_deletion/\n",
       "Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion/\n",
       "Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion/\n",
       "Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion/\n",
       "Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion/\n",
       "# T16793\n",
       "Disallow: /wiki/Wikipedia:Changing_username\n",
       "Disallow: /wiki/Wikipedia%3AChanging_username\n",
       "Disallow: /wiki/Wikipedia:Changing_username/\n",
       "Disallow: /wiki/Wikipedia%3AChanging_username/\n",
       "Disallow: /wiki/Wikipedia_talk:Changing_username\n",
       "Disallow: /wiki/Wikipedia_talk%3AChanging_username\n",
       "Disallow: /wiki/Wikipedia_talk:Changing_username/\n",
       "Disallow: /wiki/Wikipedia_talk%3AChanging_username/\n",
       "#\n",
       "# eswiki:\n",
       "# T8746\n",
       "Disallow: /wiki/Wikipedia:Consultas_de_borrado/\n",
       "Disallow: /wiki/Wikipedia%3AConsultas_de_borrado/\n",
       "#\n",
       "# fiwiki:\n",
       "# T10695\n",
       "Disallow: /wiki/Wikipedia:Poistettavat_sivut\n",
       "Disallow: /wiki/K%C3%A4ytt%C3%A4j%C3%A4:\n",
       "Disallow: /wiki/Käyttäjä:\n",
       "Disallow: /wiki/Keskustelu_k%C3%A4ytt%C3%A4j%C3%A4st%C3%A4:\n",
       "Disallow: /wiki/Keskustelu_käyttäjästä:\n",
       "Disallow: /wiki/Wikipedia:Yll%C3%A4pit%C3%A4j%C3%A4t/\n",
       "Disallow: /wiki/Wikipedia:Ylläpitäjät/\n",
       "#\n",
       "# hewiki:\n",
       "Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93:Search\n",
       "Disallow: /wiki/%D7%9E%D7%99%D7%95%D7%97%D7%93%3ASearch\n",
       "#T11517\n",
       "Disallow: /wiki/ויקיפדיה:רשימת_מועמדים_למחיקה/\n",
       "Disallow: /wiki/ויקיפדיה%3Aרשימת_מועמדים_למחיקה/\n",
       "Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/\n",
       "Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A8%D7%A9%D7%99%D7%9E%D7%AA_%D7%9E%D7%95%D7%A2%D7%9E%D7%93%D7%99%D7%9D_%D7%9C%D7%9E%D7%97%D7%99%D7%A7%D7%94/\n",
       "Disallow: /wiki/ויקיפדיה:ערכים_לא_קיימים_ומוגנים\n",
       "Disallow: /wiki/ויקיפדיה%3Aערכים_לא_קיימים_ומוגנים\n",
       "Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D\n",
       "Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%A2%D7%A8%D7%9B%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D\n",
       "Disallow: /wiki/ויקיפדיה:דפים_לא_קיימים_ומוגנים\n",
       "Disallow: /wiki/ויקיפדיה%3Aדפים_לא_קיימים_ומוגנים\n",
       "Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94:%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D\n",
       "Disallow: /wiki/%D7%95%D7%99%D7%A7%D7%99%D7%A4%D7%93%D7%99%D7%94%3A%D7%93%D7%A4%D7%99%D7%9D_%D7%9C%D7%90_%D7%A7%D7%99%D7%99%D7%9E%D7%99%D7%9D_%D7%95%D7%9E%D7%95%D7%92%D7%A0%D7%99%D7%9D\n",
       "#\n",
       "# huwiki:\n",
       "Disallow: /wiki/Speci%C3%A1lis:Search\n",
       "Disallow: /wiki/Speci%C3%A1lis%3ASearch\n",
       "#\n",
       "# itwiki:\n",
       "# T7545\n",
       "Disallow: /wiki/Wikipedia:Pagine_da_cancellare\n",
       "Disallow: /wiki/Wikipedia%3APagine_da_cancellare\n",
       "Disallow: /wiki/Wikipedia:Utenti_problematici\n",
       "Disallow: /wiki/Wikipedia%3AUtenti_problematici\n",
       "Disallow: /wiki/Wikipedia:Vandalismi_in_corso\n",
       "Disallow: /wiki/Wikipedia%3AVandalismi_in_corso\n",
       "Disallow: /wiki/Wikipedia:Amministratori\n",
       "Disallow: /wiki/Wikipedia%3AAmministratori\n",
       "Disallow: /wiki/Wikipedia:Proposte_di_cancellazione_semplificata\n",
       "Disallow: /wiki/Wikipedia%3AProposte_di_cancellazione_semplificata\n",
       "Disallow: /wiki/Categoria:Da_cancellare_subito\n",
       "Disallow: /wiki/Categoria%3ADa_cancellare_subito\n",
       "Disallow: /wiki/Wikipedia:Sospette_violazioni_di_copyright\n",
       "Disallow: /wiki/Wikipedia%3ASospette_violazioni_di_copyright\n",
       "Disallow: /wiki/Categoria:Da_controllare_per_copyright\n",
       "Disallow: /wiki/Categoria%3ADa_controllare_per_copyright\n",
       "Disallow: /wiki/Progetto:Rimozione_contributi_sospetti\n",
       "Disallow: /wiki/Progetto%3ARimozione_contributi_sospetti\n",
       "Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright\n",
       "Disallow: /wiki/Categoria%3ADa_cancellare_subito_per_violazione_integrale_copyright\n",
       "Disallow: /wiki/Progetto:Cococo\n",
       "Disallow: /wiki/Progetto%3ACococo\n",
       "Disallow: /wiki/Discussioni_progetto:Cococo\n",
       "Disallow: /wiki/Discussioni_progetto%3ACococo\n",
       "#\n",
       "# jawiki\n",
       "Disallow: /wiki/%E7%89%B9%E5%88%A5:Search\n",
       "Disallow: /wiki/%E7%89%B9%E5%88%A5%3ASearch\n",
       "# T7239\n",
       "Disallow: /wiki/Wikipedia:%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/\n",
       "Disallow: /wiki/Wikipedia%3A%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC/\n",
       "Disallow: /wiki/Wikipedia:%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC\n",
       "Disallow: /wiki/Wikipedia%3A%E5%88%A9%E7%94%A8%E8%80%85%E3%83%9A%E3%83%BC%E3%82%B8%E3%81%AE%E5%89%8A%E9%99%A4%E4%BE%9D%E9%A0%BC\n",
       "# nowiki\n",
       "# T13432\n",
       "Disallow: /wiki/Bruker:\n",
       "Disallow: /wiki/Bruker%3A\n",
       "Disallow: /wiki/Brukerdiskusjon\n",
       "Disallow: /wiki/Wikipedia:Administratorer\n",
       "Disallow: /wiki/Wikipedia%3AAdministratorer\n",
       "Disallow: /wiki/Wikipedia-diskusjon:Administratorer\n",
       "Disallow: /wiki/Wikipedia-diskusjon%3AAdministratorer\n",
       "Disallow: /wiki/Wikipedia:Sletting\n",
       "Disallow: /wiki/Wikipedia%3ASletting\n",
       "Disallow: /wiki/Wikipedia-diskusjon:Sletting\n",
       "Disallow: /wiki/Wikipedia-diskusjon%3ASletting\n",
       "#\n",
       "# plwiki\n",
       "# T10067\n",
       "Disallow: /wiki/Wikipedia:Strony_do_usuni%C4%99cia\n",
       "Disallow: /wiki/Wikipedia%3AStrony_do_usuni%C4%99cia\n",
       "Disallow: /wiki/Wikipedia:Do_usuni%C4%99cia\n",
       "Disallow: /wiki/Wikipedia%3ADo_usuni%C4%99cia\n",
       "Disallow: /wiki/Wikipedia:SDU/\n",
       "Disallow: /wiki/Wikipedia%3ASDU/\n",
       "Disallow: /wiki/Wikipedia:Strony_podejrzane_o_naruszenie_praw_autorskich\n",
       "Disallow: /wiki/Wikipedia%3AStrony_podejrzane_o_naruszenie_praw_autorskich\n",
       "#\n",
       "# ptwiki:\n",
       "# T7394\n",
       "Disallow: /wiki/Wikipedia:Páginas_para_eliminar/\n",
       "Disallow: /wiki/Wikipedia:P%C3%A1ginas_para_eliminar/\n",
       "Disallow: /wiki/Wikipedia%3AP%C3%A1ginas_para_eliminar/\n",
       "Disallow: /wiki/Wikipedia_Discussão:Páginas_para_eliminar/\n",
       "Disallow: /wiki/Wikipedia_Discuss%C3%A3o:P%C3%A1ginas_para_eliminar/\n",
       "Disallow: /wiki/Wikipedia_Discuss%C3%A3o%3AP%C3%A1ginas_para_eliminar/\n",
       "#\n",
       "# rowiki:\n",
       "# T14546\n",
       "Disallow: /wiki/Wikipedia:Pagini_de_%C5%9Fters\n",
       "Disallow: /wiki/Wikipedia%3APagini_de_%C5%9Fters\n",
       "Disallow: /wiki/Discu%C5%A3ie_Wikipedia:Pagini_de_%C5%9Fters\n",
       "Disallow: /wiki/Discu%C5%A3ie_Wikipedia%3APagini_de_%C5%9Fters\n",
       "#\n",
       "# ruwiki:\n",
       "Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5:Search\n",
       "Disallow: /wiki/%D0%A1%D0%BF%D0%B5%D1%86%D0%B8%D0%B0%D0%BB%D1%8C%D0%BD%D1%8B%D0%B5%3ASearch\n",
       "#\n",
       "# svwiki:\n",
       "# T12229\n",
       "Disallow: /wiki/Wikipedia%3ASidor_f%C3%B6reslagna_f%C3%B6r_radering\n",
       "Disallow: /wiki/Wikipedia:Sidor_f%C3%B6reslagna_f%C3%B6r_radering\n",
       "Disallow: /wiki/Wikipedia:Sidor_föreslagna_för_radering\n",
       "Disallow: /wiki/Användare\n",
       "Disallow: /wiki/Anv%C3%A4ndare\n",
       "Disallow: /wiki/Användardiskussion\n",
       "Disallow: /wiki/Anv%C3%A4ndardiskussion\n",
       "Disallow: /wiki/Wikipedia:Skyddade_sidnamn\n",
       "Disallow: /wiki/Wikipedia%3ASkyddade_sidnamn\n",
       "# T13291\n",
       "Disallow: /wiki/Wikipedia:Sidor_som_bör_raderas\n",
       "Disallow: /wiki/Wikipedia:Sidor_som_b%C3%B6r_raderas\n",
       "Disallow: /wiki/Wikipedia%3ASidor_som_b%C3%B6r_raderas\n",
       "#\n",
       "# zhwiki:\n",
       "# T7104\n",
       "Disallow: /wiki/Wikipedia:删除投票/侵权\n",
       "Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8/%E4%BE%B5%E6%9D%83\n",
       "Disallow: /wiki/Wikipedia:删除投票和请求\n",
       "Disallow: /wiki/Wikipedia:%E5%88%A0%E9%99%A4%E6%8A%95%E7%A5%A8%E5%92%8C%E8%AF%B7%E6%B1%82\n",
       "Disallow: /wiki/Category:快速删除候选\n",
       "Disallow: /wiki/Category:%E5%BF%AB%E9%80%9F%E5%88%A0%E9%99%A4%E5%80%99%E9%80%89\n",
       "Disallow: /wiki/Category:维基百科需要翻译的文章\n",
       "Disallow: /wiki/Category:%E7%BB%B4%E5%9F%BA%E7%99%BE%E7%A7%91%E9%9C%80%E8%A6%81%E7%BF%BB%E8%AF%91%E7%9A%84%E6%96%87%E7%AB%A0\n",
       "#\n",
       "# sister projects\n",
       "#\n",
       "# enwikinews:\n",
       "# T7340\n",
       "Disallow: /wiki/Portal:Prepared_stories/\n",
       "Disallow: /wiki/Portal%3APrepared_stories/\n",
       "#\n",
       "# itwikinews\n",
       "# T11138\n",
       "Disallow: /wiki/Wikinotizie:Richieste_di_cancellazione\n",
       "Disallow: /wiki/Wikinotizie:Sospette_violazioni_di_copyright\n",
       "Disallow: /wiki/Categoria:Da_cancellare_subito\n",
       "Disallow: /wiki/Categoria:Da_cancellare_subito_per_violazione_integrale_copyright\n",
       "Disallow: /wiki/Wikinotizie:Storie_in_preparazione\n",
       "#\n",
       "# enwikiquote:\n",
       "# T17095\n",
       "Disallow: /wiki/Wikiquote:Votes_for_deletion/\n",
       "Disallow: /wiki/Wikiquote%3AVotes_for_deletion/\n",
       "Disallow: /wiki/Wikiquote_talk:Votes_for_deletion/\n",
       "Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion/\n",
       "Disallow: /wiki/Wikiquote:Votes_for_deletion_archive/\n",
       "Disallow: /wiki/Wikiquote%3AVotes_for_deletion_archive/\n",
       "Disallow: /wiki/Wikiquote_talk:Votes_for_deletion_archive/\n",
       "Disallow: /wiki/Wikiquote_talk%3AVotes_for_deletion_archive/\n",
       "#\n",
       "# enwikibooks\n",
       "Disallow: /wiki/Wikibooks:Votes_for_deletion\n",
       "#\n",
       "# working...\n",
       "Disallow: /wiki/Fundraising_2007/comments\n",
       "#\n",
       "#\n",
       "#\n",
       "#----------------------------------------------------------#\n",
       "#\n",
       "#\n",
       "#\n",
       " # <!-- Please do not remove the space at the start of this line, it breaks the rendering.  http://www.robotstxt.org/orig.html says spaces before comments are OK. --></p><pre>\n",
       "#\n",
       "# Localisable part of robots.txt for en.wikipedia.org\n",
       "#\n",
       "# Edit at https://en.wikipedia.org/w/index.php?title=MediaWiki:Robots.txt&amp;action=edit\n",
       "# Don't add newlines here. All rules set here are active for every user-agent.\n",
       "#\n",
       "# Please check any changes using a syntax validator such as http://tool.motoricerca.info/robots-checker.phtml\n",
       "# Enter https://en.wikipedia.org/robots.txt as the URL to check.\n",
       "#\n",
       "# https://bugzilla.wikimedia.org/show_bug.cgi?id=14075\n",
       "Disallow: /wiki/MediaWiki:Spam-blacklist\n",
       "Disallow: /wiki/MediaWiki%3ASpam-blacklist\n",
       "Disallow: /wiki/MediaWiki_talk:Spam-blacklist\n",
       "Disallow: /wiki/MediaWiki_talk%3ASpam-blacklist\n",
       "Disallow: /wiki/Wikipedia:WikiProject_Spam\n",
       "Disallow: /wiki/Wikipedia_talk:WikiProject_Spam\n",
       "#\n",
       "# Folks get annoyed when XfD discussions end up the number 1 google hit for\n",
       "# their name. \n",
       "# https://phabricator.wikimedia.org/T16075\n",
       "Disallow: /wiki/Wikipedia:Articles_for_deletion\n",
       "Disallow: /wiki/Wikipedia%3AArticles_for_deletion\n",
       "Disallow: /wiki/Wikipedia:Votes_for_deletion\n",
       "Disallow: /wiki/Wikipedia%3AVotes_for_deletion\n",
       "Disallow: /wiki/Wikipedia:Pages_for_deletion\n",
       "Disallow: /wiki/Wikipedia%3APages_for_deletion\n",
       "Disallow: /wiki/Wikipedia:Miscellany_for_deletion\n",
       "Disallow: /wiki/Wikipedia%3AMiscellany_for_deletion\n",
       "Disallow: /wiki/Wikipedia:Miscellaneous_deletion\n",
       "Disallow: /wiki/Wikipedia%3AMiscellaneous_deletion\n",
       "Disallow: /wiki/Wikipedia:Categories_for_discussion\n",
       "Disallow: /wiki/Wikipedia%3ACategories_for_discussion\n",
       "Disallow: /wiki/Wikipedia:Templates_for_deletion\n",
       "Disallow: /wiki/Wikipedia%3ATemplates_for_deletion\n",
       "Disallow: /wiki/Wikipedia:Redirects_for_discussion\n",
       "Disallow: /wiki/Wikipedia%3ARedirects_for_discussion\n",
       "Disallow: /wiki/Wikipedia:Deletion_review\n",
       "Disallow: /wiki/Wikipedia%3ADeletion_review\n",
       "Disallow: /wiki/Wikipedia:WikiProject_Deletion_sorting\n",
       "Disallow: /wiki/Wikipedia%3AWikiProject_Deletion_sorting\n",
       "Disallow: /wiki/Wikipedia:Files_for_deletion\n",
       "Disallow: /wiki/Wikipedia%3AFiles_for_deletion\n",
       "Disallow: /wiki/Wikipedia:Files_for_discussion\n",
       "Disallow: /wiki/Wikipedia%3AFiles_for_discussion\n",
       "Disallow: /wiki/Wikipedia:Possibly_unfree_files\n",
       "Disallow: /wiki/Wikipedia%3APossibly_unfree_files\n",
       "#\n",
       "# https://phabricator.wikimedia.org/T12288\n",
       "Disallow: /wiki/Wikipedia_talk:Articles_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk%3AArticles_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk:Votes_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk%3AVotes_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk:Pages_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk%3APages_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk:Miscellany_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk%3AMiscellany_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk:Miscellaneous_deletion\n",
       "Disallow: /wiki/Wikipedia_talk%3AMiscellaneous_deletion\n",
       "Disallow: /wiki/Wikipedia_talk:Templates_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk%3ATemplates_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk:Categories_for_discussion\n",
       "Disallow: /wiki/Wikipedia_talk%3ACategories_for_discussion\n",
       "Disallow: /wiki/Wikipedia_talk:Deletion_review\n",
       "Disallow: /wiki/Wikipedia_talk%3ADeletion_review\n",
       "Disallow: /wiki/Wikipedia_talk:WikiProject_Deletion_sorting\n",
       "Disallow: /wiki/Wikipedia_talk%3AWikiProject_Deletion_sorting\n",
       "Disallow: /wiki/Wikipedia_talk:Files_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk%3AFiles_for_deletion\n",
       "Disallow: /wiki/Wikipedia_talk:Files_for_discussion\n",
       "Disallow: /wiki/Wikipedia_talk%3AFiles_for_discussion\n",
       "Disallow: /wiki/Wikipedia_talk:Possibly_unfree_files\n",
       "Disallow: /wiki/Wikipedia_talk%3APossibly_unfree_files\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Copyright_problems\n",
       "Disallow: /wiki/Wikipedia%3ACopyright_problems\n",
       "Disallow: /wiki/Wikipedia_talk:Copyright_problems\n",
       "Disallow: /wiki/Wikipedia_talk%3ACopyright_problems\n",
       "Disallow: /wiki/Wikipedia:Suspected_copyright_violations\n",
       "Disallow: /wiki/Wikipedia%3ASuspected_copyright_violations\n",
       "Disallow: /wiki/Wikipedia_talk:Suspected_copyright_violations\n",
       "Disallow: /wiki/Wikipedia_talk%3ASuspected_copyright_violations\n",
       "Disallow: /wiki/Wikipedia:Contributor_copyright_investigations\n",
       "Disallow: /wiki/Wikipedia%3AContributor_copyright_investigations\n",
       "Disallow: /wiki/Wikipedia:Contributor_copyright_investigations\n",
       "Disallow: /wiki/Wikipedia%3AContributor_copyright_investigations\n",
       "Disallow: /wiki/Wikipedia_talk:Contributor_copyright_investigations\n",
       "Disallow: /wiki/Wikipedia_talk%3AContributor_copyright_investigations\n",
       "Disallow: /wiki/Wikipedia_talk:Contributor_copyright_investigations\n",
       "Disallow: /wiki/Wikipedia_talk%3AContributor_copyright_investigations\n",
       "Disallow: /wiki/Wikipedia:Protected_titles\n",
       "Disallow: /wiki/Wikipedia%3AProtected_titles\n",
       "Disallow: /wiki/Wikipedia_talk:Protected_titles\n",
       "Disallow: /wiki/Wikipedia_talk%3AProtected_titles\n",
       "Disallow: /wiki/Wikipedia:Articles_for_creation\n",
       "Disallow: /wiki/Wikipedia%3AArticles_for_creation\n",
       "Disallow: /wiki/Wikipedia_talk:Articles_for_creation\n",
       "Disallow: /wiki/Wikipedia_talk%3AArticles_for_creation\n",
       "Disallow: /wiki/Wikipedia_talk:Article_wizard\n",
       "Disallow: /wiki/Wikipedia_talk%3AArticle_wizard\n",
       "#\n",
       "# https://phabricator.wikimedia.org/T13261\n",
       "Disallow: /wiki/Wikipedia:Requests_for_arbitration\n",
       "Disallow: /wiki/Wikipedia%3ARequests_for_arbitration\n",
       "Disallow: /wiki/Wikipedia_talk:Requests_for_arbitration\n",
       "Disallow: /wiki/Wikipedia_talk%3ARequests_for_arbitration\n",
       "Disallow: /wiki/Wikipedia:Requests_for_comment\n",
       "Disallow: /wiki/Wikipedia%3ARequests_for_comment\n",
       "Disallow: /wiki/Wikipedia_talk:Requests_for_comment\n",
       "Disallow: /wiki/Wikipedia_talk%3ARequests_for_comment\n",
       "Disallow: /wiki/Wikipedia:Requests_for_adminship\n",
       "Disallow: /wiki/Wikipedia%3ARequests_for_adminship\n",
       "Disallow: /wiki/Wikipedia_talk:Requests_for_adminship\n",
       "Disallow: /wiki/Wikipedia_talk%3ARequests_for_adminship\n",
       "#\n",
       "# https://phabricator.wikimedia.org/T14111\n",
       "Disallow: /wiki/Wikipedia:Requests_for_checkuser\n",
       "Disallow: /wiki/Wikipedia%3ARequests_for_checkuser\n",
       "Disallow: /wiki/Wikipedia_talk:Requests_for_checkuser\n",
       "Disallow: /wiki/Wikipedia_talk%3ARequests_for_checkuser\n",
       "#\n",
       "# https://phabricator.wikimedia.org/T15398\n",
       "Disallow: /wiki/Wikipedia:WikiProject_Spam\n",
       "Disallow: /wiki/Wikipedia%3AWikiProject_Spam\n",
       "#\n",
       "# https://phabricator.wikimedia.org/T16793\n",
       "Disallow: /wiki/Wikipedia:Changing_username\n",
       "Disallow: /wiki/Wikipedia%3AChanging_username\n",
       "Disallow: /wiki/Wikipedia:Changing_username\n",
       "Disallow: /wiki/Wikipedia%3AChanging_username\n",
       "Disallow: /wiki/Wikipedia_talk:Changing_username\n",
       "Disallow: /wiki/Wikipedia_talk%3AChanging_username\n",
       "Disallow: /wiki/Wikipedia_talk:Changing_username\n",
       "Disallow: /wiki/Wikipedia_talk%3AChanging_username\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Administrators%27_noticeboard\n",
       "Disallow: /wiki/Wikipedia%3AAdministrators%27_noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk:Administrators%27_noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk%3AAdministrators%27_noticeboard\n",
       "Disallow: /wiki/Wikipedia:Community_sanction_noticeboard\n",
       "Disallow: /wiki/Wikipedia%3ACommunity_sanction_noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk:Community_sanction_noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk%3ACommunity_sanction_noticeboard\n",
       "Disallow: /wiki/Wikipedia:Bureaucrats%27_noticeboard\n",
       "Disallow: /wiki/Wikipedia%3ABureaucrats%27_noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk:Bureaucrats%27_noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk%3ABureaucrats%27_noticeboard\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Sockpuppet_investigations\n",
       "Disallow: /wiki/Wikipedia%3ASockpuppet_investigations\n",
       "Disallow: /wiki/Wikipedia_talk:Sockpuppet_investigations\n",
       "Disallow: /wiki/Wikipedia_talk%3ASockpuppet_investigations\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Neutral_point_of_view/Noticeboard\n",
       "Disallow: /wiki/Wikipedia%3ANeutral_point_of_view/Noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk:Neutral_point_of_view/Noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk%3ANeutral_point_of_view/Noticeboard\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:No_original_research/noticeboard\n",
       "Disallow: /wiki/Wikipedia%3ANo_original_research/noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk:No_original_research/noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk%3ANo_original_research/noticeboard\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Fringe_theories/Noticeboard\n",
       "Disallow: /wiki/Wikipedia%3AFringe_theories/Noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk:Fringe_theories/Noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk%3AFringe_theories/Noticeboard\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Conflict_of_interest/Noticeboard\n",
       "Disallow: /wiki/Wikipedia%3AConflict_of_interest/Noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk:Conflict_of_interest/Noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk%3AConflict_of_interest/Noticeboard\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Long-term_abuse\n",
       "Disallow: /wiki/Wikipedia%3ALong-term_abuse\n",
       "Disallow: /wiki/Wikipedia_talk:Long-term_abuse\n",
       "Disallow: /wiki/Wikipedia_talk%3ALong-term_abuse\n",
       "Disallow: /wiki/Wikipedia:Long_term_abuse\n",
       "Disallow: /wiki/Wikipedia%3ALong_term_abuse\n",
       "Disallow: /wiki/Wikipedia_talk:Long_term_abuse\n",
       "Disallow: /wiki/Wikipedia_talk%3ALong_term_abuse\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Wikiquette_assistance\n",
       "Disallow: /wiki/Wikipedia%3AWikiquette_assistance\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Abuse_reports\n",
       "Disallow: /wiki/Wikipedia%3AAbuse_reports\n",
       "Disallow: /wiki/Wikipedia_talk:Abuse_reports\n",
       "Disallow: /wiki/Wikipedia_talk%3AAbuse_reports\n",
       "Disallow: /wiki/Wikipedia:Abuse_response\n",
       "Disallow: /wiki/Wikipedia%3AAbuse_response\n",
       "Disallow: /wiki/Wikipedia_talk:Abuse_response\n",
       "Disallow: /wiki/Wikipedia_talk%3AAbuse_response\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Reliable_sources/Noticeboard\n",
       "Disallow: /wiki/Wikipedia%3AReliable_sources/Noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk:Reliable_sources/Noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk%3AReliable_sources/Noticeboard\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Suspected_sock_puppets\n",
       "Disallow: /wiki/Wikipedia%3ASuspected_sock_puppets\n",
       "Disallow: /wiki/Wikipedia_talk:Suspected_sock_puppets\n",
       "Disallow: /wiki/Wikipedia_talk%3ASuspected_sock_puppets\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Biographies_of_living_persons/Noticeboard\n",
       "Disallow: /wiki/Wikipedia%3ABiographies_of_living_persons/Noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk:Biographies_of_living_persons/Noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk%3ABiographies_of_living_persons/Noticeboard\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Content_noticeboard\n",
       "Disallow: /wiki/Wikipedia%3AContent_noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk:Content_noticeboard\n",
       "Disallow: /wiki/Wikipedia_talk%3AContent_noticeboard\n",
       "#\n",
       "Disallow: /wiki/Template:Editnotices\n",
       "Disallow: /wiki/Template%3AEditnotices\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Arbitration\n",
       "Disallow: /wiki/Wikipedia%3AArbitration\n",
       "Disallow: /wiki/Wikipedia_talk:Arbitration\n",
       "Disallow: /wiki/Wikipedia_talk%3AArbitration\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Arbitration_Committee\n",
       "Disallow: /wiki/Wikipedia%3AArbitration_Committee\n",
       "Disallow: /wiki/Wikipedia_talk:Arbitration_Committee\n",
       "Disallow: /wiki/Wikipedia_talk%3AArbitration_Committee\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Arbitration_Committee_Elections\n",
       "Disallow: /wiki/Wikipedia%3AArbitration_Committee_Elections\n",
       "Disallow: /wiki/Wikipedia_talk:Arbitration_Committee_Elections\n",
       "Disallow: /wiki/Wikipedia_talk%3AArbitration_Committee_Elections\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Mediation_Committee\n",
       "Disallow: /wiki/Wikipedia%3AMediation_Committee\n",
       "Disallow: /wiki/Wikipedia_talk:Mediation_Committee\n",
       "Disallow: /wiki/Wikipedia_talk%3AMediation_Committee\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Mediation_Cabal/Cases\n",
       "Disallow: /wiki/Wikipedia%3AMediation_Cabal/Cases\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Requests_for_bureaucratship\n",
       "Disallow: /wiki/Wikipedia%3ARequests_for_bureaucratship\n",
       "Disallow: /wiki/Wikipedia_talk:Requests_for_bureaucratship\n",
       "Disallow: /wiki/Wikipedia_talk%3ARequests_for_bureaucratship\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Administrator_review\n",
       "Disallow: /wiki/Wikipedia%3AAdministrator_review\n",
       "Disallow: /wiki/Wikipedia_talk:Administrator_review\n",
       "Disallow: /wiki/Wikipedia_talk%3AAdministrator_review\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Editor_review\n",
       "Disallow: /wiki/Wikipedia%3AEditor_review\n",
       "Disallow: /wiki/Wikipedia_talk:Editor_review\n",
       "Disallow: /wiki/Wikipedia_talk%3AEditor_review\n",
       "#\n",
       "Disallow: /wiki/Wikipedia:Article_Incubator\n",
       "Disallow: /wiki/Wikipedia%3AArticle_Incubator\n",
       "Disallow: /wiki/Wikipedia_talk:Article_Incubator\n",
       "Disallow: /wiki/Wikipedia_talk%3AArticle_Incubator\n",
       "#\n",
       "Disallow: /wiki/Category:Noindexed_pages\n",
       "Disallow: /wiki/Category%3ANoindexed_pages\n",
       "#\n",
       "# </pre></body></html>"
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = \"https://en.wikipedia.org/robots.txt\"\n",
    "\n",
    "source = urllib.request.urlopen(url)\n",
    "\n",
    "soup = bs4.BeautifulSoup(source, 'lxml')\n",
    "soup"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to get the number of datasets currently listed on data.gov."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['302,946 datasets']"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = \"https://www.data.gov/\"\n",
    "\n",
    "page = urllib.request.urlopen(url)\n",
    "soup = bs4.BeautifulSoup(page, 'lxml')\n",
    "\n",
    "small_tags = soup.find('small')\n",
    "num_datasets = small_tags.find('a').contents\n",
    "num_datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to display the name of the most recently added dataset on data.gov."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['Region 7 Laboratory Information Management System']"
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = \"https://catalog.data.gov/dataset?q=&sort=metadata_created+desc\"\n",
    "\n",
    "source = urllib.request.urlopen(url)\n",
    "soup = bs4.BeautifulSoup(source, 'lxml')\n",
    "\n",
    "h3_heading = soup.find('h3')\n",
    "\n",
    "fetch_recent = h3_heading.find('a').contents\n",
    "fetch_recent"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to extract h1 tag from example.com"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 56,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<h1>Example Domain</h1>\n"
     ]
    }
   ],
   "source": [
    "url = \"http://example.com/\"\n",
    "\n",
    "source = urllib.request.urlopen(url)\n",
    "soup = bs4.BeautifulSoup(source, 'lxml')\n",
    "\n",
    "print(soup.h1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to extract and display all the header tags from [here](https://en.wikipedia.org/wiki/Main_Page)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 65,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['<h1 class=\"firstHeading\" id=\"firstHeading\" lang=\"en\">Main Page</h1>',\n",
       " '<h2 id=\"mp-tfa-h2\" style=\"margin:0.5em; background:#cef2e0; font-family:inherit; font-size:120%; font-weight:bold; border:1px solid #a3bfb1; color:#000; padding:0.2em 0.4em;\"><span id=\"From_today.27s_featured_article\"></span><span class=\"mw-headline\" id=\"From_today\\'s_featured_article\">From today\\'s featured article</span></h2>',\n",
       " '<h2 id=\"mp-dyk-h2\" style=\"clear:both; margin:0.5em; background:#cef2e0; font-family:inherit; font-size:120%; font-weight:bold; border:1px solid #a3bfb1; color:#000; padding:0.2em 0.4em;\"><span class=\"mw-headline\" id=\"Did_you_know...\">Did you know...</span></h2>',\n",
       " '<h2 id=\"mp-itn-h2\" style=\"margin:0.5em; background:#cedff2; font-family:inherit; font-size:120%; font-weight:bold; border:1px solid #a3b0bf; color:#000; padding:0.2em 0.4em;\"><span class=\"mw-headline\" id=\"In_the_news\">In the news</span></h2>',\n",
       " '<h2 id=\"mp-otd-h2\" style=\"clear:both; margin:0.5em; background:#cedff2; font-family:inherit; font-size:120%; font-weight:bold; border:1px solid #a3b0bf; color:#000; padding:0.2em 0.4em;\"><span class=\"mw-headline\" id=\"On_this_day\">On this day</span></h2>',\n",
       " '<h2 id=\"mp-tfp-h2\" style=\"margin:0.5em; background:#ddcef2; font-family:inherit; font-size:120%; font-weight:bold; border:1px solid #afa3bf; color:#000; padding:0.2em 0.4em\"><span id=\"Today.27s_featured_picture\"></span><span class=\"mw-headline\" id=\"Today\\'s_featured_picture\">Today\\'s featured picture</span></h2>',\n",
       " '<h2 id=\"mp-other\" style=\"margin:0.5em; background:#eeeeee; border:1px solid #ddd; color:#222; padding:0.2em 0.4em; font-size:120%; font-weight:bold; font-family:inherit;\"><span class=\"mw-headline\" id=\"Other_areas_of_Wikipedia\">Other areas of Wikipedia</span></h2>',\n",
       " '<h2 id=\"mp-sister\" style=\"margin:0.5em; background:#eeeeee; border:1px solid #ddd; color:#222; padding:0.2em 0.4em; font-size:120%; font-weight:bold; font-family:inherit;\"><span id=\"Wikipedia.27s_sister_projects\"></span><span class=\"mw-headline\" id=\"Wikipedia\\'s_sister_projects\">Wikipedia\\'s sister projects</span></h2>',\n",
       " '<h2 id=\"mp-lang\" style=\"margin:0.5em; background:#efefef; border:1px solid #ddd; color:#222; padding:0.2em 0.4em; font-size:120%; font-weight:bold; font-family:inherit;\"><span class=\"mw-headline\" id=\"Wikipedia_languages\">Wikipedia languages</span></h2>',\n",
       " '<h2>Navigation menu</h2>',\n",
       " '<h3 id=\"p-personal-label\">Personal tools</h3>',\n",
       " '<h3 id=\"p-namespaces-label\">Namespaces</h3>',\n",
       " '<h3 id=\"p-variants-label\">\\n<span>Variants</span>\\n</h3>',\n",
       " '<h3 id=\"p-views-label\">Views</h3>',\n",
       " '<h3 id=\"p-cactions-label\"><span>More</span></h3>',\n",
       " '<h3>\\n<label for=\"searchInput\">Search</label>\\n</h3>',\n",
       " '<h3 id=\"p-navigation-label\">Navigation</h3>',\n",
       " '<h3 id=\"p-interaction-label\">Interaction</h3>',\n",
       " '<h3 id=\"p-tb-label\">Tools</h3>',\n",
       " '<h3 id=\"p-coll-print_export-label\">Print/export</h3>',\n",
       " '<h3 id=\"p-wikibase-otherprojects-label\">In other projects</h3>',\n",
       " '<h3 id=\"p-lang-label\">Languages</h3>']"
      ]
     },
     "execution_count": 65,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = \"https://en.wikipedia.org/wiki/Main_Page\"\n",
    "\n",
    "page = urllib.request.urlopen(url)\n",
    "soup = bs4.BeautifulSoup(page, 'lxml')\n",
    "\n",
    "headers = ['h1', 'h2', 'h3', 'h4', 'h5', 'h6']\n",
    "\n",
    "all_headers = soup.findAll(headers)\n",
    "\n",
    "[\"\".join(str(header)) for header in all_headers]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to extract and display all the image links from [here](https://en.wikipedia.org/wiki/Peter_Jeffrey_(RAAF_officer%29)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 89,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['//upload.wikimedia.org/wikipedia/commons/thumb/a/af/NlaJeffrey1942-43.jpg/220px-NlaJeffrey1942-43.jpg',\n",
       " '//upload.wikimedia.org/wikipedia/commons/thumb/c/c5/008315JeffreyTurnbull1941.jpg/260px-008315JeffreyTurnbull1941.jpg',\n",
       " '//upload.wikimedia.org/wikipedia/commons/e/ea/021807CameronJeffrey1941.jpg',\n",
       " '//upload.wikimedia.org/wikipedia/commons/thumb/9/92/AC0072JeffreyTruscottKittyhawks1942.jpg/280px-AC0072JeffreyTruscottKittyhawks1942.jpg',\n",
       " '//upload.wikimedia.org/wikipedia/commons/thumb/2/26/VIC1689Jeffrey1945.jpg/280px-VIC1689Jeffrey1945.jpg']"
      ]
     },
     "execution_count": 89,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import re\n",
    "\n",
    "url = \"https://en.wikipedia.org/wiki/Peter_Jeffrey_(RAAF_officer)\"\n",
    "\n",
    "source = urllib.request.urlopen(url)\n",
    "soup = bs4.BeautifulSoup(source, 'lxml')\n",
    "\n",
    "all_images = soup.findAll('img', attrs = {'src': re.compile('.jpg')})\n",
    "\n",
    "[image['src'] for image in all_images]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to get 90 days of visits broken down by browser for all sites on data.gov using [json](https://analytics.usa.gov/data/live/browsers.json). "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 111,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{'Chrome': 1203235445, 'Safari': 775739626, 'Internet Explorer': 299122430, 'Firefox': 126520887, 'Edge': 112544875, 'Samsung Internet': 59587258, 'Android Webview': 23956003, 'Safari (in-app)': 24562643, 'Amazon Silk': 8171301, 'Opera': 7488451, 'Opera Mini': 3292658, 'UC Browser': 1853153, 'Android Browser': 1139210, 'YaBrowser': 520468, 'Mozilla Compatible Agent': 423482, 'BlackBerry': 234050, 'Puffin': 314303, 'Coc Coc': 202558, 'SeaMonkey': 206362, 'Maxthon': 156059, 'Mozilla': 154471, 'UCWEB': 37873, 'Nintendo Browser': 23047, 'Amazon.com': 63252, 'Carousel': 26134, 'Nokia Browser': 12025, 'MRCHROME': 14848, 'ios-app': 2890, 'Playstation 3': 12064, 'ThousandEyes': 56881, 'BestBuy': 21213, 'Iron': 17640, 'Playstation Vita Browser': 7637, 'osee2unifiedRelease': 7615, 'Uzbl': 8398, 'Google-Test2': 6606, 'AdobeAIR': 7876, 'Nintendo 3DS Browser': 3299, 'Netscape': 3088, 'NetFront': 3076, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-TOUCHID-6.4.0-Mozilla': 927, 'IE with Chrome Frame': 3875, 'YE': 2410, 'Android Runtime': 4343, 'Outlook-iOS': 1776, 'Mercari_d': 9075, '(not set)': 24472, 'Seznam': 4114, 'DDG-Android-3.1.1': 1798, 'InteriaFakty': 18, 'iPhone': 2186, 'Lunascape': 1586, 'Safari Mozilla': 2724, 'LG-B470': 701, 'BrowserNG': 1730, '+Simple Browser': 2285, 'DirectorInsight Autosourcing www.directorinsight.com': 1156, 'Sephora 18.4.1': 650, 'Bluebeam Revu Browser - cef version: 57.0.0.0': 2204, 'CareDroidMedications': 368, 'Browser': 72, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-FACEID-6.4.0-Mozilla': 121, 'Optional': 133, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-6.4.0-Mozilla': 99, 'com.seekingalpha.webwrapper': 2006, 'Grailed': 897, 'DDG-Android-3.1.0': 1053, 'Caremark iPhone': 89, 'ScooperBot': 324, 'Web Science and Digital Libraries Group': 10149, '\"Mozilla': 6471, 'LeapPadPlatinum': 47505, 'LeapPad3Explorer': 5047, 'LeapPadUltimate': 51968, 'LiveSlides': 426, 'Snowball iPhone 2.5.0': 13, 'Android ExpediaBookings': 50, 'OPS_WATCHER': 4383, 'UP.Browser': 157, 'DoCoMo': 101, 'BlackBerry8330': 67, 'BlackBerry8320': 66, 'Cricket-A200': 67, 'YelpWebView': 28, 'Vodafone': 72, 'AU-MIC': 34, 'BenQ-CF61': 34, 'BlackBerry7100i': 36, 'BlackBerry7250': 34, 'BlackBerry8110': 34, 'BlackBerry8130': 34, 'BlackBerry8310': 33, 'BlackBerry8703e': 34, 'BlackBerry8820': 34, 'BlackBerry8830': 34, 'BlackBerry9000': 34, 'BlackBerry9630': 35, 'BlackBerry9650': 34, 'BlackBerry9700': 45, 'Cricket-A310': 34, 'Cricket-A410': 34, 'HTC_HD2_T8585': 17, 'HTC_Touch_HD_T8282': 17, 'J-PHONE': 33, 'MAUI': 17, 'OPENWAVE': 17, 'OPWV-SDK': 17, 'PDXGW': 33, 'ReqwirelessWeb': 34, 'SEC-SGH600': 34, 'SEC-SGHD410': 34, 'SIE-ME45': 33, 'SIE-S55': 34, 'SIE-S65': 34, 'SonyEricssonP1i': 17, 'SonyEricssonT200': 34, 'UPG1': 17, 'Xiino': 34, 'iTunes': 99, 'Astel': 33, 'BlackBerry7130e': 33, 'SEC-SGHE900': 16, 'WM5': 16, 'NokiaE52-1': 74, 'Emb': 1815, 'Blazer': 152, 'Nokia7610': 68, 'Nokia2700c-2': 51, 'WebPro': 67, 'Dolfin': 34, 'LGE-LG290C': 68, 'Links': 34, 'MobileExplorer': 34, 'Nokia3120Classic': 34, 'Nokia3200': 34, 'Nokia3510i': 34, 'Nokia3650': 34, 'Nokia5130c-2': 34, 'Nokia5140': 34, 'Nokia6230': 34, 'Nokia6600': 34, 'Nokia6630': 34, 'Nokia6680': 34, 'Nokia6800': 34, 'Nokia7250': 34, 'Nokia7250I': 34, 'Nokia8310': 34, 'NokiaN-Gage': 34, 'NokiaN80-3': 34, 'NokiaN90-1': 34, 'NokiaX3-02': 34, 'Profile': 34, 'Nokia2760': 33, 'NokiaN70-1': 33, 'SAGEM-myX5-2': 33, 'Palm680': 32, 'HTC-P4600': 34, 'HTC-ST7377': 17, 'HTC_Dream': 17, 'HTC_Smart_F3188': 17, 'HTC_Smart_F3188 Mozilla': 17, 'HTC_Touch_Pro_T7272': 17, 'LG': 33, 'LG-CT810': 34, 'LG-GS290': 33, 'LG-GT400': 33, 'LG-LG260': 17, 'LG-LX550': 17, 'LGE-MX380': 34, 'LGE-VM510': 17, 'MOT': 17, 'MOT-A-1C': 34, 'MOT-COOL0': 34, 'MOT-E398': 34, 'MOT-EX431G': 34, 'MOT-V3i': 34, 'MOT-V3r': 34, 'MOT-V600': 34, 'MOT-V620': 34, 'MOT-V9mm': 34, 'Modzilla': 34, 'Nokia6212 classic': 17, 'Nokia6300': 17, 'NokiaC3-00': 17, 'POLARIS': 34, 'PalmCentro': 17, 'PalmSource': 17, 'PantechP7040': 17, 'SAMSUNG-GT-S3653W': 17, 'SAMSUNG-S8003': 17, 'SAMSUNG-SGH-i900': 17, 'SCH-U350': 17, 'XV6850': 17, 'htc_touch_pro2_t7373': 17, 'mot-V3': 34, 'nook': 17, 'nook browser': 17, 'sam-r380 UP.Browser': 17, 'samr810 Netfront': 17, 'samsung-gt-s5620': 17, 'MOT-L6': 33, 'Nokia6212': 16, 'Opera 6.0[en]Nokia': 16, 'sam-r350 UP.Browser': 16, 'Nichrome': 81, 'EudoraWeb': 34, 'SAMSUNG-SGH-D900': 34, 'mwb-db-client Opera': 34, 'sonyericssonk800i': 34, 'Roku': 33, 'ActiveVideo H5 HbbTV': 17, 'Apple TV': 17, 'HTC_HD2_T8585 Opera': 17, 'HTC_Touch_HD_T8282 Mozilla': 17, 'HTC_Touch_Pro_T7272 Opera': 17, 'LG-BDP Linux': 17, 'LG-LG260 POLARIS-LG260': 17, 'LG-LX550 AU-MIC-LX550': 17, 'LGE-VM510 NetFront': 17, 'MOT 24.1 _': 17, 'OPENWAVE UNTRUSTED': 17, 'OPWV-SDK UP.Browser': 17, 'Playstation Portable': 17, 'Presto': 17, 'SCH-A950+UP.Browser': 17, 'SEC-SGHE600 UP.Link': 17, 'SHARP-TQ-GX10i': 17, 'Samsung-SPHM800 AU-MIC-M800': 17, 'SonyEricssonK700i': 17, 'SonyEricssonT280i': 17, 'TELECA-': 17, 'WM5 PIE': 17, 'htc_touch_pro2_t7373 opera': 17, 'iTunes-AppleTV': 17, 'samsung sgh-e900': 17, '\"Opera': 16, 'MAUI WAP Browser': 16, 'NOKIAN95': 16, 'SonyEricssonK608i': 16, 'XV6850 Opera': 16, 'DDG-Android-3.0.14': 123, 'stash-invest-android': 67, 'QZONEJSSDK': 31, 'Snowball iPhone 2.6.0': 36, 'MagentaNews': 36372, '--ignore-ssl-errors=true': 109, 'NIH': 65, 'AncestryAndroid': 11, 'NNLM': 62, 'Airbnb': 35, 'NokiaC7-00': 45, 'SalesforceMobileSDK': 46, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-TOUCHID-6.6.0-Mozilla': 1380, 'Windowshop': 32, 'Empty': 62, 'Sephora 18.5': 1268, 'chrome-Selenium': 21, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-FACEID-6.6.0-Mozilla': 98, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-6.6.0-Mozilla': 260, 'Digg': 175, 'liip': 58, 'StatusCake_Pagespeed_Indev': 16205, 'CBC': 256, 'Android': 15, 'Snowball iPhone 2.7.0': 48, 'LeapPadUltra': 25, 'iPad': 127, 'Konqueror': 51, 'Snowball iPhone 2.8.0': 130, 'SureFox': 11, 'None': 845, 'One Drop Web Agent': 12, 'MSIE 11.0': 73, 'RTLNieuws': 72, 'Camino': 359, 'ELinks': 29, 'CoolNovo': 12, 'Papers': 38, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-TOUCHID-6.8.0-Mozilla': 1650, 'Snowball iPhone 2.9.0': 24, 'HRB-MOBILE-IOS-PHONE-TAXES-TOUCHID-7.7.1-Mozilla': 87, 'Sephora 18.6': 1282, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-FACEID-6.8.0-Mozilla': 80, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-6.8.0-Mozilla': 141, 'Opera 9.4': 11, 'Infowars Android': 569, 'google.com': 20, '__undefined__': 31, 'WHY DO I HEAR YOUR AGENTS ON THE OTHER END OF THESE BUGS': 153, 'YOUR LET 49 PEOPLE DIE IN THE PULSE NIGHTCLUB SHOOTING!!': 14, 'YOUR AGENTS LET THE PULSE NIGHT CLUB ATTACK HAPPEN!': 386140, 'GeenStijlAndroidApp': 15, 'DEA ORLANDO LET 49 PEOPLE DIE IN PULSE NIGHT CLUB ATTACK!!!': 30, '{Mozilla': 4854, '{Opera': 1620, 'SpeedaNewsPicks': 18, 'GetWellNetwork-Release': 17, 'LinkKiosk': 274, 'Sephora 18.6.1': 1260, 'PlayFreeBrowser': 14, '[FBAN': 117, 'PuppeteerAgent': 30, 'EduFileViewer-Android': 11, 'Snowball iPhone 2.11.0': 12, 'pa11y': 27, 'Pulsepoint XT3 web scraper': 58, 'random': 19, 'Snowball iPhone 2.11.1': 14, 'iGotcha': 22, 'ArcGISRuntime-iOS': 12, 'Chromeless 1.3.0': 11, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-TOUCHID-7.0.0-Mozilla': 215, 'HRB-MOBILE-IOS-PHONE-MYBLOCK-FACEID-7.0.0-Mozilla': 11, 'DESKTOP': 43, 'Sephora 18.7': 46}\n"
     ]
    }
   ],
   "source": [
    "import json\n",
    "import requests\n",
    "\n",
    "url = \"https://analytics.usa.gov/data/live/browsers.json\"\n",
    "\n",
    "page = requests.get(url)\n",
    "\n",
    "json_file = page.json()\n",
    "\n",
    "print(json_file['totals']['browser'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to that retrieves an arbitary Wikipedia page of \"Python\" and creates a list of links on that page."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[None,\n",
       " '/wiki/Wikipedia:Good_articles',\n",
       " '/wiki/Wikipedia:Protection_policy#semi',\n",
       " '#mw-head',\n",
       " '#p-search',\n",
       " '/wiki/File:Python_logo_and_wordmark.svg',\n",
       " '/wiki/Programming_paradigm',\n",
       " '/wiki/Object-oriented_programming',\n",
       " '/wiki/Imperative_programming',\n",
       " '/wiki/Functional_programming',\n",
       " '/wiki/Procedural_programming',\n",
       " '/wiki/Reflective_programming',\n",
       " '/wiki/Software_design',\n",
       " '/wiki/Guido_van_Rossum',\n",
       " '/wiki/Software_developer',\n",
       " '/wiki/Python_Software_Foundation',\n",
       " '#cite_note-guttag-1',\n",
       " '/wiki/Software_release_life_cycle',\n",
       " '#cite_note-2',\n",
       " '#cite_note-3',\n",
       " '/wiki/Type_system',\n",
       " '/wiki/Duck_typing',\n",
       " '/wiki/Dynamic_typing',\n",
       " '/wiki/Strong_typing',\n",
       " '/wiki/Gradual_typing',\n",
       " '#cite_note-4',\n",
       " '/wiki/Software_license',\n",
       " '/wiki/Python_Software_Foundation_License',\n",
       " '/wiki/Filename_extension',\n",
       " '#cite_note-5',\n",
       " '#cite_note-6',\n",
       " 'https://www.python.org/',\n",
       " '/wiki/Programming_language_implementation',\n",
       " '/wiki/CPython',\n",
       " '/wiki/IronPython',\n",
       " '/wiki/Jython',\n",
       " '/wiki/MicroPython',\n",
       " '/wiki/Numba',\n",
       " '/wiki/PyPy',\n",
       " '/wiki/Stackless_Python',\n",
       " '/wiki/Dialect_(computing)',\n",
       " '/wiki/Cython',\n",
       " '/wiki/RPython',\n",
       " '/wiki/ABC_(programming_language)',\n",
       " '#cite_note-faq-created-7',\n",
       " '/wiki/ALGOL_68',\n",
       " '#cite_note-98-interview-8',\n",
       " '/wiki/APL_(programming_language)',\n",
       " '#cite_note-9',\n",
       " '/wiki/C_(programming_language)',\n",
       " '#cite_note-AutoNT-1-10',\n",
       " '/wiki/C%2B%2B',\n",
       " '#cite_note-classmix-11',\n",
       " '/wiki/CLU_(programming_language)',\n",
       " '#cite_note-effbot-call-by-object-12',\n",
       " '/wiki/Dylan_(programming_language)',\n",
       " '#cite_note-AutoNT-2-13',\n",
       " '/wiki/Haskell_(programming_language)',\n",
       " '#cite_note-AutoNT-3-14',\n",
       " '/wiki/Icon_(programming_language)',\n",
       " '#cite_note-AutoNT-4-15',\n",
       " '/wiki/Java_(programming_language)',\n",
       " '#cite_note-AutoNT-5-16',\n",
       " '/wiki/Lisp_(programming_language)',\n",
       " '#cite_note-AutoNT-6-17',\n",
       " '/wiki/Modula-3',\n",
       " '#cite_note-classmix-11',\n",
       " '/wiki/Perl',\n",
       " '/wiki/Standard_ML',\n",
       " '#cite_note-18',\n",
       " '/wiki/Boo_(programming_language)',\n",
       " '/wiki/Cobra_(programming_language)',\n",
       " '/wiki/CoffeeScript',\n",
       " '#cite_note-19',\n",
       " '/wiki/D_(programming_language)',\n",
       " '/wiki/F_Sharp_(programming_language)',\n",
       " '/wiki/Falcon_(programming_language)',\n",
       " '/wiki/Genie_(programming_language)',\n",
       " '#cite_note-20',\n",
       " '/wiki/Go_(programming_language)',\n",
       " '/wiki/Apache_Groovy',\n",
       " '/wiki/JavaScript',\n",
       " '#cite_note-21',\n",
       " '#cite_note-22',\n",
       " '/wiki/Julia_(programming_language)',\n",
       " '#cite_note-Julia-23',\n",
       " '/wiki/Nim_(programming_language)',\n",
       " '/wiki/Ring_(programming_language)',\n",
       " '#cite_note-The_Ring_programming_language_and_other_languages-24',\n",
       " '/wiki/Ruby_(programming_language)',\n",
       " '#cite_note-bini-25',\n",
       " '/wiki/Swift_(programming_language)',\n",
       " '#cite_note-lattner2014-26',\n",
       " '/wiki/File:Wikibooks-logo-en-noslogan.svg',\n",
       " 'https://en.wikibooks.org/wiki/Python_Programming',\n",
       " '/wiki/Interpreted_language',\n",
       " '/wiki/High-level_programming_language',\n",
       " '/wiki/General-purpose_programming_language',\n",
       " '/wiki/Guido_van_Rossum',\n",
       " '/wiki/Code_readability',\n",
       " '/wiki/Significant_whitespace',\n",
       " '#cite_note-AutoNT-7-27',\n",
       " '#cite_note-28',\n",
       " '#cite_note-29',\n",
       " '/wiki/Dynamic_type',\n",
       " '/wiki/Memory_management',\n",
       " '/wiki/Programming_paradigm',\n",
       " '/wiki/Object-oriented_programming',\n",
       " '/wiki/Imperative_programming',\n",
       " '/wiki/Functional_programming',\n",
       " '/wiki/Procedural_programming',\n",
       " '/wiki/Standard_library',\n",
       " '#cite_note-About-30',\n",
       " '/wiki/Operating_system',\n",
       " '/wiki/CPython',\n",
       " '/wiki/Reference_implementation',\n",
       " '/wiki/Open_source',\n",
       " '#cite_note-31',\n",
       " '/wiki/Python_Software_Foundation',\n",
       " '#History',\n",
       " '#Features_and_philosophy',\n",
       " '#Syntax_and_semantics',\n",
       " '#Indentation',\n",
       " '#Statements_and_control_flow',\n",
       " '#Expressions',\n",
       " '#Methods',\n",
       " '#Typing',\n",
       " '#Mathematics',\n",
       " '#Libraries',\n",
       " '#Terminology:_Modules_and_packages',\n",
       " '#Development_environments',\n",
       " '#Implementations',\n",
       " '#Reference_implementation',\n",
       " '#Other_implementations',\n",
       " '#Unsupported_implementations',\n",
       " '#Cross-compilers_to_other_languages',\n",
       " '#Performance',\n",
       " '#Development',\n",
       " '#Naming',\n",
       " '#API_documentation_generators',\n",
       " '#Uses',\n",
       " '#Languages_influenced_by_Python',\n",
       " '#See_also',\n",
       " '#References',\n",
       " '#Further_reading',\n",
       " '#External_links',\n",
       " '/wiki/File:Guido_van_Rossum_OSCON_2006.jpg',\n",
       " '/wiki/File:Guido_van_Rossum_OSCON_2006.jpg',\n",
       " '/wiki/Guido_van_Rossum',\n",
       " '/wiki/History_of_Python',\n",
       " '#cite_note-venners-interview-pt-1-32',\n",
       " '#cite_note-timeline-of-python-33',\n",
       " '/wiki/Guido_van_Rossum',\n",
       " '/wiki/Centrum_Wiskunde_%26_Informatica',\n",
       " '/wiki/Netherlands',\n",
       " '/wiki/ABC_(programming_language)',\n",
       " '/wiki/SETL',\n",
       " '#cite_note-AutoNT-12-34',\n",
       " '/wiki/Exception_handling',\n",
       " '/wiki/Amoeba_(operating_system)',\n",
       " '#cite_note-faq-created-7',\n",
       " '/wiki/Benevolent_Dictator_For_Life',\n",
       " '#cite_note-lj-bdfl-resignation-35',\n",
       " '#cite_note-36',\n",
       " '/wiki/ABC_(programming_language)',\n",
       " '/wiki/Unix',\n",
       " '/wiki/C_(programming_language)',\n",
       " '/wiki/Hacker_(programmer_subculture)',\n",
       " '/wiki/Monty_Python%27s_Flying_Circus',\n",
       " '/wiki/Cycle_detection',\n",
       " '/wiki/Garbage_collection_(computer_science)',\n",
       " '/wiki/Unicode',\n",
       " '#cite_note-newin-2.0-37',\n",
       " '/wiki/Backward_compatibility',\n",
       " '#cite_note-3.0-release-38',\n",
       " '/wiki/Backporting',\n",
       " '#cite_note-pep-3000-39',\n",
       " '#cite_note-40',\n",
       " '/wiki/End-of-life_(product)',\n",
       " '#cite_note-41',\n",
       " '#cite_note-42',\n",
       " '/wiki/Go_(programming_language)',\n",
       " '/wiki/Transcompiler',\n",
       " '#cite_note-43',\n",
       " '/wiki/Multi-paradigm_programming_language',\n",
       " '/wiki/Object-oriented_programming',\n",
       " '/wiki/Structured_programming',\n",
       " '/wiki/Functional_programming',\n",
       " '/wiki/Aspect-oriented_programming',\n",
       " '/wiki/Metaprogramming',\n",
       " '#cite_note-AutoNT-13-44',\n",
       " '/wiki/Metaobject',\n",
       " '#cite_note-AutoNT-14-45',\n",
       " '/wiki/Design_by_contract',\n",
       " '#cite_note-AutoNT-15-46',\n",
       " '#cite_note-AutoNT-16-47',\n",
       " '/wiki/Logic_programming',\n",
       " '#cite_note-AutoNT-17-48',\n",
       " '/wiki/Dynamic_typing',\n",
       " '/wiki/Reference_counting',\n",
       " '/wiki/Memory_management',\n",
       " '/wiki/Name_resolution_(programming_languages)',\n",
       " '/wiki/Late_binding',\n",
       " '/wiki/Functional_programming',\n",
       " '/wiki/Lisp_(programming_language)',\n",
       " '/wiki/List_comprehension',\n",
       " '/wiki/Associative_array',\n",
       " '/wiki/Generator_(computer_programming)',\n",
       " '#cite_note-AutoNT-59-49',\n",
       " '/wiki/Haskell_(programming_language)',\n",
       " '/wiki/Standard_ML',\n",
       " '#cite_note-AutoNT-18-50',\n",
       " '/wiki/Zen_of_Python',\n",
       " '/wiki/Aphorism',\n",
       " '#cite_note-PEP20-51',\n",
       " '/wiki/ABC_(programming_language)',\n",
       " '#cite_note-venners-interview-pt-1-32',\n",
       " '/wiki/Perl',\n",
       " '/wiki/Alex_Martelli',\n",
       " '#cite_note-AutoNT-19-52',\n",
       " '/wiki/There_is_more_than_one_way_to_do_it',\n",
       " '#cite_note-PEP20-51',\n",
       " '/wiki/Premature_optimization',\n",
       " '#cite_note-AutoNT-20-53',\n",
       " '/wiki/PyPy',\n",
       " '/wiki/Just-in-time_compilation',\n",
       " '/wiki/Cython',\n",
       " '/wiki/Monty_Python',\n",
       " '#cite_note-AutoNT-24-54',\n",
       " '/wiki/Spam_(Monty_Python)',\n",
       " '/wiki/Foobar',\n",
       " '#cite_note-55',\n",
       " '#cite_note-56',\n",
       " '/wiki/Neologism',\n",
       " '#cite_note-AutoNT-27-57',\n",
       " '#cite_note-AutoNT-25-58',\n",
       " '/wiki/Python_syntax_and_semantics',\n",
       " '/wiki/Curly_bracket_programming_language',\n",
       " '/wiki/C_(programming_language)',\n",
       " '/wiki/Pascal_(programming_language)',\n",
       " '#cite_note-AutoNT-52-59',\n",
       " '/wiki/Python_syntax_and_semantics#Indentation',\n",
       " '/wiki/Whitespace_character',\n",
       " '/wiki/Curly_bracket_programming_language',\n",
       " '/wiki/Block_(programming)',\n",
       " '#cite_note-AutoNT-53-60',\n",
       " '#cite_note-guttag-1',\n",
       " '/wiki/Off-side_rule',\n",
       " '/wiki/Statement_(computer_science)',\n",
       " '/wiki/Imperative_programming',\n",
       " '/wiki/C_(programming_language)',\n",
       " '/wiki/Memory_allocation',\n",
       " '/wiki/Variable_(computer_science)',\n",
       " '/wiki/Type_system',\n",
       " '/wiki/Pointer_(computer_programming)',\n",
       " '/wiki/Object_(computer_science)',\n",
       " '/wiki/Type_system',\n",
       " '/wiki/Dynamic_type',\n",
       " '/wiki/If-then-else',\n",
       " '/wiki/Foreach#Python',\n",
       " '/wiki/While_loop#Python',\n",
       " '/wiki/Exception_handling_syntax#Python',\n",
       " '/wiki/Class_(computer_science)',\n",
       " '/wiki/Object-oriented_programming',\n",
       " '/wiki/Function_(computing)',\n",
       " '/wiki/Method_(computing)',\n",
       " '#cite_note-61',\n",
       " '/wiki/Lock_(computer_science)',\n",
       " '/wiki/Computer_file',\n",
       " '/wiki/Resource_Acquisition_Is_Initialization',\n",
       " '#cite_note-62',\n",
       " '/wiki/NOP',\n",
       " '/wiki/Assertion_(programming)',\n",
       " '/wiki/Generator_(computer_science)#Python',\n",
       " '/wiki/Coroutine',\n",
       " '#cite_note-diff_py2-3-63',\n",
       " '/wiki/Tail_call',\n",
       " '/wiki/First-class_continuations',\n",
       " '#cite_note-AutoNT-55-64',\n",
       " '#cite_note-AutoNT-56-65',\n",
       " '/wiki/Coroutine',\n",
       " '/wiki/Generator_(computer_science)',\n",
       " '#cite_note-AutoNT-57-66',\n",
       " '/wiki/Lazy_evaluation',\n",
       " '/wiki/Iterator',\n",
       " '#cite_note-AutoNT-58-67',\n",
       " '/wiki/Expression_(computer_science)',\n",
       " '/wiki/C_(programming_language)',\n",
       " '/wiki/Java_(programming_language)',\n",
       " '#cite_note-68',\n",
       " '/wiki/Matrix_multiplication',\n",
       " '#cite_note-PEP465-69',\n",
       " '#cite_note-Python3.5Changelog-70',\n",
       " '#cite_note-71',\n",
       " '#cite_note-72',\n",
       " '/wiki/List_comprehension#Python',\n",
       " '/wiki/Generator_(computer_science)',\n",
       " '#cite_note-AutoNT-59-49',\n",
       " '/wiki/Anonymous_function',\n",
       " '/wiki/Lambda_(programming)',\n",
       " '#cite_note-AutoNT-60-73',\n",
       " '/wiki/%3F:',\n",
       " '/wiki/List_(computer_science)',\n",
       " '/wiki/Tuple',\n",
       " '/wiki/Immutable',\n",
       " '#cite_note-74',\n",
       " '/wiki/Wikipedia:Citation_needed',\n",
       " '/wiki/Printf',\n",
       " '/wiki/C_(programming_language)',\n",
       " '#cite_note-pep-0498-75',\n",
       " '/wiki/String_literal',\n",
       " '/wiki/Unix_shell',\n",
       " '/wiki/Perl',\n",
       " '/wiki/Escape_character',\n",
       " '/wiki/String_interpolation',\n",
       " '#cite_note-pep-0498-75',\n",
       " '/wiki/Here_document',\n",
       " '/wiki/Ruby_(programming_language)',\n",
       " '/wiki/Raw_string',\n",
       " '/wiki/Regular_expression',\n",
       " '/wiki/Microsoft_Windows',\n",
       " '/wiki/C_Sharp_(programming_language)',\n",
       " '/wiki/Array_index',\n",
       " '/wiki/Array_slicing',\n",
       " '/wiki/Zero-based',\n",
       " '/wiki/Shallow_copy',\n",
       " '/wiki/Common_Lisp',\n",
       " '/wiki/Scheme_(programming_language)',\n",
       " '/wiki/Ruby_(programming_language)',\n",
       " '/wiki/List_comprehensions',\n",
       " '/wiki/Conditional_(programming)',\n",
       " '/wiki/Lambda_(programming)',\n",
       " '/wiki/Method_(programming)',\n",
       " '/wiki/Function_(programming)',\n",
       " '/wiki/Syntactic_sugar',\n",
       " '/wiki/This_(computer_programming)',\n",
       " '/wiki/Instance_data',\n",
       " '/wiki/C%2B%2B',\n",
       " '/wiki/Java_(programming_language)',\n",
       " '/wiki/Objective-C',\n",
       " '/wiki/Ruby_(programming_language)',\n",
       " '#cite_note-AutoNT-61-76',\n",
       " '/wiki/Duck_typing',\n",
       " '/wiki/Compile_time',\n",
       " '/wiki/Type_system#Dynamic_type_checking_and_runtime_type_information',\n",
       " '/wiki/Strongly_typed_programming_language',\n",
       " '/wiki/Class_(computer_science)',\n",
       " '/wiki/Object-oriented_programming',\n",
       " '/wiki/Object_(computer_science)',\n",
       " '/wiki/Metaclass',\n",
       " '/wiki/Metaprogramming',\n",
       " '/wiki/Reflection_(computer_science)',\n",
       " '#cite_note-classy-77',\n",
       " '/wiki/Gradual_typing',\n",
       " '#cite_note-78',\n",
       " '#cite_note-79',\n",
       " '/wiki/Boolean_value',\n",
       " '/wiki/Byte',\n",
       " '/wiki/Complex_number',\n",
       " '/wiki/Associative_array',\n",
       " '/wiki/Ellipsis_(programming_operator)',\n",
       " '/wiki/NumPy',\n",
       " '/wiki/Floating_point',\n",
       " '/wiki/Set_(computer_science)',\n",
       " '/wiki/Integer_(computer_science)',\n",
       " '#cite_note-pep0237-80',\n",
       " '/wiki/List_(computer_science)',\n",
       " '/wiki/Set_(computer_science)',\n",
       " '/wiki/Immutable_object',\n",
       " '/wiki/Character_string',\n",
       " '#cite_note-81',\n",
       " '#cite_note-82',\n",
       " '#cite_note-83',\n",
       " '#cite_note-pep0238-84',\n",
       " '/wiki/Wikipedia:Please_clarify',\n",
       " '/wiki/Half-open_interval',\n",
       " '#cite_note-AutoNT-62-85',\n",
       " '/wiki/Rounding',\n",
       " '/wiki/Rounding#Tie-breaking',\n",
       " '#cite_note-AutoNT-63-86',\n",
       " '/wiki/Round_to_even',\n",
       " '#cite_note-AutoNT-64-87',\n",
       " '#cite_note-AutoNT-65-88',\n",
       " '#cite_note-CPL-89',\n",
       " '/wiki/Arbitrary_precision_arithmetic',\n",
       " '#cite_note-90',\n",
       " '#cite_note-91',\n",
       " '#cite_note-92',\n",
       " '/wiki/NumPy',\n",
       " '/wiki/Wikipedia:Citation_needed',\n",
       " '/wiki/Standard_library',\n",
       " '#cite_note-AutoNT-86-93',\n",
       " '/wiki/MIME',\n",
       " '/wiki/Hypertext_Transfer_Protocol',\n",
       " '/wiki/Graphical_user_interface',\n",
       " '/wiki/Relational_database',\n",
       " '/wiki/Pseudorandom_number_generator',\n",
       " '#cite_note-AutoNT-88-94',\n",
       " '/wiki/Regular_expression',\n",
       " '/wiki/Unit_testing',\n",
       " '/wiki/Web_Server_Gateway_Interface',\n",
       " '#cite_note-AutoNT-89-95',\n",
       " '//en.wikipedia.org/w/index.php?title=Python_(programming_language)&action=edit',\n",
       " '/wiki/Python_Package_Index',\n",
       " '#cite_note-96',\n",
       " '#cite_note-97',\n",
       " '#cite_note-98',\n",
       " '#cite_note-99',\n",
       " '/wiki/Comparison_of_integrated_development_environments#Python',\n",
       " '/wiki/Read%E2%80%93eval%E2%80%93print_loop',\n",
       " '/wiki/Command_line_interpreter',\n",
       " '/wiki/IDLE_(Python)',\n",
       " '/wiki/IPython',\n",
       " '/wiki/Syntax_highlighting',\n",
       " '/wiki/Integrated_development_environment',\n",
       " '/wiki/Web_browser',\n",
       " '/wiki/SageMath',\n",
       " '/wiki/PythonAnywhere',\n",
       " '#cite_note-100',\n",
       " '/wiki/List_of_Python_software#Python_implementations',\n",
       " '/wiki/CPython',\n",
       " '/wiki/Reference_implementation',\n",
       " '/wiki/C_(programming_language)',\n",
       " '/wiki/C89_(C_version)',\n",
       " '/wiki/C99',\n",
       " '#cite_note-AutoNT-66-101',\n",
       " '/wiki/Bytecode',\n",
       " '#cite_note-AutoNT-67-102',\n",
       " '/wiki/Virtual_machine',\n",
       " '#cite_note-AutoNT-68-103',\n",
       " '/wiki/Microsoft_Windows',\n",
       " '/wiki/Unix-like',\n",
       " '#cite_note-AutoNT-69-104',\n",
       " '/wiki/PyPy',\n",
       " '#cite_note-AutoNT-70-105',\n",
       " '/wiki/Just-in-time_compilation',\n",
       " '#cite_note-AutoNT-71-106',\n",
       " '/wiki/Stackless_Python',\n",
       " '/wiki/Microthread',\n",
       " '#cite_note-AutoNT-73-107',\n",
       " '/wiki/MicroPython',\n",
       " '/wiki/CircuitPython',\n",
       " '/wiki/Microcontroller',\n",
       " '/wiki/Unladen_Swallow',\n",
       " '/wiki/LLVM',\n",
       " '#cite_note-AutoNT-74-108',\n",
       " '/wiki/Psyco',\n",
       " '/wiki/Just-in-time_compilation',\n",
       " '/wiki/Run-time_algorithm_specialisation',\n",
       " '/wiki/Data_type',\n",
       " '/wiki/Nokia',\n",
       " '/wiki/Series_60',\n",
       " '/wiki/PyS60',\n",
       " '/wiki/Symbian',\n",
       " '/wiki/N900',\n",
       " '/wiki/GTK',\n",
       " '#cite_note-109',\n",
       " '/wiki/Object_language',\n",
       " '/wiki/Jython',\n",
       " '/wiki/Java_virtual_machine',\n",
       " '/wiki/IronPython',\n",
       " '/wiki/Common_Language_Runtime',\n",
       " '/wiki/RPython',\n",
       " '/wiki/C_(programming_language)',\n",
       " '/wiki/Java_bytecode',\n",
       " '/wiki/Common_Intermediate_Language',\n",
       " '/wiki/Pyjs',\n",
       " '/wiki/JavaScript',\n",
       " '/wiki/Cython',\n",
       " '/wiki/C_(programming_language)',\n",
       " '/wiki/C%2B%2B',\n",
       " '/wiki/C%2B%2B',\n",
       " '/wiki/Pyrex_(programming_language)',\n",
       " '/wiki/Shed_Skin',\n",
       " '/wiki/Go_(programming_language)',\n",
       " '/wiki/MyHDL',\n",
       " '/wiki/VHDL',\n",
       " '/wiki/Nuitka',\n",
       " '#cite_note-110',\n",
       " '#cite_note-111',\n",
       " '#cite_note-PepCite000-112',\n",
       " '/wiki/Benevolent_Dictator_For_Life',\n",
       " '#cite_note-PepCite000-112',\n",
       " '/wiki/Roundup_(issue_tracker)',\n",
       " '/wiki/Bug_tracker',\n",
       " '#cite_note-AutoNT-21-113',\n",
       " '/wiki/Self-hosted',\n",
       " '/wiki/Mercurial',\n",
       " '/wiki/GitHub',\n",
       " '#cite_note-py_dev_guide-114',\n",
       " '/wiki/Ported',\n",
       " '#cite_note-release-schedule-115',\n",
       " '#cite_note-AutoNT-22-116',\n",
       " '/wiki/Beta_release',\n",
       " '/wiki/Unit_test',\n",
       " '/wiki/BuildBot',\n",
       " '/wiki/Continuous_integration',\n",
       " '#cite_note-AutoNT-23-117',\n",
       " '#cite_note-118',\n",
       " '//en.wikipedia.org/w/index.php?title=Python_(programming_language)&action=edit',\n",
       " '/wiki/Python_Package_Index',\n",
       " '/wiki/Academic_conference',\n",
       " '/wiki/PyCon',\n",
       " '/wiki/Pyladies',\n",
       " '/wiki/Monty_Python',\n",
       " '#cite_note-tutorial-chapter1-119',\n",
       " '/wiki/Metasyntactic_variable',\n",
       " '/wiki/Spam_(Monty_Python)',\n",
       " '/wiki/Foobar',\n",
       " '#cite_note-tutorial-chapter1-119',\n",
       " '#cite_note-AutoNT-26-120',\n",
       " '#cite_note-121',\n",
       " '#cite_note-122',\n",
       " '/wiki/Pygame',\n",
       " '/wiki/Language_binding',\n",
       " '/wiki/Simple_DirectMedia_Layer',\n",
       " '/wiki/PyQt',\n",
       " '/wiki/PyGTK',\n",
       " '/wiki/Qt_(software)',\n",
       " '/wiki/GTK',\n",
       " '/wiki/PyPy',\n",
       " '/wiki/Sphinx_(documentation_generator)',\n",
       " '/wiki/Epydoc',\n",
       " '/wiki/HeaderDoc',\n",
       " '/wiki/Pydoc',\n",
       " '/wiki/List_of_Python_software',\n",
       " '/wiki/TIOBE_Programming_Community_Index',\n",
       " '//en.wikipedia.org/w/index.php?title=Python_(programming_language)&action=edit',\n",
       " '/wiki/Java_(programming_language)',\n",
       " '/wiki/C_(programming_language)',\n",
       " '/wiki/C%2B%2B',\n",
       " '#cite_note-123',\n",
       " '#cite_note-AutoNT-34-124',\n",
       " '#cite_note-AutoNT-28-125',\n",
       " '/wiki/Wikipedia',\n",
       " '/wiki/Google',\n",
       " '#cite_note-quotes-about-python-126',\n",
       " '/wiki/Yahoo!',\n",
       " '#cite_note-AutoNT-29-127',\n",
       " '/wiki/CERN',\n",
       " '#cite_note-AutoNT-30-128',\n",
       " '/wiki/NASA',\n",
       " '#cite_note-AutoNT-31-129',\n",
       " '/wiki/Facebook',\n",
       " '#cite_note-130',\n",
       " '/wiki/Amazon_(company)',\n",
       " '/wiki/Instagram',\n",
       " '/wiki/Spotify',\n",
       " '#cite_note-131',\n",
       " '/wiki/Industrial_Light_%26_Magic',\n",
       " '#cite_note-AutoNT-32-132',\n",
       " '/wiki/ITA_Software',\n",
       " '#cite_note-AutoNT-33-133',\n",
       " '/wiki/Reddit',\n",
       " '/wiki/Scripting_language',\n",
       " '/wiki/Web_application',\n",
       " '/wiki/Mod_wsgi',\n",
       " '/wiki/Apache_web_server',\n",
       " '#cite_note-AutoNT-35-134',\n",
       " '/wiki/Web_Server_Gateway_Interface',\n",
       " '/wiki/Web_framework',\n",
       " '/wiki/Django_(web_framework)',\n",
       " '/wiki/Pylons_(web_framework)',\n",
       " '/wiki/Pyramid_(web_framework)',\n",
       " '/wiki/TurboGears',\n",
       " '/wiki/Web2py',\n",
       " '/wiki/Tornado_(web_server)',\n",
       " '/wiki/Flask_(web_framework)',\n",
       " '/wiki/Bottle_(web_framework)',\n",
       " '/wiki/Zope',\n",
       " '/wiki/Pyjs',\n",
       " '/wiki/IronPython',\n",
       " '/wiki/SQLAlchemy',\n",
       " '/wiki/Data_mapper_pattern',\n",
       " '/wiki/Twisted_(software)',\n",
       " '/wiki/Dropbox_(service)',\n",
       " '/wiki/NumPy',\n",
       " '/wiki/SciPy',\n",
       " '/wiki/Matplotlib',\n",
       " '#cite_note-cise-135',\n",
       " '#cite_note-millman-136',\n",
       " '/wiki/Biopython',\n",
       " '/wiki/Astropy',\n",
       " '/wiki/SageMath',\n",
       " '/wiki/Mathematical_software',\n",
       " '/w/index.php?title=Electronic_Notebook&action=edit&redlink=1',\n",
       " '/wiki/Mathematics',\n",
       " '/wiki/Algebra',\n",
       " '/wiki/Combinatorics',\n",
       " '/wiki/Numerical_mathematics',\n",
       " '/wiki/Number_theory',\n",
       " '/wiki/Calculus',\n",
       " '/wiki/Finite_element_method',\n",
       " '/wiki/Abaqus',\n",
       " '/wiki/FreeCAD',\n",
       " '/wiki/3ds_Max',\n",
       " '/wiki/Blender_(software)',\n",
       " '/wiki/Cinema_4D',\n",
       " '/wiki/Lightwave',\n",
       " '/wiki/Houdini_(software)',\n",
       " '/wiki/Maya_(software)',\n",
       " '/wiki/Modo_(software)',\n",
       " '/wiki/MotionBuilder',\n",
       " '/wiki/Autodesk_Softimage',\n",
       " '/wiki/Nuke_(software)',\n",
       " '/wiki/GIMP',\n",
       " '#cite_note-137',\n",
       " '/wiki/Inkscape',\n",
       " '/wiki/Scribus',\n",
       " '/wiki/Paint_Shop_Pro',\n",
       " '#cite_note-AutoNT-38-138',\n",
       " '/wiki/Musical_notation',\n",
       " '/wiki/Scorewriter',\n",
       " '/wiki/Capella_(notation_program)',\n",
       " '/wiki/GNU_Debugger',\n",
       " '/wiki/Prettyprint',\n",
       " '/wiki/Esri',\n",
       " '/wiki/ArcGIS',\n",
       " '#cite_note-AutoNT-39-139',\n",
       " '#cite_note-AutoNT-40-140',\n",
       " '#cite_note-AutoNT-41-141',\n",
       " '/wiki/Programming_language',\n",
       " '/wiki/Google_App_Engine',\n",
       " '/wiki/Java_(software_platform)',\n",
       " '/wiki/Go_(programming_language)',\n",
       " '#cite_note-AutoNT-42-142',\n",
       " '/wiki/Algorithmic_trading',\n",
       " '#cite_note-143',\n",
       " '#cite_note-144',\n",
       " '/wiki/Artificial_intelligence',\n",
       " '#cite_note-AutoNT-43-145',\n",
       " '#cite_note-AutoNT-44-146',\n",
       " '#cite_note-AutoNT-45-147',\n",
       " '#cite_note-AutoNT-46-148',\n",
       " '/wiki/Modular_programming',\n",
       " '/wiki/Natural_language_processing',\n",
       " '#cite_note-AutoNT-47-149',\n",
       " '/wiki/Linux_distribution',\n",
       " '/wiki/AmigaOS_4',\n",
       " '/wiki/FreeBSD',\n",
       " '/wiki/NetBSD',\n",
       " '/wiki/OpenBSD',\n",
       " '/wiki/MacOS',\n",
       " '/wiki/Ubuntu_(operating_system)',\n",
       " '/wiki/Ubiquity_(software)',\n",
       " '/wiki/Red_Hat_Linux',\n",
       " '/wiki/Fedora_(operating_system)',\n",
       " '/wiki/Anaconda_(installer)',\n",
       " '/wiki/Gentoo_Linux',\n",
       " '/wiki/Package_management_system',\n",
       " '/wiki/Portage_(software)',\n",
       " '/wiki/Information_security',\n",
       " '#cite_note-AutoNT-49-150',\n",
       " '#cite_note-AutoNT-50-151',\n",
       " '/wiki/Sugar_(software)',\n",
       " '/wiki/One_Laptop_per_Child',\n",
       " '/wiki/Sugar_Labs',\n",
       " '#cite_note-AutoNT-51-152',\n",
       " '/wiki/Raspberry_Pi',\n",
       " '/wiki/Single-board_computer',\n",
       " '/wiki/LibreOffice',\n",
       " '#cite_note-153',\n",
       " '/wiki/Boo_(programming_language)',\n",
       " '#cite_note-AutoNT-90-154',\n",
       " '/wiki/Cobra_(programming_language)',\n",
       " '#cite_note-AutoNT-91-155',\n",
       " '/wiki/Design_by_contract',\n",
       " '/wiki/Unit_testing',\n",
       " '/wiki/Static_typing',\n",
       " '#cite_note-AutoNT-92-156',\n",
       " '/wiki/CoffeeScript',\n",
       " '/wiki/ECMAScript',\n",
       " '/wiki/Iterator',\n",
       " '/wiki/Generator_(computer_science)',\n",
       " '#cite_note-AutoNT-93-157',\n",
       " '/wiki/Go_(programming_language)',\n",
       " '#cite_note-AutoNT-94-158',\n",
       " '/wiki/Groovy_(programming_language)',\n",
       " '/wiki/Java_(programming_language)',\n",
       " '#cite_note-AutoNT-95-159',\n",
       " '/wiki/Julia_(programming_language)',\n",
       " '/wiki/Hygienic_macro',\n",
       " '#cite_note-Julia-23',\n",
       " '/wiki/Kotlin_(programming_language)',\n",
       " '#cite_note-160',\n",
       " '/wiki/Ruby_(programming_language)',\n",
       " '/wiki/Yukihiro_Matsumoto',\n",
       " '#cite_note-linuxdevcenter-161',\n",
       " '/wiki/Swift_(programming_language)',\n",
       " '#cite_note-162',\n",
       " '/wiki/Gdscript',\n",
       " '/wiki/Tcl',\n",
       " '#cite_note-AutoNT-99-163',\n",
       " '/wiki/Erlang_(programming_language)',\n",
       " '#cite_note-AutoNT-100-164',\n",
       " '/wiki/TIOBE_index',\n",
       " '#cite_note-AutoNT-101-165',\n",
       " '/wiki/Portal:Free_software',\n",
       " '/wiki/Portal:Python_programming',\n",
       " '/wiki/History_of_Python',\n",
       " '/wiki/List_of_integrated_development_environments_for_Python',\n",
       " '/wiki/Comparison_of_programming_languages',\n",
       " '/wiki/List_of_programming_languages',\n",
       " '/wiki/Pip_(package_manager)',\n",
       " '/wiki/Off-side_rule',\n",
       " '#cite_ref-guttag_1-0',\n",
       " '#cite_ref-guttag_1-1',\n",
       " '/wiki/International_Standard_Book_Number',\n",
       " '/wiki/Special:BookSources/978-0-262-52962-4',\n",
       " '#cite_ref-2',\n",
       " 'https://www.python.org/downloads/release/python-370/?hn',\n",
       " '#cite_ref-3',\n",
       " 'https://blog.python.org/2018/05/python-2715-released.html',\n",
       " '#cite_ref-4',\n",
       " 'https://www.python.org/dev/peps/pep-0483/',\n",
       " '#cite_ref-5',\n",
       " 'https://www.python.org/dev/peps/pep-0488/',\n",
       " '#cite_ref-6',\n",
       " 'https://www.python.org/dev/peps/pep-0441/',\n",
       " '#cite_ref-faq-created_7-0',\n",
       " '#cite_ref-faq-created_7-1',\n",
       " 'https://docs.python.org/faq/general.html#why-was-python-created-in-the-first-place',\n",
       " '#cite_ref-98-interview_8-0',\n",
       " 'https://web.archive.org/web/20070501105422/http://www.amk.ca/python/writing/gvr-interview',\n",
       " 'http://www.amk.ca/python/writing/gvr-interview',\n",
       " '#cite_ref-9',\n",
       " 'https://docs.python.org/3/library/itertools.html',\n",
       " '#cite_ref-AutoNT-1_10-0',\n",
       " 'http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.38.2023',\n",
       " '#cite_ref-classmix_11-0',\n",
       " '#cite_ref-classmix_11-1',\n",
       " 'https://docs.python.org/tutorial/classes.html',\n",
       " '#cite_ref-effbot-call-by-object_12-0',\n",
       " 'http://effbot.org/zone/call-by-object.htm',\n",
       " '#cite_ref-AutoNT-2_13-0',\n",
       " 'https://www.python.org/download/releases/2.3/mro/',\n",
       " '#cite_ref-AutoNT-3_14-0',\n",
       " 'https://docs.python.org/howto/functional.html',\n",
       " '#cite_ref-AutoNT-4_15-0',\n",
       " 'https://www.python.org/dev/peps/pep-0255/',\n",
       " '#cite_ref-AutoNT-5_16-0',\n",
       " 'https://www.python.org/dev/peps/pep-0318/',\n",
       " '#cite_ref-AutoNT-6_17-0',\n",
       " 'https://docs.python.org/3.2/tutorial/controlflow.html',\n",
       " '#cite_ref-18',\n",
       " 'https://docs.python.org/3/library/itertools.html',\n",
       " '#cite_ref-19',\n",
       " 'https://coffeescript.org/',\n",
       " '#cite_ref-20',\n",
       " 'https://wiki.gnome.org/action/show/Projects/Genie',\n",
       " '#cite_ref-21',\n",
       " 'http://www.2ality.com/2013/02/javascript-influences.html',\n",
       " '#cite_ref-22',\n",
       " 'http://speakingjs.com/es5/ch03.html',\n",
       " '#cite_ref-Julia_23-0',\n",
       " '#cite_ref-Julia_23-1',\n",
       " 'http://julialang.org/blog/2012/02/why-we-created-julia',\n",
       " '#cite_ref-The_Ring_programming_language_and_other_languages_24-0',\n",
       " 'http://ring-lang.sourceforge.net/doc1.6/introduction.html#ring-and-other-languages',\n",
       " '/w/index.php?title=Ring-lang&action=edit&redlink=1',\n",
       " '#cite_ref-bini_25-0',\n",
       " '/wiki/International_Standard_Book_Number',\n",
       " '/wiki/Special:BookSources/978-1-59059-881-8',\n",
       " '#cite_ref-lattner2014_26-0',\n",
       " 'http://nondot.org/sabre/',\n",
       " '#cite_ref-AutoNT-7_27-0',\n",
       " 'https://web.archive.org/web/20120623165941/http://cutter.rexx.com/~dkuhlman/python_book_01.html',\n",
       " 'http://cutter.rexx.com/~dkuhlman/python_book_01.html',\n",
       " '#cite_ref-28',\n",
       " 'https://www.linuxjournal.com/content/guido-van-rossum-stepping-down-role-pythons-benevolent-dictator-life',\n",
       " '#cite_ref-29',\n",
       " 'https://www.theinquirer.net/inquirer/news/3035842/python-boss-guido-van-rossum-steps-down-after-30-years',\n",
       " 'http://www.theinquirer.net',\n",
       " '/wiki/Help:CS1_errors#param_has_ext_link',\n",
       " '#cite_ref-About_30-0',\n",
       " 'https://www.python.org/about',\n",
       " '#cite_ref-31',\n",
       " 'https://docs.python.org/3/license.html',\n",
       " '#cite_ref-venners-interview-pt-1_32-0',\n",
       " '#cite_ref-venners-interview-pt-1_32-1',\n",
       " 'http://www.artima.com/intv/pythonP.html',\n",
       " '#cite_ref-timeline-of-python_33-0',\n",
       " 'http://python-history.blogspot.com/2009/01/brief-timeline-of-python.html',\n",
       " '#cite_ref-AutoNT-12_34-0',\n",
       " '/wiki/Guido_van_Rossum',\n",
       " 'http://mail.python.org/pipermail/python-dev/2000-August/008881.html',\n",
       " '#cite_ref-lj-bdfl-resignation_35-0',\n",
       " 'https://www.linuxjournal.com/content/guido-van-rossum-stepping-down-role-pythons-benevolent-dictator-life',\n",
       " '#cite_ref-36',\n",
       " 'https://www.python.org/doc/essays/foreword/',\n",
       " '#cite_ref-newin-2.0_37-0',\n",
       " 'https://docs.python.org/whatsnew/2.0.html',\n",
       " '#cite_ref-3.0-release_38-0',\n",
       " 'https://www.python.org/download/releases/3.0/',\n",
       " '#cite_ref-pep-3000_39-0',\n",
       " 'https://www.python.org/dev/peps/pep-3000/',\n",
       " '#cite_ref-40',\n",
       " 'https://docs.python.org/3/library/2to3.html',\n",
       " '#cite_ref-41',\n",
       " 'https://legacy.python.org/dev/peps/pep-0373/',\n",
       " '#cite_ref-42',\n",
       " 'https://www.python.org/dev/peps/pep-0466/',\n",
       " '#cite_ref-43',\n",
       " 'https://opensource.googleblog.com/2017/01/grumpy-go-running-python.html',\n",
       " '#cite_ref-AutoNT-13_44-0',\n",
       " 'https://www.webcitation.org/5lubkaJRc?url=http://www.python.org/community/pycon/dc2004/papers/24/metaclasses-pycon.pdf',\n",
       " 'https://www.python.org/community/pycon/dc2004/papers/24/metaclasses-pycon.pdf',\n",
       " '#cite_ref-AutoNT-14_45-0',\n",
       " 'https://docs.python.org/3.0/reference/datamodel.html#special-method-names',\n",
       " '#cite_ref-AutoNT-15_46-0',\n",
       " 'http://www.nongnu.org/pydbc/',\n",
       " '#cite_ref-AutoNT-16_47-0',\n",
       " 'http://www.wayforward.net/pycontract/',\n",
       " '#cite_ref-AutoNT-17_48-0',\n",
       " 'https://sites.google.com/site/pydatalog/',\n",
       " '#cite_ref-AutoNT-59_49-0',\n",
       " '#cite_ref-AutoNT-59_49-1',\n",
       " 'https://www.python.org/dev/peps/pep-0289/',\n",
       " '#cite_ref-AutoNT-18_50-0',\n",
       " 'https://docs.python.org/3/library/itertools.html',\n",
       " '#cite_ref-PEP20_51-0',\n",
       " '#cite_ref-PEP20_51-1',\n",
       " 'https://www.python.org/dev/peps/pep-0020/',\n",
       " '#cite_ref-AutoNT-19_52-0',\n",
       " 'http://shop.oreilly.com/product/9780596007973.do',\n",
       " '/wiki/O%27Reilly_Media',\n",
       " '/wiki/International_Standard_Book_Number',\n",
       " '/wiki/Special:BookSources/978-0-596-00797-3',\n",
       " '#cite_ref-AutoNT-20_53-0',\n",
       " 'http://ebeab.com/2014/01/21/python-culture/',\n",
       " '#cite_ref-AutoNT-24_54-0',\n",
       " 'https://docs.python.org/2/faq/general.html#why-is-it-called-python',\n",
       " '#cite_ref-55',\n",
       " 'https://insidetech.monster.com/training/articles/8114-15-ways-python-is-a-powerful-force-on-the-web',\n",
       " '#cite_ref-56',\n",
       " 'https://docs.python.org/2/library/pprint.html',\n",
       " '#cite_ref-AutoNT-27_57-0',\n",
       " 'http://python.net/~goodger/projects/pycon/2007/idiomatic/handout.html',\n",
       " '#cite_ref-AutoNT-25_58-0',\n",
       " 'http://python.net/crew/mwh/hacks/objectthink.html',\n",
       " '#cite_ref-AutoNT-52_59-0',\n",
       " 'https://docs.python.org/faq/general.html#is-python-a-good-language-for-beginning-programmers',\n",
       " '#cite_ref-AutoNT-53_60-0',\n",
       " 'http://www.secnetix.de/~olli/Python/block_indentation.hawk',\n",
       " '#cite_ref-61',\n",
       " 'https://www.python.org/download/releases/2.5/',\n",
       " '#cite_ref-62',\n",
       " 'https://www.python.org/download/releases/2.5/highlights/',\n",
       " '#cite_ref-diff_py2-3_63-0',\n",
       " 'https://inventwithpython.com/appendixa.html',\n",
       " '/wiki/International_Standard_Book_Number',\n",
       " '/wiki/Special:BookSources/978-0-9821060-1-3',\n",
       " '#cite_ref-AutoNT-55_64-0',\n",
       " 'http://neopythonic.blogspot.be/2009/04/tail-recursion-elimination.html',\n",
       " '#cite_ref-AutoNT-56_65-0',\n",
       " 'http://www.artima.com/weblogs/viewpost.jsp?thread=147358',\n",
       " '#cite_ref-AutoNT-57_66-0',\n",
       " 'https://www.python.org/dev/peps/pep-0342/',\n",
       " '#cite_ref-AutoNT-58_67-0',\n",
       " 'https://www.python.org/dev/peps/pep-0380/',\n",
       " '#cite_ref-68',\n",
       " 'https://docs.python.org',\n",
       " '#cite_ref-PEP465_69-0',\n",
       " 'https://www.python.org/dev/peps/pep-0465/',\n",
       " '#cite_ref-Python3.5Changelog_70-0',\n",
       " 'https://www.python.org/downloads/release/python-351/',\n",
       " '#cite_ref-71',\n",
       " 'https://docs.oracle.com/javase/specs/jls/se8/html/jls-15.html#jls-15.21.1',\n",
       " '/wiki/Oracle_Corporation',\n",
       " '#cite_ref-72',\n",
       " 'https://docs.oracle.com/javase/specs/jls/se8/html/jls-15.html#jls-15.21.3',\n",
       " '#cite_ref-AutoNT-60_73-0',\n",
       " 'https://www.python.org/dev/peps/pep-0308/',\n",
       " '#cite_ref-74',\n",
       " 'https://docs.python.org/3/library/stdtypes.html#tuple',\n",
       " '#cite_ref-pep-0498_75-0',\n",
       " '#cite_ref-pep-0498_75-1',\n",
       " 'https://www.python.org/dev/peps/pep-0498/',\n",
       " '#cite_ref-AutoNT-61_76-0',\n",
       " 'https://docs.python.org/faq/design.html#why-must-self-be-used-explicitly-in-method-definitions-and-calls',\n",
       " '#cite_ref-classy_77-0',\n",
       " 'https://docs.python.org/reference/datamodel.html#new-style-and-classic-classes',\n",
       " '#cite_ref-78',\n",
       " 'https://lwn.net/Articles/627418/',\n",
       " '#cite_ref-79',\n",
       " 'http://mypy-lang.org/',\n",
       " '#cite_ref-pep0237_80-0',\n",
       " 'https://www.python.org/dev/peps/pep-0237/',\n",
       " '#cite_ref-81',\n",
       " 'https://legacy.python.org/dev/peps/pep-0465/',\n",
       " '#cite_ref-82',\n",
       " 'https://stackoverflow.com/questions/8305199/the-tilde-operator-in-python',\n",
       " '#cite_ref-83',\n",
       " 'https://wiki.python.org/moin/BitwiseOperators',\n",
       " '#cite_ref-pep0238_84-0',\n",
       " 'https://www.python.org/dev/peps/pep-0238/',\n",
       " '#cite_ref-AutoNT-62_85-0',\n",
       " 'http://python-history.blogspot.com/2010/08/why-pythons-integer-division-floors.html',\n",
       " '#cite_ref-AutoNT-63_86-0',\n",
       " 'https://docs.python.org/library/functions.html#round',\n",
       " '#cite_ref-AutoNT-64_87-0',\n",
       " 'https://docs.python.org/py3k/library/functions.html#round',\n",
       " '#cite_ref-AutoNT-65_88-0',\n",
       " '#cite_ref-CPL_89-0',\n",
       " 'https://en.wikipedia.org/wiki/The_C_Programming_Language',\n",
       " '#cite_ref-90',\n",
       " 'https://docs.python.org/2.7/library/stdtypes.html',\n",
       " '#cite_ref-91',\n",
       " 'https://www.python.org/dev/peps/pep-0327/',\n",
       " '#cite_ref-92',\n",
       " 'https://docs.python.org/2.6/whatsnew/2.6.html',\n",
       " '#cite_ref-AutoNT-86_93-0',\n",
       " 'http://www.oracle.com/technetwork/articles/piotrowski-pythoncore-084049.html',\n",
       " '#cite_ref-AutoNT-88_94-0',\n",
       " 'https://www.python.org/dev/peps/pep-0327/',\n",
       " '#cite_ref-AutoNT-89_95-0',\n",
       " 'https://www.python.org/dev/peps/pep-0333/',\n",
       " '#cite_ref-96',\n",
       " 'http://www.modulecounts.com/',\n",
       " '#cite_ref-97',\n",
       " 'https://likegeeks.com/python-web-scraping/',\n",
       " '#cite_ref-98',\n",
       " 'https://stackoverflow.com/questions/7948494/whats-the-difference-between-a-python-module-and-a-python-package',\n",
       " '#cite_ref-99',\n",
       " 'https://docs.python.org/3/index.html',\n",
       " '/wiki/Wikidata',\n",
       " 'https://www.wikidata.org/wiki/Q56718217',\n",
       " 'https://docs.python.org/3/glossary.html#term-package',\n",
       " '#cite_ref-100',\n",
       " 'https://www.enthought.com/products/canopy/',\n",
       " '#cite_ref-AutoNT-66_101-0',\n",
       " 'https://www.python.org/dev/peps/pep-0007/',\n",
       " '#cite_ref-AutoNT-67_102-0',\n",
       " 'https://docs.python.org/3/library/dis.html#python-bytecode-instructions',\n",
       " '#cite_ref-AutoNT-68_103-0',\n",
       " 'http://www.troeger.eu/teaching/pythonvm08.pdf',\n",
       " '#cite_ref-AutoNT-69_104-0',\n",
       " 'http://www.oreilly.com/pub/a/oreilly/frank/rossum_1099.html',\n",
       " '#cite_ref-AutoNT-70_105-0',\n",
       " 'http://pypy.org/compat.html',\n",
       " '#cite_ref-AutoNT-71_106-0',\n",
       " 'http://speed.pypy.org/',\n",
       " '#cite_ref-AutoNT-73_107-0',\n",
       " 'http://doc.pypy.org/en/latest/stackless.html',\n",
       " '#cite_ref-AutoNT-74_108-0',\n",
       " 'https://code.google.com/p/unladen-swallow/wiki/ProjectPlan',\n",
       " '#cite_ref-109',\n",
       " 'http://www.stochasticgeometry.ie/2010/04/29/python-on-the-nokia-n900/',\n",
       " '#cite_ref-110',\n",
       " 'http://nuitka.net/',\n",
       " '#cite_ref-111',\n",
       " '/wiki/ArXiv',\n",
       " '//arxiv.org/abs/1404.6388',\n",
       " '/wiki/Bibcode',\n",
       " 'http://adsabs.harvard.edu/abs/2014arXiv1404.6388M',\n",
       " '#cite_ref-PepCite000_112-0',\n",
       " '#cite_ref-PepCite000_112-1',\n",
       " 'https://www.python.org/dev/peps/pep-0001/',\n",
       " '#cite_ref-AutoNT-21_113-0',\n",
       " 'https://web.archive.org/web/20090601134342/http://www.python.org/dev/intro/',\n",
       " 'https://www.python.org/dev/intro/',\n",
       " '#cite_ref-py_dev_guide_114-0',\n",
       " 'https://docs.python.org/devguide/',\n",
       " '#cite_ref-release-schedule_115-0',\n",
       " 'https://mail.python.org/pipermail/python-dev/2002-April/022739.html',\n",
       " '#cite_ref-AutoNT-22_116-0',\n",
       " 'https://www.python.org/dev/peps/pep-0006/',\n",
       " '#cite_ref-AutoNT-23_117-0',\n",
       " 'https://www.python.org/dev/buildbot/',\n",
       " '#cite_ref-118',\n",
       " 'http://www.modulecounts.com/#',\n",
       " '#cite_ref-tutorial-chapter1_119-0',\n",
       " '#cite_ref-tutorial-chapter1_119-1',\n",
       " 'https://docs.python.org/tutorial/appetite.html',\n",
       " '#cite_ref-AutoNT-26_120-0',\n",
       " 'https://stackoverflow.com/questions/5033906/in-python-should-i-use-else-after-a-return-in-an-if-block',\n",
       " '/wiki/Stack_Overflow',\n",
       " '#cite_ref-121',\n",
       " 'https://books.google.com/books?id=1HxWGezDZcgC&pg=PA17',\n",
       " '/wiki/International_Standard_Book_Number',\n",
       " '/wiki/Special:BookSources/9781449379322',\n",
       " '#cite_ref-122',\n",
       " 'https://books.google.com/books?id=carqdIdfVlYC&pg=PR15',\n",
       " '/wiki/International_Standard_Book_Number',\n",
       " '/wiki/Special:BookSources/9780201748840',\n",
       " '#cite_ref-123',\n",
       " 'http://www.tiobe.com/tiobe-index/',\n",
       " '#cite_ref-AutoNT-34_124-0',\n",
       " 'http://www.tiobe.com/index.php/paperinfo/tpci/Python.html',\n",
       " '#cite_ref-AutoNT-28_125-0',\n",
       " 'http://page.mi.fu-berlin.de/prechelt/Biblio/jccpprt_computer2000.pdf',\n",
       " '#cite_ref-quotes-about-python_126-0',\n",
       " 'https://www.python.org/about/quotes/',\n",
       " '#cite_ref-AutoNT-29_127-0',\n",
       " 'https://wiki.python.org/moin/OrganizationsUsingPython',\n",
       " '#cite_ref-AutoNT-30_128-0',\n",
       " 'http://cdsweb.cern.ch/journal/CERNBulletin/2006/31/News%20Articles/974627?ln=en',\n",
       " '#cite_ref-AutoNT-31_129-0',\n",
       " 'https://www.python.org/about/success/usa/',\n",
       " '#cite_ref-130',\n",
       " ...]"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = \"https://en.wikipedia.org/wiki/Python_(programming_language)\"\n",
    "\n",
    "page = urllib.request.urlopen(url)\n",
    "soup = bs4.BeautifulSoup(page, 'lxml')\n",
    "\n",
    "[link.get('href') for link in soup.findAll('a')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to check whether a page contains a title or not."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Title Present\n",
      "Title Present\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "<title>Parth Shah</title>"
      ]
     },
     "execution_count": 33,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "def check_title(url):\n",
    "    \n",
    "    page = urllib.request.urlopen(url)\n",
    "    soup = bs4.BeautifulSoup(page, 'lxml')\n",
    "    if soup.title:\n",
    "        print(\"Title Present\")\n",
    "        return soup.title\n",
    "    else:\n",
    "        print(\"None\")\n",
    "        return False\n",
    "    \n",
    "check_title(\"http://example.com/\")\n",
    "check_title(\"http://shahparth.us/\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to list all language names and number of related articles in the order they appear in wikipedia.org"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 91,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "English\n",
      "5 725 000+ articles\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "日本語\n",
      "1 122 000+ 記事\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Español\n",
      "1 478 000+ artículos\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Deutsch\n",
      "2 224 000+ Artikel\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Русский\n",
      "1 499 000+ статей\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Français\n",
      "2 044 000+ articles\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Italiano\n",
      "1 464 000+ voci\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "中文\n",
      "1 024 000+ 條目\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Português\n",
      "1 006 000+ artigos\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "Polski\n",
      "1 301 000+ haseł\n",
      "\n",
      "\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[None, None, None, None, None, None, None, None, None, None]"
      ]
     },
     "execution_count": 91,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "url = \"https://www.wikipedia.org/\"\n",
    "\n",
    "source = urllib.request.urlopen(url)\n",
    "soup = bs4.BeautifulSoup(source, 'lxml')\n",
    "\n",
    "lang = soup.findAll('div', attrs = {'class': 'central-featured-lang'})\n",
    "#print(lang)\n",
    "\n",
    "[print(l.getText()) for l in lang]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to get the number of people visiting a U.S. government website right now.\n",
    "Source: https://analytics.usa.gov/data/live/realtime.json"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 93,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[{'active_visitors': '287151'}]\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "\n",
    "page = requests.get(\"https://analytics.usa.gov/data/live/realtime.json\")\n",
    "\n",
    "print(page.json()['data'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q.  Write a Python program get the number of security alerts issued by US-CERT in the current year.\n",
    "Source: https://www.us-cert.gov/ncas/alerts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "11\n"
     ]
    }
   ],
   "source": [
    "def which_year(year):\n",
    "    url = \"https://www.us-cert.gov/ncas/alerts/{}\".format(year)\n",
    "    get_num_alerts(url)\n",
    "\n",
    "def get_num_alerts(url):\n",
    "    \n",
    "    r = requests.get(url)\n",
    "    soup = bs4.BeautifulSoup(r.text, 'lxml')\n",
    "    \n",
    "    alerts = soup.find('div', attrs={'class': 'item-list'}).findAll('li', attrs={'class': 'views-row'})\n",
    "    print(len(alerts))\n",
    "\n",
    "which_year(2018)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to get the number of followers of a given twitter account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter your account name: Twitter\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'56,230,535 Followers'"
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "account_name = input(\"Enter your account name: \")\n",
    "twitter_url = \"https://twitter.com/{}\".format(account_name)\n",
    "page = urllib.request.urlopen(twitter_url)\n",
    "soup = bs4.BeautifulSoup(page, 'lxml')\n",
    "\n",
    "li_stats = soup.find('li', {'class': 'ProfileNav-item ProfileNav-item--followers'})\n",
    "li_stats.find('a').get('title')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": true
   },
   "source": [
    "### Q. Write a Python program to scrap number of tweets of a given Twitter account."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Enter your Twitter account name: shahp7575\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "'554 Tweets'"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "account_name = input(\"Enter your Twitter account name: \")\n",
    "twitter_url = \"https://twitter.com/{}\".format(account_name)\n",
    "page = urllib.request.urlopen(twitter_url)\n",
    "soup = bs4.BeautifulSoup(page, 'lxml')\n",
    "\n",
    "li_stats = soup.find('li', {'class': 'ProfileNav-item--tweets'})\n",
    "li_stats.find('a').get('title')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to display the date, days, title, city, country of next 25 Hackevents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 123,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 ) Oct 11 , Thu-Sat , chainPORT hackathon 2018 - Antwerp Edition at Antwerpen Belgium\n",
      "2 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "3 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "4 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "5 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "6 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "7 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "8 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "9 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "10 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "11 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "12 ) Oct 12 , Fri-Sun , Blackforest Hackathon at Offenburg Germany\n",
      "13 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "14 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "15 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "16 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "17 ) Oct 12 , Fri , UChicago 4th Annual South Side Civic Scopeathon at Chicago USA\n",
      "18 ) Oct 12 , Fri-Sat , Chainport Hackathon - USC Marshall Global Supply Chain Mgmt at Los Angeles USA\n",
      "19 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "20 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "21 ) Oct 12 , Fri-Sun , Hackathon Bremen 2018 vol. 2.0 at Bremen Germany\n",
      "22 ) Oct 12 , Fri-Sat , Chainport Hackathon - USC Marshall Global Supply Chain Mgmt at Los Angeles USA\n",
      "23 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "24 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n",
      "25 ) Oct 12 , Fri-Sun , Future of Insurance at Düsseldorf Germany\n"
     ]
    }
   ],
   "source": [
    "import requests\n",
    "url = \"https://www.hackevents.co/hackathons\"\n",
    "r = requests.get(url)\n",
    "soup= bs4.BeautifulSoup(r.text, 'lxml')\n",
    "\n",
    "hackathon_class = soup.findAll('div', {'class': 'hackathon'})\n",
    "\n",
    "for i,f in enumerate(hackathon_class, 1):\n",
    "\n",
    "    date_month = f.find('div', {'class': 'date-month'}).text.strip()\n",
    "    date_day = f.find('div', {'class': 'date-day-number'}).text.strip()\n",
    "    date_weekdays = f.find('div', {'class': 'date-week-days'}).text.strip()\n",
    "    event_title = f.find('a', {'class': 'title'}).text.strip()\n",
    "    event_city = f.find('p', {'class': 'location-and-date'}).find('span', {'class': 'city'}).text.strip()\n",
    "    event_country = f.find('p', {'class': 'location-and-date'}).find('span', {'class': 'country'}).text.strip()\n",
    "\n",
    "    print(i,\")\", date_month, date_day, \",\", date_weekdays, \",\", event_title, \"at\", event_city, event_country)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to download IMDB's Top 250 data (movie name, Initial release, director name and stars)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 122,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 ) The Shawshank Redemption (1994) , Starring:  Frank Darabont (dir.), Tim Robbins, Morgan Freeman , Rating:  9.2\n",
      "2 ) The Godfather (1972) , Starring:  Francis Ford Coppola (dir.), Marlon Brando, Al Pacino , Rating:  9.2\n",
      "3 ) The Godfather: Part II (1974) , Starring:  Francis Ford Coppola (dir.), Al Pacino, Robert De Niro , Rating:  9.0\n",
      "4 ) The Dark Knight (2008) , Starring:  Christopher Nolan (dir.), Christian Bale, Heath Ledger , Rating:  9.0\n",
      "5 ) 12 Angry Men (1957) , Starring:  Sidney Lumet (dir.), Henry Fonda, Lee J. Cobb , Rating:  8.9\n",
      "6 ) Schindler's List (1993) , Starring:  Steven Spielberg (dir.), Liam Neeson, Ralph Fiennes , Rating:  8.9\n",
      "7 ) The Lord of the Rings: The Return of the King (2003) , Starring:  Peter Jackson (dir.), Elijah Wood, Viggo Mortensen , Rating:  8.9\n",
      "8 ) Pulp Fiction (1994) , Starring:  Quentin Tarantino (dir.), John Travolta, Uma Thurman , Rating:  8.9\n",
      "9 ) The Good, the Bad and the Ugly (1966) , Starring:  Sergio Leone (dir.), Clint Eastwood, Eli Wallach , Rating:  8.8\n",
      "10 ) Fight Club (1999) , Starring:  David Fincher (dir.), Brad Pitt, Edward Norton , Rating:  8.8\n",
      "11 ) The Lord of the Rings: The Fellowship of the Ring (2001) , Starring:  Peter Jackson (dir.), Elijah Wood, Ian McKellen , Rating:  8.8\n",
      "12 ) Forrest Gump (1994) , Starring:  Robert Zemeckis (dir.), Tom Hanks, Robin Wright , Rating:  8.7\n",
      "13 ) Star Wars: Episode V - The Empire Strikes Back (1980) , Starring:  Irvin Kershner (dir.), Mark Hamill, Harrison Ford , Rating:  8.7\n",
      "14 ) Inception (2010) , Starring:  Christopher Nolan (dir.), Leonardo DiCaprio, Joseph Gordon-Levitt , Rating:  8.7\n",
      "15 ) The Lord of the Rings: The Two Towers (2002) , Starring:  Peter Jackson (dir.), Elijah Wood, Ian McKellen , Rating:  8.7\n",
      "16 ) One Flew Over the Cuckoo's Nest (1975) , Starring:  Milos Forman (dir.), Jack Nicholson, Louise Fletcher , Rating:  8.7\n",
      "17 ) Goodfellas (1990) , Starring:  Martin Scorsese (dir.), Robert De Niro, Ray Liotta , Rating:  8.7\n",
      "18 ) The Matrix (1999) , Starring:  Lana Wachowski (dir.), Keanu Reeves, Laurence Fishburne , Rating:  8.6\n",
      "19 ) Seven Samurai (1954) , Starring:  Akira Kurosawa (dir.), Toshirô Mifune, Takashi Shimura , Rating:  8.6\n",
      "20 ) City of God (2002) , Starring:  Fernando Meirelles (dir.), Alexandre Rodrigues, Leandro Firmino , Rating:  8.6\n",
      "21 ) Se7en (1995) , Starring:  David Fincher (dir.), Morgan Freeman, Brad Pitt , Rating:  8.6\n",
      "22 ) Star Wars: Episode IV - A New Hope (1977) , Starring:  George Lucas (dir.), Mark Hamill, Harrison Ford , Rating:  8.6\n",
      "23 ) The Silence of the Lambs (1991) , Starring:  Jonathan Demme (dir.), Jodie Foster, Anthony Hopkins , Rating:  8.6\n",
      "24 ) It's a Wonderful Life (1946) , Starring:  Frank Capra (dir.), James Stewart, Donna Reed , Rating:  8.6\n",
      "25 ) Life Is Beautiful (1997) , Starring:  Roberto Benigni (dir.), Roberto Benigni, Nicoletta Braschi , Rating:  8.6\n",
      "26 ) The Usual Suspects (1995) , Starring:  Bryan Singer (dir.), Kevin Spacey, Gabriel Byrne , Rating:  8.5\n",
      "27 ) Spirited Away (2001) , Starring:  Hayao Miyazaki (dir.), Daveigh Chase, Suzanne Pleshette , Rating:  8.5\n",
      "28 ) Saving Private Ryan (1998) , Starring:  Steven Spielberg (dir.), Tom Hanks, Matt Damon , Rating:  8.5\n",
      "29 ) Léon: The Professional (1994) , Starring:  Luc Besson (dir.), Jean Reno, Gary Oldman , Rating:  8.5\n",
      "30 ) The Green Mile (1999) , Starring:  Frank Darabont (dir.), Tom Hanks, Michael Clarke Duncan , Rating:  8.5\n",
      "31 ) Interstellar (2014) , Starring:  Christopher Nolan (dir.), Matthew McConaughey, Anne Hathaway , Rating:  8.5\n",
      "32 ) American History X (1998) , Starring:  Tony Kaye (dir.), Edward Norton, Edward Furlong , Rating:  8.5\n",
      "33 ) Psycho (1960) , Starring:  Alfred Hitchcock (dir.), Anthony Perkins, Janet Leigh , Rating:  8.5\n",
      "34 ) City Lights (1931) , Starring:  Charles Chaplin (dir.), Charles Chaplin, Virginia Cherrill , Rating:  8.5\n",
      "35 ) Once Upon a Time in the West (1968) , Starring:  Sergio Leone (dir.), Henry Fonda, Charles Bronson , Rating:  8.5\n",
      "36 ) Casablanca (1942) , Starring:  Michael Curtiz (dir.), Humphrey Bogart, Ingrid Bergman , Rating:  8.5\n",
      "37 ) Modern Times (1936) , Starring:  Charles Chaplin (dir.), Charles Chaplin, Paulette Goddard , Rating:  8.5\n",
      "38 ) The Intouchables (2011) , Starring:  Olivier Nakache (dir.), François Cluzet, Omar Sy , Rating:  8.5\n",
      "39 ) The Pianist (2002) , Starring:  Roman Polanski (dir.), Adrien Brody, Thomas Kretschmann , Rating:  8.5\n",
      "40 ) The Departed (2006) , Starring:  Martin Scorsese (dir.), Leonardo DiCaprio, Matt Damon , Rating:  8.5\n",
      "41 ) Terminator 2 (1991) , Starring:  James Cameron (dir.), Arnold Schwarzenegger, Linda Hamilton , Rating:  8.5\n",
      "42 ) Back to the Future (1985) , Starring:  Robert Zemeckis (dir.), Michael J. Fox, Christopher Lloyd , Rating:  8.5\n",
      "43 ) Rear Window (1954) , Starring:  Alfred Hitchcock (dir.), James Stewart, Grace Kelly , Rating:  8.5\n",
      "44 ) Whiplash (2014) , Starring:  Damien Chazelle (dir.), Miles Teller, J.K. Simmons , Rating:  8.5\n",
      "45 ) Raiders of the Lost Ark (1981) , Starring:  Steven Spielberg (dir.), Harrison Ford, Karen Allen , Rating:  8.5\n",
      "46 ) Gladiator (2000) , Starring:  Ridley Scott (dir.), Russell Crowe, Joaquin Phoenix , Rating:  8.5\n",
      "47 ) The Lion King (1994) , Starring:  Roger Allers (dir.), Matthew Broderick, Jeremy Irons , Rating:  8.5\n",
      "48 ) The Prestige (2006) , Starring:  Christopher Nolan (dir.), Christian Bale, Hugh Jackman , Rating:  8.5\n",
      "49 ) Avengers: Infinity War (2018) , Starring:  Anthony Russo (dir.), Robert Downey Jr., Chris Hemsworth , Rating:  8.5\n",
      "50 ) Memento (2000) , Starring:  Christopher Nolan (dir.), Guy Pearce, Carrie-Anne Moss , Rating:  8.4\n",
      "51 ) Apocalypse Now (1979) , Starring:  Francis Ford Coppola (dir.), Martin Sheen, Marlon Brando , Rating:  8.4\n",
      "52 ) Alien (1979) , Starring:  Ridley Scott (dir.), Sigourney Weaver, Tom Skerritt , Rating:  8.4\n",
      "53 ) The Great Dictator (1940) , Starring:  Charles Chaplin (dir.), Charles Chaplin, Paulette Goddard , Rating:  8.4\n",
      "54 ) Cinema Paradiso (1988) , Starring:  Giuseppe Tornatore (dir.), Philippe Noiret, Enzo Cannavale , Rating:  8.4\n",
      "55 ) Grave of the Fireflies (1988) , Starring:  Isao Takahata (dir.), Tsutomu Tatsumi, Ayano Shiraishi , Rating:  8.4\n",
      "56 ) Sunset Blvd. (1950) , Starring:  Billy Wilder (dir.), William Holden, Gloria Swanson , Rating:  8.4\n",
      "57 ) The Lives of Others (2006) , Starring:  Florian Henckel von Donnersmarck (dir.), Ulrich Mühe, Martina Gedeck , Rating:  8.4\n",
      "58 ) Dr. Strangelove or: How I Learned to Stop Worrying and Love the Bomb (1964) , Starring:  Stanley Kubrick (dir.), Peter Sellers, George C. Scott , Rating:  8.4\n",
      "59 ) Paths of Glory (1957) , Starring:  Stanley Kubrick (dir.), Kirk Douglas, Ralph Meeker , Rating:  8.4\n",
      "60 ) The Shining (1980) , Starring:  Stanley Kubrick (dir.), Jack Nicholson, Shelley Duvall , Rating:  8.4\n",
      "61 ) Django Unchained (2012) , Starring:  Quentin Tarantino (dir.), Jamie Foxx, Christoph Waltz , Rating:  8.4\n",
      "62 ) WALL·E (2008) , Starring:  Andrew Stanton (dir.), Ben Burtt, Elissa Knight , Rating:  8.4\n",
      "63 ) Princess Mononoke (1997) , Starring:  Hayao Miyazaki (dir.), Yôji Matsuda, Yuriko Ishida , Rating:  8.4\n",
      "64 ) Witness for the Prosecution (1957) , Starring:  Billy Wilder (dir.), Tyrone Power, Marlene Dietrich , Rating:  8.4\n",
      "65 ) Coco (2017) , Starring:  Lee Unkrich (dir.), Anthony Gonzalez, Gael García Bernal , Rating:  8.4\n",
      "66 ) American Beauty (1999) , Starring:  Sam Mendes (dir.), Kevin Spacey, Annette Bening , Rating:  8.4\n",
      "67 ) The Dark Knight Rises (2012) , Starring:  Christopher Nolan (dir.), Christian Bale, Tom Hardy , Rating:  8.4\n",
      "68 ) Oldeuboi (2003) , Starring:  Chan-wook Park (dir.), Min-sik Choi, Ji-tae Yu , Rating:  8.3\n",
      "69 ) Aliens (1986) , Starring:  James Cameron (dir.), Sigourney Weaver, Michael Biehn , Rating:  8.3\n",
      "70 ) Once Upon a Time in America (1984) , Starring:  Sergio Leone (dir.), Robert De Niro, James Woods , Rating:  8.3\n",
      "71 ) Das Boot (1981) , Starring:  Wolfgang Petersen (dir.), Jürgen Prochnow, Herbert Grönemeyer , Rating:  8.3\n",
      "72 ) Citizen Kane (1941) , Starring:  Orson Welles (dir.), Orson Welles, Joseph Cotten , Rating:  8.3\n",
      "73 ) Braveheart (1995) , Starring:  Mel Gibson (dir.), Mel Gibson, Sophie Marceau , Rating:  8.3\n",
      "74 ) Vertigo (1958) , Starring:  Alfred Hitchcock (dir.), James Stewart, Kim Novak , Rating:  8.3\n",
      "75 ) North by Northwest (1959) , Starring:  Alfred Hitchcock (dir.), Cary Grant, Eva Marie Saint , Rating:  8.3\n",
      "76 ) Reservoir Dogs (1992) , Starring:  Quentin Tarantino (dir.), Harvey Keitel, Tim Roth , Rating:  8.3\n",
      "77 ) Your Name. (2016) , Starring:  Makoto Shinkai (dir.), Ryûnosuke Kamiki, Mone Kamishiraishi , Rating:  8.3\n",
      "78 ) Star Wars: Episode VI - Return of the Jedi (1983) , Starring:  Richard Marquand (dir.), Mark Hamill, Harrison Ford , Rating:  8.3\n",
      "79 ) M (1931) , Starring:  Fritz Lang (dir.), Peter Lorre, Ellen Widmann , Rating:  8.3\n",
      "80 ) Dangal (2016) , Starring:  Nitesh Tiwari (dir.), Aamir Khan, Sakshi Tanwar , Rating:  8.3\n",
      "81 ) Requiem for a Dream (2000) , Starring:  Darren Aronofsky (dir.), Ellen Burstyn, Jared Leto , Rating:  8.3\n",
      "82 ) Amadeus (1984) , Starring:  Milos Forman (dir.), F. Murray Abraham, Tom Hulce , Rating:  8.3\n",
      "83 ) Taare Zameen Par (2007) , Starring:  Aamir Khan (dir.), Darsheel Safary, Aamir Khan , Rating:  8.3\n",
      "84 ) Lawrence of Arabia (1962) , Starring:  David Lean (dir.), Peter O'Toole, Alec Guinness , Rating:  8.3\n",
      "85 ) Eternal Sunshine of the Spotless Mind (2004) , Starring:  Michel Gondry (dir.), Jim Carrey, Kate Winslet , Rating:  8.3\n",
      "86 ) A Clockwork Orange (1971) , Starring:  Stanley Kubrick (dir.), Malcolm McDowell, Patrick Magee , Rating:  8.3\n",
      "87 ) Amélie (2001) , Starring:  Jean-Pierre Jeunet (dir.), Audrey Tautou, Mathieu Kassovitz , Rating:  8.3\n",
      "88 ) Double Indemnity (1944) , Starring:  Billy Wilder (dir.), Fred MacMurray, Barbara Stanwyck , Rating:  8.3\n",
      "89 ) 3 Idiots (2009) , Starring:  Rajkumar Hirani (dir.), Aamir Khan, Madhavan , Rating:  8.3\n",
      "90 ) 2001: A Space Odyssey (1968) , Starring:  Stanley Kubrick (dir.), Keir Dullea, Gary Lockwood , Rating:  8.3\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "91 ) Toy Story (1995) , Starring:  John Lasseter (dir.), Tom Hanks, Tim Allen , Rating:  8.3\n",
      "92 ) Taxi Driver (1976) , Starring:  Martin Scorsese (dir.), Robert De Niro, Jodie Foster , Rating:  8.3\n",
      "93 ) Singin' in the Rain (1952) , Starring:  Stanley Donen (dir.), Gene Kelly, Donald O'Connor , Rating:  8.3\n",
      "94 ) Full Metal Jacket (1987) , Starring:  Stanley Kubrick (dir.), Matthew Modine, R. Lee Ermey , Rating:  8.3\n",
      "95 ) Inglourious Basterds (2009) , Starring:  Quentin Tarantino (dir.), Brad Pitt, Diane Kruger , Rating:  8.3\n",
      "96 ) To Kill a Mockingbird (1962) , Starring:  Robert Mulligan (dir.), Gregory Peck, John Megna , Rating:  8.3\n",
      "97 ) Bicycle Thieves (1948) , Starring:  Vittorio De Sica (dir.), Lamberto Maggiorani, Enzo Staiola , Rating:  8.3\n",
      "98 ) The Kid (1921) , Starring:  Charles Chaplin (dir.), Charles Chaplin, Edna Purviance , Rating:  8.3\n",
      "99 ) The Sting (1973) , Starring:  George Roy Hill (dir.), Paul Newman, Robert Redford , Rating:  8.3\n",
      "100 ) Toy Story 3 (2010) , Starring:  Lee Unkrich (dir.), Tom Hanks, Tim Allen , Rating:  8.3\n",
      "101 ) Good Will Hunting (1997) , Starring:  Gus Van Sant (dir.), Robin Williams, Matt Damon , Rating:  8.3\n",
      "102 ) The Hunt (2012) , Starring:  Thomas Vinterberg (dir.), Mads Mikkelsen, Thomas Bo Larsen , Rating:  8.3\n",
      "103 ) Snatch (2000) , Starring:  Guy Ritchie (dir.), Jason Statham, Brad Pitt , Rating:  8.3\n",
      "104 ) Monty Python and the Holy Grail (1975) , Starring:  Terry Gilliam (dir.), Graham Chapman, John Cleese , Rating:  8.2\n",
      "105 ) Scarface (1983) , Starring:  Brian De Palma (dir.), Al Pacino, Michelle Pfeiffer , Rating:  8.2\n",
      "106 ) For a Few Dollars More (1965) , Starring:  Sergio Leone (dir.), Clint Eastwood, Lee Van Cleef , Rating:  8.2\n",
      "107 ) The Apartment (1960) , Starring:  Billy Wilder (dir.), Jack Lemmon, Shirley MacLaine , Rating:  8.2\n",
      "108 ) L.A. Confidential (1997) , Starring:  Curtis Hanson (dir.), Kevin Spacey, Russell Crowe , Rating:  8.2\n",
      "109 ) Metropolis (1927) , Starring:  Fritz Lang (dir.), Brigitte Helm, Alfred Abel , Rating:  8.2\n",
      "110 ) A Separation (2011) , Starring:  Asghar Farhadi (dir.), Payman Maadi, Leila Hatami , Rating:  8.2\n",
      "111 ) Indiana Jones and the Last Crusade (1989) , Starring:  Steven Spielberg (dir.), Harrison Ford, Sean Connery , Rating:  8.2\n",
      "112 ) Rashômon (1950) , Starring:  Akira Kurosawa (dir.), Toshirô Mifune, Machiko Kyô , Rating:  8.2\n",
      "113 ) Up (2009) , Starring:  Pete Docter (dir.), Edward Asner, Jordan Nagai , Rating:  8.2\n",
      "114 ) All About Eve (1950) , Starring:  Joseph L. Mankiewicz (dir.), Bette Davis, Anne Baxter , Rating:  8.2\n",
      "115 ) Yojimbo (1961) , Starring:  Akira Kurosawa (dir.), Toshirô Mifune, Eijirô Tôno , Rating:  8.2\n",
      "116 ) Batman Begins (2005) , Starring:  Christopher Nolan (dir.), Christian Bale, Michael Caine , Rating:  8.2\n",
      "117 ) Some Like It Hot (1959) , Starring:  Billy Wilder (dir.), Marilyn Monroe, Tony Curtis , Rating:  8.2\n",
      "118 ) Unforgiven (1992) , Starring:  Clint Eastwood (dir.), Clint Eastwood, Gene Hackman , Rating:  8.2\n",
      "119 ) Downfall (2004) , Starring:  Oliver Hirschbiegel (dir.), Bruno Ganz, Alexandra Maria Lara , Rating:  8.2\n",
      "120 ) The Treasure of the Sierra Madre (1948) , Starring:  John Huston (dir.), Humphrey Bogart, Walter Huston , Rating:  8.2\n",
      "121 ) Die Hard (1988) , Starring:  John McTiernan (dir.), Bruce Willis, Alan Rickman , Rating:  8.2\n",
      "122 ) Heat (1995) , Starring:  Michael Mann (dir.), Al Pacino, Robert De Niro , Rating:  8.2\n",
      "123 ) Ikiru (1952) , Starring:  Akira Kurosawa (dir.), Takashi Shimura, Nobuo Kaneko , Rating:  8.2\n",
      "124 ) Raging Bull (1980) , Starring:  Martin Scorsese (dir.), Robert De Niro, Cathy Moriarty , Rating:  8.2\n",
      "125 ) Incendies (2010) , Starring:  Denis Villeneuve (dir.), Lubna Azabal, Mélissa Désormeaux-Poulin , Rating:  8.2\n",
      "126 ) The Great Escape (1963) , Starring:  John Sturges (dir.), Steve McQueen, James Garner , Rating:  8.2\n",
      "127 ) Babam ve Oglum (2005) , Starring:  Çagan Irmak (dir.), Çetin Tekindor, Fikret Kuskan , Rating:  8.2\n",
      "128 ) Children of Heaven (1997) , Starring:  Majid Majidi (dir.), Mohammad Amir Naji, Amir Farrokh Hashemian , Rating:  8.2\n",
      "129 ) Pan's Labyrinth (2006) , Starring:  Guillermo del Toro (dir.), Ivana Baquero, Ariadna Gil , Rating:  8.2\n",
      "130 ) The Third Man (1949) , Starring:  Carol Reed (dir.), Orson Welles, Joseph Cotten , Rating:  8.2\n",
      "131 ) Chinatown (1974) , Starring:  Roman Polanski (dir.), Jack Nicholson, Faye Dunaway , Rating:  8.2\n",
      "132 ) My Neighbor Totoro (1988) , Starring:  Hayao Miyazaki (dir.), Hitoshi Takagi, Noriko Hidaka , Rating:  8.2\n",
      "133 ) Ran (1985) , Starring:  Akira Kurosawa (dir.), Tatsuya Nakadai, Akira Terao , Rating:  8.2\n",
      "134 ) Howl's Moving Castle (2004) , Starring:  Hayao Miyazaki (dir.), Chieko Baishô, Takuya Kimura , Rating:  8.2\n",
      "135 ) Judgment at Nuremberg (1961) , Starring:  Stanley Kramer (dir.), Spencer Tracy, Burt Lancaster , Rating:  8.2\n",
      "136 ) The Secret in Their Eyes (2009) , Starring:  Juan José Campanella (dir.), Ricardo Darín, Soledad Villamil , Rating:  8.2\n",
      "137 ) The Gold Rush (1925) , Starring:  Charles Chaplin (dir.), Charles Chaplin, Mack Swain , Rating:  8.2\n",
      "138 ) The Bridge on the River Kwai (1957) , Starring:  David Lean (dir.), William Holden, Alec Guinness , Rating:  8.2\n",
      "139 ) Three Billboards Outside Ebbing, Missouri (2017) , Starring:  Martin McDonagh (dir.), Frances McDormand, Woody Harrelson , Rating:  8.2\n",
      "140 ) On the Waterfront (1954) , Starring:  Elia Kazan (dir.), Marlon Brando, Karl Malden , Rating:  8.2\n",
      "141 ) A Beautiful Mind (2001) , Starring:  Ron Howard (dir.), Russell Crowe, Ed Harris , Rating:  8.2\n",
      "142 ) Lock, Stock and Two Smoking Barrels (1998) , Starring:  Guy Ritchie (dir.), Jason Flemyng, Dexter Fletcher , Rating:  8.2\n",
      "143 ) Casino (1995) , Starring:  Martin Scorsese (dir.), Robert De Niro, Sharon Stone , Rating:  8.2\n",
      "144 ) The Seventh Seal (1957) , Starring:  Ingmar Bergman (dir.), Max von Sydow, Gunnar Björnstrand , Rating:  8.2\n",
      "145 ) Inside Out (2015) , Starring:  Pete Docter (dir.), Amy Poehler, Bill Hader , Rating:  8.2\n",
      "146 ) Room (2015) , Starring:  Lenny Abrahamson (dir.), Brie Larson, Jacob Tremblay , Rating:  8.1\n",
      "147 ) The Elephant Man (1980) , Starring:  David Lynch (dir.), Anthony Hopkins, John Hurt , Rating:  8.1\n",
      "148 ) Mr. Smith Goes to Washington (1939) , Starring:  Frank Capra (dir.), James Stewart, Jean Arthur , Rating:  8.1\n",
      "149 ) The Wolf of Wall Street (2013) , Starring:  Martin Scorsese (dir.), Leonardo DiCaprio, Jonah Hill , Rating:  8.1\n",
      "150 ) V for Vendetta (2005) , Starring:  James McTeigue (dir.), Hugo Weaving, Natalie Portman , Rating:  8.1\n",
      "151 ) Warrior (2011) , Starring:  Gavin O'Connor (dir.), Tom Hardy, Nick Nolte , Rating:  8.1\n",
      "152 ) Blade Runner (1982) , Starring:  Ridley Scott (dir.), Harrison Ford, Rutger Hauer , Rating:  8.1\n",
      "153 ) The General (1926) , Starring:  Clyde Bruckman (dir.), Buster Keaton, Marion Mack , Rating:  8.1\n",
      "154 ) Wild Strawberries (1957) , Starring:  Ingmar Bergman (dir.), Victor Sjöström, Bibi Andersson , Rating:  8.1\n",
      "155 ) Dial M for Murder (1954) , Starring:  Alfred Hitchcock (dir.), Ray Milland, Grace Kelly , Rating:  8.1\n",
      "156 ) Trainspotting (1996) , Starring:  Danny Boyle (dir.), Ewan McGregor, Ewen Bremner , Rating:  8.1\n",
      "157 ) No Country for Old Men (2007) , Starring:  Ethan Coen (dir.), Tommy Lee Jones, Javier Bardem , Rating:  8.1\n",
      "158 ) There Will Be Blood (2007) , Starring:  Paul Thomas Anderson (dir.), Daniel Day-Lewis, Paul Dano , Rating:  8.1\n",
      "159 ) Gone with the Wind (1939) , Starring:  Victor Fleming (dir.), Clark Gable, Vivien Leigh , Rating:  8.1\n",
      "160 ) The Sixth Sense (1999) , Starring:  M. Night Shyamalan (dir.), Bruce Willis, Haley Joel Osment , Rating:  8.1\n",
      "161 ) Fargo (1996) , Starring:  Joel Coen (dir.), William H. Macy, Frances McDormand , Rating:  8.1\n",
      "162 ) The Thing (1982) , Starring:  John Carpenter (dir.), Kurt Russell, Wilford Brimley , Rating:  8.1\n",
      "163 ) Gran Torino (2008) , Starring:  Clint Eastwood (dir.), Clint Eastwood, Bee Vang , Rating:  8.1\n",
      "164 ) The Deer Hunter (1978) , Starring:  Michael Cimino (dir.), Robert De Niro, Christopher Walken , Rating:  8.1\n",
      "165 ) Finding Nemo (2003) , Starring:  Andrew Stanton (dir.), Albert Brooks, Ellen DeGeneres , Rating:  8.1\n",
      "166 ) Sherlock Jr. (1924) , Starring:  Buster Keaton (dir.), Buster Keaton, Kathryn McGuire , Rating:  8.1\n",
      "167 ) Come and See (1985) , Starring:  Elem Klimov (dir.), Aleksey Kravchenko, Olga Mironova , Rating:  8.1\n",
      "168 ) The Big Lebowski (1998) , Starring:  Joel Coen (dir.), Jeff Bridges, John Goodman , Rating:  8.1\n",
      "169 ) Kill Bill: Vol. 1 (2003) , Starring:  Quentin Tarantino (dir.), Uma Thurman, David Carradine , Rating:  8.1\n",
      "170 ) Shutter Island (2010) , Starring:  Martin Scorsese (dir.), Leonardo DiCaprio, Emily Mortimer , Rating:  8.1\n",
      "171 ) Cool Hand Luke (1967) , Starring:  Stuart Rosenberg (dir.), Paul Newman, George Kennedy , Rating:  8.1\n",
      "172 ) Rebecca (1940) , Starring:  Alfred Hitchcock (dir.), Laurence Olivier, Joan Fontaine , Rating:  8.1\n",
      "173 ) Tokyo Story (1953) , Starring:  Yasujirô Ozu (dir.), Chishû Ryû, Chieko Higashiyama , Rating:  8.1\n",
      "174 ) Hacksaw Ridge (2016) , Starring:  Mel Gibson (dir.), Andrew Garfield, Sam Worthington , Rating:  8.1\n",
      "175 ) Sunrise (1927) , Starring:  F.W. Murnau (dir.), George O'Brien, Janet Gaynor , Rating:  8.1\n",
      "176 ) Mary and Max (2009) , Starring:  Adam Elliot (dir.), Toni Collette, Philip Seymour Hoffman , Rating:  8.1\n",
      "177 ) How to Train Your Dragon (2010) , Starring:  Dean DeBlois (dir.), Jay Baruchel, Gerard Butler , Rating:  8.1\n",
      "178 ) Gone Girl (2014) , Starring:  David Fincher (dir.), Ben Affleck, Rosamund Pike , Rating:  8.1\n",
      "179 ) Relatos salvajes (2014) , Starring:  Damián Szifron (dir.), Darío Grandinetti, María Marull , Rating:  8.1\n",
      "180 ) Jurassic Park (1993) , Starring:  Steven Spielberg (dir.), Sam Neill, Laura Dern , Rating:  8.1\n",
      "181 ) Into the Wild (2007) , Starring:  Sean Penn (dir.), Emile Hirsch, Vince Vaughn , Rating:  8.1\n",
      "182 ) Life of Brian (1979) , Starring:  Terry Jones (dir.), Graham Chapman, John Cleese , Rating:  8.1\n",
      "183 ) The Bandit (1996) , Starring:  Yavuz Turgul (dir.), Sener Sen, Ugur Yücel , Rating:  8.1\n",
      "184 ) It Happened One Night (1934) , Starring:  Frank Capra (dir.), Clark Gable, Claudette Colbert , Rating:  8.1\n",
      "185 ) In the Name of the Father (1993) , Starring:  Jim Sheridan (dir.), Daniel Day-Lewis, Pete Postlethwaite , Rating:  8.1\n",
      "186 ) Platoon (1986) , Starring:  Oliver Stone (dir.), Charlie Sheen, Tom Berenger , Rating:  8.1\n",
      "187 ) The Grand Budapest Hotel (2014) , Starring:  Wes Anderson (dir.), Ralph Fiennes, F. Murray Abraham , Rating:  8.1\n",
      "188 ) Stand by Me (1986) , Starring:  Rob Reiner (dir.), Wil Wheaton, River Phoenix , Rating:  8.1\n",
      "189 ) Network (1976) , Starring:  Sidney Lumet (dir.), Faye Dunaway, William Holden , Rating:  8.1\n",
      "190 ) The Truman Show (1998) , Starring:  Peter Weir (dir.), Jim Carrey, Ed Harris , Rating:  8.1\n",
      "191 ) Stalker (1979) , Starring:  Andrei Tarkovsky (dir.), Alisa Freyndlikh, Aleksandr Kaydanovskiy , Rating:  8.1\n",
      "192 ) Hotel Rwanda (2004) , Starring:  Terry George (dir.), Don Cheadle, Sophie Okonedo , Rating:  8.1\n",
      "193 ) Andrei Rublev (1966) , Starring:  Andrei Tarkovsky (dir.), Anatoliy Solonitsyn, Ivan Lapikov , Rating:  8.1\n",
      "194 ) Ben-Hur (1959) , Starring:  William Wyler (dir.), Charlton Heston, Jack Hawkins , Rating:  8.1\n",
      "195 ) Persona (1966) , Starring:  Ingmar Bergman (dir.), Bibi Andersson, Liv Ullmann , Rating:  8.1\n",
      "196 ) Memories of Murder (2003) , Starring:  Joon-ho Bong (dir.), Kang-ho Song, Sang-kyung Kim , Rating:  8.1\n",
      "197 ) 12 Years a Slave (2013) , Starring:  Steve McQueen (dir.), Chiwetel Ejiofor, Michael Kenneth Williams , Rating:  8.1\n",
      "198 ) The Wages of Fear (1953) , Starring:  Henri-Georges Clouzot (dir.), Yves Montand, Charles Vanel , Rating:  8.1\n",
      "199 ) Rang De Basanti (2006) , Starring:  Rakeysh Omprakash Mehra (dir.), Aamir Khan, Soha Ali Khan , Rating:  8.1\n",
      "200 ) Million Dollar Baby (2004) , Starring:  Clint Eastwood (dir.), Hilary Swank, Clint Eastwood , Rating:  8.1\n",
      "201 ) The Passion of Joan of Arc (1928) , Starring:  Carl Theodor Dreyer (dir.), Maria Falconetti, Eugene Silvain , Rating:  8.1\n",
      "202 ) Rush (2013) , Starring:  Ron Howard (dir.), Daniel Brühl, Chris Hemsworth , Rating:  8.1\n",
      "203 ) Mad Max: Fury Road (2015) , Starring:  George Miller (dir.), Tom Hardy, Charlize Theron , Rating:  8.1\n",
      "204 ) Before Sunrise (1995) , Starring:  Richard Linklater (dir.), Ethan Hawke, Julie Delpy , Rating:  8.1\n",
      "205 ) The 400 Blows (1959) , Starring:  François Truffaut (dir.), Jean-Pierre Léaud, Albert Rémy , Rating:  8.1\n",
      "206 ) Spotlight (2015) , Starring:  Tom McCarthy (dir.), Mark Ruffalo, Michael Keaton , Rating:  8.1\n",
      "207 ) Logan (2017) , Starring:  James Mangold (dir.), Hugh Jackman, Patrick Stewart , Rating:  8.1\n",
      "208 ) Amores Perros (2000) , Starring:  Alejandro G. Iñárritu (dir.), Emilio Echevarría, Gael García Bernal , Rating:  8.1\n",
      "209 ) Hachi: A Dog's Tale (2009) , Starring:  Lasse Hallström (dir.), Richard Gere, Joan Allen , Rating:  8.1\n",
      "210 ) Prisoners (2013) , Starring:  Denis Villeneuve (dir.), Hugh Jackman, Jake Gyllenhaal , Rating:  8.1\n",
      "211 ) The Princess Bride (1987) , Starring:  Rob Reiner (dir.), Cary Elwes, Mandy Patinkin , Rating:  8.1\n",
      "212 ) Nausicaä of the Valley of the Wind (1984) , Starring:  Hayao Miyazaki (dir.), Sumi Shimamoto, Mahito Tsujimura , Rating:  8.0\n",
      "213 ) Butch Cassidy and the Sundance Kid (1969) , Starring:  George Roy Hill (dir.), Paul Newman, Robert Redford , Rating:  8.0\n",
      "214 ) Catch Me If You Can (2002) , Starring:  Steven Spielberg (dir.), Leonardo DiCaprio, Tom Hanks , Rating:  8.0\n",
      "215 ) Harry Potter and the Deathly Hallows: Part 2 (2011) , Starring:  David Yates (dir.), Daniel Radcliffe, Emma Watson , Rating:  8.0\n",
      "216 ) Rocky (1976) , Starring:  John G. Avildsen (dir.), Sylvester Stallone, Talia Shire , Rating:  8.0\n",
      "217 ) Barry Lyndon (1975) , Starring:  Stanley Kubrick (dir.), Ryan O'Neal, Marisa Berenson , Rating:  8.0\n",
      "218 ) Monsters, Inc. (2001) , Starring:  Pete Docter (dir.), Billy Crystal, John Goodman , Rating:  8.0\n",
      "219 ) The Maltese Falcon (1941) , Starring:  John Huston (dir.), Humphrey Bogart, Mary Astor , Rating:  8.0\n",
      "220 ) The Grapes of Wrath (1940) , Starring:  John Ford (dir.), Henry Fonda, Jane Darwell , Rating:  8.0\n",
      "221 ) Donnie Darko (2001) , Starring:  Richard Kelly (dir.), Jake Gyllenhaal, Jena Malone , Rating:  8.0\n",
      "222 ) Diabolique (1955) , Starring:  Henri-Georges Clouzot (dir.), Simone Signoret, Véra Clouzot , Rating:  8.0\n",
      "223 ) The Terminator (1984) , Starring:  James Cameron (dir.), Arnold Schwarzenegger, Linda Hamilton , Rating:  8.0\n",
      "224 ) Gandhi (1982) , Starring:  Richard Attenborough (dir.), Ben Kingsley, John Gielgud , Rating:  8.0\n",
      "225 ) Blade Runner 2049 (2017) , Starring:  Denis Villeneuve (dir.), Harrison Ford, Ryan Gosling , Rating:  8.0\n",
      "226 ) Dead Poets Society (1989) , Starring:  Peter Weir (dir.), Robin Williams, Robert Sean Leonard , Rating:  8.0\n",
      "227 ) La La Land (2016) , Starring:  Damien Chazelle (dir.), Ryan Gosling, Emma Stone , Rating:  8.0\n",
      "228 ) The Nights of Cabiria (1957) , Starring:  Federico Fellini (dir.), Giulietta Masina, François Périer , Rating:  8.0\n",
      "229 ) La Haine (1995) , Starring:  Mathieu Kassovitz (dir.), Vincent Cassel, Hubert Koundé , Rating:  8.0\n",
      "230 ) Groundhog Day (1993) , Starring:  Harold Ramis (dir.), Bill Murray, Andie MacDowell , Rating:  8.0\n",
      "231 ) The Wizard of Oz (1939) , Starring:  Victor Fleming (dir.), Judy Garland, Frank Morgan , Rating:  8.0\n",
      "232 ) Jaws (1975) , Starring:  Steven Spielberg (dir.), Roy Scheider, Robert Shaw , Rating:  8.0\n",
      "233 ) Paper Moon (1973) , Starring:  Peter Bogdanovich (dir.), Ryan O'Neal, Tatum O'Neal , Rating:  8.0\n",
      "234 ) The Help (2011) , Starring:  Tate Taylor (dir.), Emma Stone, Viola Davis , Rating:  8.0\n",
      "235 ) In the Mood for Love (2000) , Starring:  Kar-Wai Wong (dir.), Tony Chiu-Wai Leung, Maggie Cheung , Rating:  8.0\n",
      "236 ) A Wednesday (2008) , Starring:  Neeraj Pandey (dir.), Anupam Kher, Naseeruddin Shah , Rating:  8.0\n",
      "237 ) Gangs of Wasseypur (2012) , Starring:  Anurag Kashyap (dir.), Manoj Bajpayee, Richa Chadha , Rating:  8.0\n",
      "238 ) Before Sunset (2004) , Starring:  Richard Linklater (dir.), Ethan Hawke, Julie Delpy , Rating:  8.0\n",
      "239 ) The Bourne Ultimatum (2007) , Starring:  Paul Greengrass (dir.), Matt Damon, Edgar Ramírez , Rating:  8.0\n",
      "240 ) Paris, Texas (1984) , Starring:  Wim Wenders (dir.), Harry Dean Stanton, Nastassja Kinski , Rating:  8.0\n",
      "241 ) Guardians of the Galaxy (2014) , Starring:  James Gunn (dir.), Chris Pratt, Vin Diesel , Rating:  8.0\n",
      "242 ) Tangerines (2013) , Starring:  Zaza Urushadze (dir.), Lembit Ulfsak, Elmo Nüganen , Rating:  8.0\n",
      "243 ) Castle in the Sky (1986) , Starring:  Hayao Miyazaki (dir.), Anna Paquin, James Van Der Beek , Rating:  8.0\n",
      "244 ) Pirates of the Caribbean: The Curse of the Black Pearl (2003) , Starring:  Gore Verbinski (dir.), Johnny Depp, Geoffrey Rush , Rating:  8.0\n",
      "245 ) Ah-ga-ssi (2016) , Starring:  Chan-wook Park (dir.), Min-hee Kim, Jung-woo Ha , Rating:  8.0\n",
      "246 ) PK (2014) , Starring:  Rajkumar Hirani (dir.), Aamir Khan, Anushka Sharma , Rating:  8.0\n",
      "247 ) La leggenda del pianista sull'oceano (1998) , Starring:  Giuseppe Tornatore (dir.), Tim Roth, Pruitt Taylor Vince , Rating:  8.0\n",
      "248 ) Lagaan: Once Upon a Time in India (2001) , Starring:  Ashutosh Gowariker (dir.), Aamir Khan, Raghuvir Yadav , Rating:  8.0\n",
      "249 ) Kis Uykusu (2014) , Starring:  Nuri Bilge Ceylan (dir.), Haluk Bilginer, Melisa Sözen , Rating:  8.0\n",
      "250 ) Song of the Sea (2014) , Starring:  Tomm Moore (dir.), David Rawle, Brendan Gleeson , Rating:  8.0\n"
     ]
    }
   ],
   "source": [
    "url = \"https://www.imdb.com/chart/top\"\n",
    "page = urllib.request.urlopen(url)\n",
    "soup = bs4.BeautifulSoup(page, 'lxml')\n",
    "\n",
    "movie_list = soup.select('td.titleColumn')\n",
    "movie_rating_list = soup.select('td.imdbRating')\n",
    "\n",
    "for i in range(0, len(movie_list)):\n",
    "    movie_title = movie_list[i].find('a').text.strip()\n",
    "    movie_year = movie_list[i].find('span').text.strip()\n",
    "    movie_stars = movie_list[i].find('a').get('title')\n",
    "    movie_rating = movie_rating_list[i].find('strong').text.strip()\n",
    "    print(i + 1, \")\", movie_title, movie_year, \",\", \"Starring: \", movie_stars, \",\", \"Rating: \", movie_rating)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Q. Write a Python program to get movie name, year and a brief summary of the top 10 random movies."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1 ) Se7en (1995) \n",
      " Two detectives, a rookie and a veteran, hunt a serial killer who uses the seven deadly sins as his motives.\n",
      "2 ) The Help (2011) \n",
      " An aspiring author during the civil rights movement of the 1960s decides to write a book detailing the African American maids' point of view on the white families for which they work, and the hardships they go through on a daily basis.\n",
      "3 ) The Treasure of the Sierra Madre (1948) \n",
      " Fred Dobbs and Bob Curtin, two Americans searching for work in Mexico, convince an old prospector to help them mine for gold in the Sierra Madre Mountains.\n",
      "4 ) Diabolique (1955) \n",
      " The wife and mistress of a loathed school principal hatch a plan to murder him while having the perfect alibi. They carry out the plan...but then his body disappears.\n",
      "5 ) Sherlock Jr. (1924) \n",
      " A film projectionist longs to be a detective, and puts his meagre skills to work when he is framed by a rival for stealing his girlfriend's father's pocketwatch.\n",
      "6 ) The Lives of Others (2006) \n",
      " In 1984 East Berlin, an agent of the secret police, conducting surveillance on a writer and his lover, finds himself becoming increasingly absorbed by their lives.\n",
      "7 ) Casablanca (1942) \n",
      " A cynical nightclub owner protects an old flame and her husband from Nazis in Morocco.\n",
      "8 ) How to Train Your Dragon (2010) \n",
      " A hapless young Viking who aspires to hunt dragons becomes the unlikely friend of a young dragon himself, and learns there may be more to the creatures than he assumed.\n",
      "9 ) My Neighbor Totoro (1988) \n",
      " When two girls move to the country to be near their ailing mother, they have adventures with the wondrous forest spirits who live nearby.\n",
      "10 ) Princess Mononoke (1997) \n",
      " On a journey to find the cure for a Tatarigami's curse, Ashitaka finds himself in the middle of a war between the forest gods and Tatara, a mining colony. In this quest he also meets San, the Mononoke Hime.\n"
     ]
    }
   ],
   "source": [
    "import random\n",
    "\n",
    "url = \"https://www.imdb.com/chart/top\"\n",
    "page = urllib.request.urlopen(url)\n",
    "soup = bs4.BeautifulSoup(page, 'lxml')\n",
    "\n",
    "all_movies = soup.select('td.titleColumn')\n",
    "random.shuffle(all_movies)\n",
    "\n",
    "def brief_url(urlb):\n",
    "    brief_page = \"https://www.imdb.com{}\".format(urlb)\n",
    "    brief_url = urllib.request.urlopen(brief_page)\n",
    "    soup_brief = bs4.BeautifulSoup(brief_url, 'lxml')\n",
    "    brief = soup_brief.find('div', {'class': 'summary_text'}).text.strip()\n",
    "    return brief\n",
    "\n",
    "for i in range(0, 10):\n",
    "    movie_title = all_movies[i].find('a').text.strip()\n",
    "    movie_year = all_movies[i].find('span').text.strip()\n",
    "    movie_brief = brief_url(all_movies[i].find('a').get('href'))\n",
    "    print(i + 1,\")\", movie_title, movie_year, \"\\n\", movie_brief)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.1"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}