S19-App-7-Scrape-Real-Estate-Property-Data-from-the-Web.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# The Python Mega Course: Build 10 Real World Applications\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook is a summary of [The Python Mega Course: Build 10 Real World Applciations](https://www.udemy.com/the-python-mega-course), a comprehensive online Python course taught by Ardit Sulce. Each lecture name is clickable and takes you to the video lecture in the course."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Section 19: Application 7: Scrape Real Estate Property Data from the Web\n",
    "***"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lecture:** [Program Demonstration](https://www.udemy.com/the-python-mega-course/learn/v4/t/lecture/9439078?start=0)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This video lecture shows the finished version of the website running on a browser."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lecture:** [Loading the Webpage in Python](https://www.udemy.com/the-python-mega-course/learn/v4/t/lecture/9439078?start=0)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code loads the webpage source code into Python ready for extracting information from it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests, re\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "r = requests.get(\"http://www.pythonhow.com/real-estate/rock-springs-wy/LCWYROCKSPRINGS/\")\n",
    "c = r.content\n",
    "\n",
    "soup=BeautifulSoup(c, \"html.parser\")\n",
    "print(soup.prettify())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lecture:** [Extracting \"div\" Tags](https://www.udemy.com/the-python-mega-course/learn/v4/t/lecture/9439078?start=0)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We start extracting HTML tags starting from `div` tags."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests, re\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "r = requests.get(\"http://www.pythonhow.com/real-estate/rock-springs-wy/LCWYROCKSPRINGS/\")\n",
    "c = r.content\n",
    "\n",
    "soup = BeautifulSoup(c,\"html.parser\")\n",
    "\n",
    "all = soup.find_all(\"div\", {\"class\":\"propertyRow\"})\n",
    "all[0].find(\"h4\", {\"class\":\"propPrice\"}).text.replace(\"\\n\", \"\").replace(\" \", \"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lecture:** [Extracting Addresses and Property Details](https://www.udemy.com/the-python-mega-course/learn/v4/t/lecture/9439078?start=0)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Most of the data are stored inside `span` tags so we extract those data in this code."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests, re\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "r = requests.get(\"http://www.pythonhow.com/real-estate/rock-springs-wy/LCWYROCKSPRINGS/\")\n",
    "c = r.content\n",
    "\n",
    "soup = BeautifulSoup(c,\"html.parser\")\n",
    "\n",
    "all = soup.find_all(\"div\", {\"class\":\"propertyRow\"})\n",
    "all[0].find(\"h4\", {\"class\":\"propPrice\"}).text.replace(\"\\n\", \"\").replace(\" \", \"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for item in all:\n",
    "    print(item.find(\"h4\", {\"class\", \"propPrice\"}).text.replace(\"\\n\",\"\").replace(\" \", \"\"))\n",
    "    print(item.find_all(\"span\", {\"class\",\"propAddressCollapse\"})[0].text)\n",
    "    print(item.find_all(\"span\", {\"class\",\"propAddressCollapse\"})[1].text)\n",
    "\n",
    "    try:\n",
    "        print(item.find(\"span\", {\"class\", \"infoBed\"}).find(\"b\").text)\n",
    "    except:\n",
    "        print(None)\n",
    "\n",
    "    try:\n",
    "        print(item.find(\"span\", {\"class\", \"infoSqFt\"}).find(\"b\").text)\n",
    "    except:\n",
    "        print(None)\n",
    "\n",
    "    try:\n",
    "        print(item.find(\"span\", {\"class\", \"infoValueFullBath\"}).find(\"b\").text)\n",
    "    except:\n",
    "        print(None)\n",
    "\n",
    "    try:\n",
    "        print(item.find(\"span\", {\"class\", \"infoValueHalfBath\"}).find(\"b\").text)\n",
    "    except:\n",
    "        print(None)\n",
    "        \n",
    "    print(\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lecture:** [Extracting Elements without Unique Identifiers](https://www.udemy.com/the-python-mega-course/learn/v4/t/lecture/9439078?start=0)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we extract some more elements."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "for item in all:\n",
    "    print(item.find(\"h4\", {\"class\", \"propPrice\"}).text.replace(\"\\n\",\"\").replace(\" \", \"\"))\n",
    "    print(item.find_all(\"span\", {\"class\",\"propAddressCollapse\"})[0].text)\n",
    "    print(item.find_all(\"span\", {\"class\",\"propAddressCollapse\"})[1].text)\n",
    "\n",
    "    try:\n",
    "        print(item.find(\"span\", {\"class\", \"infoBed\"}).find(\"b\").text)\n",
    "    except:\n",
    "        print(None)\n",
    "\n",
    "    try:\n",
    "        print(item.find(\"span\", {\"class\", \"infoSqFt\"}).find(\"b\").text)\n",
    "    except:\n",
    "        print(None)\n",
    "\n",
    "    try:\n",
    "        print(item.find(\"span\", {\"class\", \"infoValueFullBath\"}).find(\"b\").text)\n",
    "    except:\n",
    "        print(None)\n",
    "\n",
    "    try:\n",
    "        print(item.find(\"span\", {\"class\", \"infoValueHalfBath\"}).find(\"b\").text)\n",
    "    except:\n",
    "        print(None)\n",
    "        \n",
    "    for column_group in item.find_all(\"div\", {\"class\":\"columnGroup\"}):\n",
    "        for feature_group, feature_name in zip(column_group.find_all(\"span\", {\"class\":\"featureGroup\"}), column_group.find_all(\"span\", {\"class\":\"featureName\"})):\n",
    "            if \"Lot Size\" in feature_group.text:\n",
    "                print(feature_name.text)\n",
    "\n",
    "    print(\" \")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lecture:** [Saving the Extracted Data in CSV Files](https://www.udemy.com/the-python-mega-course/learn/v4/t/lecture/9439078?start=0)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we save the extracted data into a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests, re\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "r = requests.get(\"http://www.pythonhow.com/real-estate/rock-springs-wy/LCWYROCKSPRINGS/\")\n",
    "c = r.content\n",
    "\n",
    "soup = BeautifulSoup(c,\"html.parser\")\n",
    "\n",
    "all = soup.find_all(\"div\",{\"class\":\"propertyRow\"})\n",
    "\n",
    "all[0].find(\"h4\", {\"class\":\"propPrice\"}).text.replace(\"\\n\", \"\").replace(\" \", \"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "l = []\n",
    "for item in all:\n",
    "    d = {}\n",
    "    df[\"Address\"] = item.find_all(\"span\", {\"class\", \"propAddressCollapse\"})[0].text\n",
    "    df[\"Locality\"] = item.find_all(\"span\", {\"class\", \"propAddressCollapse\"})[1].text\n",
    "    df[\"Price\"] = item.find(\"h4\", {\"class\", \"propPrice\"}).text.replace(\"\\n\",\"\").replace(\" \", \"\")\n",
    "    \n",
    "    try:\n",
    "        d[\"Beds\"] = item.find(\"span\", {\"class\", \"infoBed\"}).find(\"b\").text\n",
    "    except:\n",
    "        d[\"Beds\"] = None\n",
    "\n",
    "    try:\n",
    "        d[\"Area\"] = item.find(\"span\", {\"class\", \"infoSqFt\"}).find(\"b\").text\n",
    "    except:\n",
    "        d[\"Area\"] = None\n",
    "\n",
    "    try:\n",
    "        d[\"Full Baths\"] = item.find(\"span\", {\"class\", \"infoValueFullBath\"}).find(\"b\").text\n",
    "    except:\n",
    "        d[\"Full Baths\"] = None\n",
    "\n",
    "    try:\n",
    "        d[\"Half Baths\"] = item.find(\"span\", {\"class\", \"infoValueHalfBath\"}).find(\"b\").text\n",
    "    except:\n",
    "        d[\"Half Baths\"] = None\n",
    "\n",
    "    for column_group in item.find_all(\"div\", {\"class\":\"columnGroup\"}):\n",
    "        for feature_group, feature_name in zip(column_group.find_all(\"span\", {\"class\":\"featureGroup\"}), column_group.find_all(\"span\", {\"class\":\"featureName\"})):\n",
    "            if \"Lot Size\" in feature_group.text:\n",
    "                print(feature_name.text)\n",
    "                d[\"Lot Size\"] = feature_name.text\n",
    "    l.append(d)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import pandas\n",
    "df = pandas.DataFrame(l)\n",
    "df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "df.to_csv(\"Output.csv\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lecture:** [Crawling Through Webpages](https://www.udemy.com/the-python-mega-course/learn/v4/t/lecture/9439078?start=0)\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In case you need to extract data from multiple pages, here is how to do it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests, re\n",
    "from bs4 import BeautifulSoup\n",
    "\n",
    "r = requests.get(\"http://www.pythonhow.com/real-estate/rock-springs-wy/LCWYROCKSPRINGS/\")\n",
    "c = r.content\n",
    "\n",
    "soup = BeautifulSoup(c,\"html.parser\")\n",
    "\n",
    "all = soup.find_all(\"div\",{\"class\":\"propertyRow\"})\n",
    "\n",
    "all[0].find(\"h4\", {\"class\":\"propPrice\"}).text.replace(\"\\n\", \"\").replace(\" \", \"\")\n",
    "\n",
    "page_nr = soup.find_all(\"a\",{\"class\":\"Page\"})[-1].text\n",
    "print(page_nr, \"number of pages were found\")\n",
    "\n",
    "l = []\n",
    "base_url = \"http://www.pythonhow.com/real-estate/rock-springs-wy/LCWYROCKSPRINGS/t=0&s=\"\n",
    "for page in range(0, int(page_nr)*10, 10):\n",
    "    print( )\n",
    "    r = requests.get(base_url + str(page) + \".html\")\n",
    "    c = r.content\n",
    "    soup = BeautifulSoup(c, \"html.parser\")\n",
    "    all = soup.find_all(\"div\", {\"class\":\"propertyRow\"})\n",
    "    for item in all:\n",
    "        d = {}\n",
    "        d[\"Address\"] = item.find_all(\"span\", {\"class\",\"propAddressCollapse\"})[0].text\n",
    "        \n",
    "        try:\n",
    "            d[\"Locality\"] = item.find_all(\"span\",{\"class\",\"propAddressCollapse\"})[1].text\n",
    "        except:\n",
    "            d[\"Locality\"] = None\n",
    "        d[\"Price\"] = item.find(\"h4\", {\"class\", \"propPrice\"}).text.replace(\"\\n\",\"\").replace(\" \", \"\")\n",
    "        \n",
    "        try:\n",
    "            d[\"Beds\"] = item.find(\"span\", {\"class\", \"infoBed\"}).find(\"b\").text\n",
    "        except:\n",
    "            d[\"Beds\"] = None\n",
    "\n",
    "        try:\n",
    "            d[\"Area\"] = item.find(\"span\", {\"class\", \"infoSqFt\"}).find(\"b\").text\n",
    "        except:\n",
    "            d[\"Area\"] = None\n",
    "    \n",
    "        try:\n",
    "            d[\"Full Baths\"] = item.find(\"span\", {\"class\", \"infoValueFullBath\"}).find(\"b\").text\n",
    "        except:\n",
    "            d[\"Full Baths\"] = None\n",
    "\n",
    "        try:\n",
    "            d[\"Half Baths\"] = item.find(\"span\", {\"class\", \"infoValueHalfBath\"}).find(\"b\").text\n",
    "        except:\n",
    "            d[\"Half Baths\"] = None\n",
    "        \n",
    "        for column_group in item.find_all(\"div\", {\"class\":\"columnGroup\"}):\n",
    "            for feature_group, feature_name in zip(column_group.find_all(\"span\", {\"class\":\"featureGroup\"}), column_group.find_all(\"span\", {\"class\":\"featureName\"})):\n",
    "                if \"Lot Size\" in feature_group.text:\n",
    "                    print(feature_name.text)\n",
    "                    d[\"Lot Size\"] = feature_name.text\n",
    "        l.append(d)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Lecture:** [Final Code of Application 7]()\n",
    "---"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This is the final code. It accesses a webpage and it extracts data from that webpage and save those data in a CSV file."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "**Note**: You need internet connection for the code to work."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": true
   },
   "outputs": [],
   "source": [
    "import requests, re\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas\n",
    "\n",
    "r = requests.get(\"http://www.pythonhow.com/real-estate/rock-springs-wy/LCWYROCKSPRINGS/\")\n",
    "c = r.content\n",
    "\n",
    "soup = BeautifulSoup(c,\"html.parser\")\n",
    "\n",
    "all = soup.find_all(\"div\",{\"class\":\"propertyRow\"})\n",
    "\n",
    "all[0].find(\"h4\", {\"class\":\"propPrice\"}).text.replace(\"\\n\", \"\").replace(\" \", \"\")\n",
    "\n",
    "page_nr = soup.find_all(\"a\",{\"class\":\"Page\"})[-1].text\n",
    "print(page_nr, \"number of pages were found\")\n",
    "\n",
    "l = []\n",
    "base_url = \"http://www.pythonhow.com/real-estate/rock-springs-wy/LCWYROCKSPRINGS/t=0&s=\"\n",
    "for page in range(0, int(page_nr)*10, 10):\n",
    "    print( )\n",
    "    r = requests.get(base_url + str(page) + \".html\")\n",
    "    c = r.content\n",
    "    soup = BeautifulSoup(c, \"html.parser\")\n",
    "    all = soup.find_all(\"div\", {\"class\":\"propertyRow\"})\n",
    "    for item in all:\n",
    "        d = {}\n",
    "        d[\"Address\"] = item.find_all(\"span\", {\"class\",\"propAddressCollapse\"})[0].text\n",
    "        \n",
    "        try:\n",
    "            d[\"Locality\"] = item.find_all(\"span\",{\"class\",\"propAddressCollapse\"})[1].text\n",
    "        except:\n",
    "            d[\"Locality\"] = None\n",
    "        d[\"Price\"] = item.find(\"h4\", {\"class\", \"propPrice\"}).text.replace(\"\\n\",\"\").replace(\" \", \"\")\n",
    "        \n",
    "        try:\n",
    "            d[\"Beds\"] = item.find(\"span\", {\"class\", \"infoBed\"}).find(\"b\").text\n",
    "        except:\n",
    "            d[\"Beds\"] = None\n",
    "\n",
    "        try:\n",
    "            d[\"Area\"] = item.find(\"span\", {\"class\", \"infoSqFt\"}).find(\"b\").text\n",
    "        except:\n",
    "            d[\"Area\"] = None\n",
    "    \n",
    "        try:\n",
    "            d[\"Full Baths\"] = item.find(\"span\", {\"class\", \"infoValueFullBath\"}).find(\"b\").text\n",
    "        except:\n",
    "            d[\"Full Baths\"] = None\n",
    "\n",
    "        try:\n",
    "            d[\"Half Baths\"] = item.find(\"span\", {\"class\", \"infoValueHalfBath\"}).find(\"b\").text\n",
    "        except:\n",
    "            d[\"Half Baths\"] = None\n",
    "        \n",
    "        for column_group in item.find_all(\"div\", {\"class\":\"columnGroup\"}):\n",
    "            for feature_group, feature_name in zip(column_group.find_all(\"span\", {\"class\":\"featureGroup\"}), column_group.find_all(\"span\", {\"class\":\"featureName\"})):\n",
    "                if \"Lot Size\" in feature_group.text:\n",
    "                    print(feature_name.text)\n",
    "                    d[\"Lot Size\"] = feature_name.text\n",
    "        l.append(d)\n",
    "        \n",
    "df = pandas.DataFrame(l)\n",
    "df.to_csv(\"Output.csv\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}