Week3-Part1.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "b1fb9378",
   "metadata": {},
   "source": [
    "# IBM Apliend data science capstone\n",
    "\n",
    "## Segmenting and Clustering Neighborhoods in Toronto\n",
    "\n",
    "### Web Scraping"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa640a78",
   "metadata": {},
   "source": [
    "__Objetive:__\n",
    "It is required explore and cluster the neighborhoods in Toronto.\n",
    "\n",
    " 1. The dataframe will consist of three columns: PostalCode, Borough, and Neighborhood\n",
    " 2. Only process the cells that have an assigned borough. Ignore cells with a borough that is Not assigned.\n",
    " 3. More than one neighborhood can exist in one postal code area. For example, in the table on the Wikipedia page, you will notice that M5A is listed twice and has two neighborhoods: Harbourfront and Regent Park. These two rows will be combined into one row with the neighborhoods separated with a comma as shown in row 11  in the above table.\n",
    " 4. If a cell has a borough but a Not assigned  neighborhood, then the neighborhood will be the same as the borough.\n",
    " 5. Clean your Notebook and add Markdown cells to explain your work and any assumptions you are making.\n",
    " 6. In the last cell of your notebook, use the .shape method to print the number of rows of your dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9bc34d4c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np \n",
    "import pandas as pd\n",
    "pd.set_option(\"display.max_columns\", None)\n",
    "pd.set_option(\"display.max_rows\", None)\n",
    "\n",
    "import requests \n",
    "\n",
    "# Library to parse HTML and XML documents\n",
    "from bs4 import BeautifulSoup \n",
    "\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f8638682",
   "metadata": {},
   "source": [
    "### Data scraping from Wikipedia"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "4cfbd29d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# send the GET request\n",
    "data = requests.get('https://en.wikipedia.org/wiki/List_of_postal_codes_of_Canada:_M').text\n",
    "\n",
    "# parse data from the html into a beautifulsoup object\n",
    "soup = BeautifulSoup(data, 'html.parser')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0b292d96",
   "metadata": {},
   "source": [
    "### Tips for  Webscraping Updated Table in Week3 \n",
    "\n",
    "After retreiving the URL and creating a Beautiful soup object \n",
    "\n",
    "Firstly create a list  \n",
    "\n",
    "Later after finding the table and table data  create a dictionary called cell having 3 keys PostalCode, Borough and Neighborhood.\n",
    "\n",
    "As postal code contains upto 3 characters extract that using `tablerow.p.text`\n",
    "\n",
    " Next use split ,strip and replace functions for getting Borough and Neighborhood information.\n",
    "\n",
    " Append to the list \n",
    "\n",
    "Create a dataframe with list"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "41406b25",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Webscraping Hints\n",
    "\n",
    "table_contents=[]\n",
    "table=soup.find('table')\n",
    "for row in table.findAll('td'):\n",
    "    cell = {}\n",
    "    if row.span.text=='Not assigned':\n",
    "        # Drop cells with a borough that is \"Not assigned\"\n",
    "        pass\n",
    "    else:\n",
    "        cell['PostalCode'] = row.p.text[:3]\n",
    "        cell['Borough'] = (row.span.text).split('(')[0]\n",
    "        cell['Neighborhood'] = (((((row.span.text).split('(')[1]).strip(')')).replace(' /',',')).replace(')',' ')).strip(' ')\n",
    "        table_contents.append(cell)\n",
    "\n",
    "# print(table_contents)\n",
    "df=pd.DataFrame(table_contents)\n",
    "df['Borough']=df['Borough'].replace({'Downtown TorontoStn A PO Boxes25 The Esplanade':'Downtown Toronto Stn A',\n",
    "                                             'East TorontoBusiness reply mail Processing Centre969 Eastern':'East Toronto Business',\n",
    "                                             'EtobicokeNorthwest':'Etobicoke Northwest','East YorkEast Toronto':'East York/East Toronto',\n",
    "                                             'MississaugaCanada Post Gateway Processing Centre':'Mississauga'})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "b8b83ab3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>PostalCode</th>\n",
       "      <th>Borough</th>\n",
       "      <th>Neighborhood</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>M3A</td>\n",
       "      <td>North York</td>\n",
       "      <td>Parkwoods</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>M4A</td>\n",
       "      <td>North York</td>\n",
       "      <td>Victoria Village</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>M5A</td>\n",
       "      <td>Downtown Toronto</td>\n",
       "      <td>Regent Park, Harbourfront</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>M6A</td>\n",
       "      <td>North York</td>\n",
       "      <td>Lawrence Manor, Lawrence Heights</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>M7A</td>\n",
       "      <td>Queen's Park</td>\n",
       "      <td>Ontario Provincial Government</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  PostalCode           Borough                      Neighborhood\n",
       "0        M3A        North York                         Parkwoods\n",
       "1        M4A        North York                  Victoria Village\n",
       "2        M5A  Downtown Toronto         Regent Park, Harbourfront\n",
       "3        M6A        North York  Lawrence Manor, Lawrence Heights\n",
       "4        M7A      Queen's Park     Ontario Provincial Government"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9aaef35f",
   "metadata": {},
   "source": [
    "### Print the number of rows of the cleaned dataframe\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7888f9d6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Number of rows of the cleaned dataframe 103\n"
     ]
    }
   ],
   "source": [
    "# print the number of rows of the cleaned dataframe\n",
    "print(\"Number of rows of the cleaned dataframe\", df.shape[0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "4d5541ac",
   "metadata": {},
   "outputs": [],
   "source": [
    "df.to_csv(\"Toronto_PostalCode.csv\",index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "0a90e715",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}