Web Scraper - FIFA World Cup Data.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Collection for 2018 FIFA World Cup Russia™ Prediction Project\n",
    "<hr>\n",
    "The goal of this notebook is to show how I collected data from the FIFA World Cup website for the 2018 World Cup Russia - Group Stage Prediction Project. Data was collected using a web crawler (build in Python), due to lack of API. \n",
    "\n",
    "The order of this notebook is as follows:\n",
    "1. Information about the data source\n",
    "2. Step by step instruction on how the web crawler was built\n",
    "3. Result of data collection\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data source\n",
    "\n",
    "I have collected groups statistics for all FIFA World Cups between 1994 and 2014. \n",
    "\n",
    "That gives us 6 FIFA World Cups:\n",
    "\n",
    " - 2014 FIFA World Cup Brazil™\n",
    " - 2010 FIFA World Cup South Africa™\n",
    " - 2006 FIFA World Cup Germany™\n",
    " - 2002 FIFA World Cup Korea/Japan™\n",
    " - 1998 FIFA World Cup France™\n",
    " - 1994 FIFA World Cup USA™\n",
    "\n",
    "The tournaments between 1998 and 2014 included 32 teams, divided into eight groups (A to H) - four teams (countries) per group. 1994 FIFA World Cup USA™ included 24 teams, divided into six groups (A to F) - four teams (countries) per group.\n",
    "\n",
    "For each World Cup edition, I collected the following groups' statistics:\n",
    "\n",
    " - Group - name of the group\n",
    " - Teams - name of the team (country)\n",
    " - Match played - how many matches did the team play\n",
    " - Match won - how many matches it won    \n",
    " - Draw - number of draws\n",
    " - Lost - number of loses\n",
    " - Goals for - total number of goals\n",
    " - Goals against - total number of lost goals\n",
    " - Goals difference - difference between goals for and against\n",
    " - Points - total points\n",
    "\n",
    "\n",
    "This data is publicaly accessible on the FIFA website in the <a href=\"https://www.fifa.com/fifa-tournaments/statistics-and-records/worldcup/index.html\">statistics and records section</a>. According to the robots.txt, data scraping from the statistics and records page is allowed [June 10, 2018].\n",
    "\n",
    "I went through the process of building this web scraper in the [Let the robot do your work! Web scraping with Python!](https://medium.com/ub-women-data-scholars/let-the-robot-do-your-work-web-scraping-with-python-9c147fb7690f) article posted on Medium. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Web crawler\n",
    "\n",
    "This part of the notebook will explan how the web crawler was built.\n",
    "\n",
    "Program was built in Python 3.x\n",
    " \n",
    "The following libraries were used:\n",
    " - <a href=\"http://docs.python-requests.org/en/master/\">Requests</a>\n",
    " - <a href=\"https://www.crummy.com/software/BeautifulSoup/bs4/doc/#\">BeautifulSoup</a>\n",
    " - <a href=\"https://pandas.pydata.org/\">Pandas</a>\n",
    "\n",
    "The structure of the code:\n",
    "1. Libraries import\n",
    "2. Obtaining data from the FIFA Tournaments page\n",
    " - GET request to the FIFA Tournaments page\n",
    " - saving the page content in Beautiful Soup format \n",
    " - extraction of links to individual FIFA World Cup editions\n",
    " - edition of links in order to generate links to the pages containing groups statistics\n",
    "3. Obtaining group statistics data from the individual World Cup pages\n",
    " - GET request to each page containing groups statistics\n",
    " - saving the page content in Beautiful Soup format\n",
    " - extraction of data from the HTML\n",
    " - saving data as panda's dataframe\n",
    "4. Export of dataframe to Excel"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 1. Libraries import\n",
    "\n",
    "- Request library, which allows to send HTTP requests\n",
    "- BeautifulSoup library for pulling data out of HTML\n",
    "- Pandas library to store data as dataframe"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 2. Obtaining data from the FIFA Tournaments page\n",
    "\n",
    "I set url variable to FIFA Tournaments page and then use try/except block to check if page is accessible. I set timeout to 5 seconds, and store HTTP status code in page_response variable. \n",
    "\n",
    "If page is not accessible, I print page status to console. If timeout occurred, I print notification to console.\n",
    "\n",
    "If page is accessible (200 is a standard code for successful HTTP requests) I save HTML to page_content variable using BeautifulSoup format. Then, I use BeautifulSoup lxml parser to find element, which contains links to all World Cup editions. I extract these links and edit them to generate new links, which direct to the groups statistics. \n",
    "\n",
    "As I am only interested in the last six World Cups, I keep the first 6 elements of the list."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "url = \"http://www.fifa.com/fifa-tournaments/statistics-and-records/worldcup/index.html\"\n",
    "\n",
    "try:\n",
    "    page_response = requests.get(url, timeout=5)  \n",
    "    \n",
    "    if page_response.status_code == 200:\n",
    "        page_content = BeautifulSoup(page_response.content,'lxml')\n",
    "        tournaments = page_content.find(attrs={'id':'alleditions'})\n",
    "\n",
    "        link_list = []\n",
    "        for link in tournaments.find_all('a'):\n",
    "            group_link = 'http://www.fifa.com' + link.get('href').replace('index.html','groups/index.html')\n",
    "            link_list.append(group_link)\n",
    "        link_list = link_list[0:6]\n",
    "\n",
    "    else:\n",
    "        print(page_response.status_code)\n",
    "        \n",
    "except requests.Timeout as e:\n",
    "        print('Timeout occurred for requested page: ' + url)\n",
    "        print(str(e))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As a result of these operations, I receive a list containing 6 links to group statistics, one link for each edition of the World Cup."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "http://www.fifa.com/worldcup/archive/brazil2014/groups/index.html\n",
      "http://www.fifa.com/worldcup/archive/southafrica2010/groups/index.html\n",
      "http://www.fifa.com/worldcup/archive/germany2006/groups/index.html\n",
      "http://www.fifa.com/worldcup/archive/koreajapan2002/groups/index.html\n",
      "http://www.fifa.com/worldcup/archive/france1998/groups/index.html\n",
      "http://www.fifa.com/worldcup/archive/usa1994/groups/index.html\n"
     ]
    }
   ],
   "source": [
    "for link in link_list:\n",
    "    print(link)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 3. Obtaining group statistics data from the individual World Cup pages\n",
    "\n",
    "\n",
    "First, I create an empty list to store groups' statistics and set counter variable to 0. \n",
    "\n",
    "I use while loop to iterate through the link_list, and increment counter by 1 on each iteration. While loop will not be executed if link_list is empty, and will break when counter is equal to the list's lenght. \n",
    "\n",
    "Inside the loop, I set url variable to the link from the link_list, using counter as list index. Then I use try/except block to check if page is accessible. I set timeout to 5 seconds, and store HTTP status code in page_response variable. \n",
    "\n",
    "If page is not accessible, I print page status to console. If timeout occurred, I print notification to console.\n",
    "\n",
    "If page is accessible (200 is a standard code for successful HTTP requests) I save HTML to page_content variable using BeautifulSoup format. Then, I use BeautifulSoup lxml parser to find HTML elements, which contain the data:\n",
    "\n",
    " - take the title of the webpage (e.g. 2014 FIFA World Cup Brazil™) and split it to extract World Cup year \n",
    " - select HTML element that contains statistics for all groups and save it in the standings_table variable\n",
    " - select all HTML elements that contain group letters and use list comprehensions to extract their inner text;\n",
    " - repeat each group letter four times (on the FIFA website, for each group (A - H), there is only one HTML element containing the letter of the group; since there are four teams in each group, repeat the letter of the group four times)\n",
    " - select all HTML elements that contain relevant statistics (teams, match_played, match_won, draw, lost, goals_for, goals_against, goals_difference, points) and use list comprehensions to extract their inner text;\n",
    "\n",
    "After collecting the data, I save it in the pandas dataframe format. I append dataframe to append_data list.\n",
    "\n",
    "As a result of while loop, I receive list of dataframes. I use Pandas concat method to concatenate all dataframes into one dataframe. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "append_data = []\n",
    "counter = 0\n",
    "\n",
    "while counter < len(link_list) and len(link_list) > 0:\n",
    "    \n",
    "    url = link_list[counter]\n",
    "    try:\n",
    "        page_response = requests.get(url, timeout=5)\n",
    "\n",
    "        if page_response.status_code == 200:\n",
    "            page_content = BeautifulSoup(page_response.content,'lxml')\n",
    "\n",
    "            page_title = page_content.title.text\n",
    "            world_cup = page_title.split(' ')[0]\n",
    "\n",
    "            standings_table = page_content.find('div', attrs={'id':'standings'})\n",
    "\n",
    "            group_letters = [gl.get_text() for gl in standings_table.select('.group-wrap .caption-nolink')]\n",
    "            group_letter = [gl for gl in group_letters for i in range(4)]\n",
    "\n",
    "            teams = [tn.get_text() for tn in standings_table.select('.group-wrap .teamname-nolink span.t-nText')]\n",
    "            match_played = [mp.get_text() for mp in standings_table.select('.group-wrap .tbl-matchplayed span.text')]\n",
    "            match_won = [mw.get_text() for mw in standings_table.select('.group-wrap .tbl-win span.text')]\n",
    "            draw = [d.get_text() for d in standings_table.select('.group-wrap .tbl-draw span.text')]\n",
    "            lost = [l.get_text() for l in standings_table.select('.group-wrap .tbl-lost span.text')]\n",
    "            goals_for = [gf.get_text() for gf in standings_table.select('.group-wrap .tbl-goalfor span.text')]\n",
    "            goals_against = [ga.get_text() for ga in standings_table.select('.group-wrap .tbl-goalagainst span.text')]\n",
    "            goals_difference = [gd.get_text() for gd in standings_table.select('.group-wrap .tbl-diffgoal span.text')]\n",
    "            points = [p.get_text() for p in standings_table.select('.group-wrap .tbl-pts span.text')]\n",
    "\n",
    "            group_tables = pd.DataFrame({\n",
    "                \"Fifa World Cup\": world_cup,\n",
    "                \"Group\": group_letter, \n",
    "                \"Teams\": teams, \n",
    "                \"Match played\": match_played, \n",
    "                \"Match won\": match_won,\n",
    "                \"Draw\": draw,\n",
    "                \"Lost\": lost,\n",
    "                \"Goals for\": goals_for,\n",
    "                \"Goals against\": goals_against,\n",
    "                \"Goals difference\": goals_difference,\n",
    "                \"Points\": points\n",
    "            })\n",
    "\n",
    "            append_data.append(group_tables)\n",
    "\n",
    "        else:\n",
    "            print(page_response.status_code)\n",
    "\n",
    "    except requests.Timeout as e:\n",
    "        print('Timeout occurred for requested page: ' + url)\n",
    "        print(str(e))\n",
    "\n",
    "    counter += 1\n",
    "    \n",
    "worldcup_data = pd.concat(append_data, axis=0, ignore_index=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### 4. Export of dataframe to Excel file\n",
    "\n",
    "As the last step, I export dataframe to Excel."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>Fifa World Cup</th>\n",
       "      <th>Group</th>\n",
       "      <th>Teams</th>\n",
       "      <th>Match played</th>\n",
       "      <th>Match won</th>\n",
       "      <th>Draw</th>\n",
       "      <th>Lost</th>\n",
       "      <th>Goals for</th>\n",
       "      <th>Goals against</th>\n",
       "      <th>Goals difference</th>\n",
       "      <th>Points</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2014</td>\n",
       "      <td>Group A</td>\n",
       "      <td>Brazil</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>7</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2014</td>\n",
       "      <td>Group A</td>\n",
       "      <td>Mexico</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2014</td>\n",
       "      <td>Group A</td>\n",
       "      <td>Croatia</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "      <td>6</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2014</td>\n",
       "      <td>Group A</td>\n",
       "      <td>Cameroon</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>-8</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2014</td>\n",
       "      <td>Group B</td>\n",
       "      <td>Netherlands</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>10</td>\n",
       "      <td>3</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2014</td>\n",
       "      <td>Group B</td>\n",
       "      <td>Chile</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>5</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2014</td>\n",
       "      <td>Group B</td>\n",
       "      <td>Spain</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>7</td>\n",
       "      <td>-3</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2014</td>\n",
       "      <td>Group B</td>\n",
       "      <td>Australia</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>-6</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2014</td>\n",
       "      <td>Group C</td>\n",
       "      <td>Colombia</td>\n",
       "      <td>3</td>\n",
       "      <td>3</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>9</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>9</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2014</td>\n",
       "      <td>Group C</td>\n",
       "      <td>Greece</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>-2</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "  Fifa World Cup    Group        Teams Match played Match won Draw Lost  \\\n",
       "0           2014  Group A       Brazil            3         2    1    0   \n",
       "1           2014  Group A       Mexico            3         2    1    0   \n",
       "2           2014  Group A      Croatia            3         1    0    2   \n",
       "3           2014  Group A     Cameroon            3         0    0    3   \n",
       "4           2014  Group B  Netherlands            3         3    0    0   \n",
       "5           2014  Group B        Chile            3         2    0    1   \n",
       "6           2014  Group B        Spain            3         1    0    2   \n",
       "7           2014  Group B    Australia            3         0    0    3   \n",
       "8           2014  Group C     Colombia            3         3    0    0   \n",
       "9           2014  Group C       Greece            3         1    1    1   \n",
       "\n",
       "  Goals for Goals against Goals difference Points  \n",
       "0         7             2                5      7  \n",
       "1         4             1                3      7  \n",
       "2         6             6                0      3  \n",
       "3         1             9               -8      0  \n",
       "4        10             3                7      9  \n",
       "5         5             3                2      6  \n",
       "6         4             7               -3      3  \n",
       "7         3             9               -6      0  \n",
       "8         9             2                7      9  \n",
       "9         2             4               -2      4  "
      ]
     },
     "execution_count": 5,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "worldcup_data.to_excel('WorldCupData.xlsx')\n",
    "\n",
    "worldcup_data.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.0"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}