Airline_Reviews_Scraper.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "# Task 1\n",
    "\n",
    "---\n",
    "\n",
    "## Web scraping and analysis\n",
    "\n",
    "This Jupyter notebook includes some code to get you started with web scraping. We will use a package called `BeautifulSoup` to collect the data from the web. Once you've collected your data and saved it into a local `.csv` file you should start with your analysis.\n",
    "\n",
    "### Scraping data from Skytrax\n",
    "If you visit [https://www.airlinequality.com] you can see that there is a lot of data there. For this task, we are only interested in reviews related to British Airways and the Airline itself.\n",
    "If you navigate to this link: [https://www.airlinequality.com/airline-reviews/british-airways] you will see this data. Now, we can use `Python` and `BeautifulSoup` to collect all the links to the reviews and then to collect the text data on each of the individual review links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "import csv\n",
    "import requests\n",
    "from bs4 import BeautifulSoup\n",
    "import re\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Scraping page 1\n",
      "   ---> 100 total reviews\n",
      "Scraping page 2\n",
      "   ---> 200 total reviews\n",
      "Scraping page 3\n",
      "   ---> 300 total reviews\n",
      "Scraping page 4\n",
      "   ---> 400 total reviews\n",
      "Scraping page 5\n",
      "   ---> 500 total reviews\n",
      "Scraping page 6\n",
      "   ---> 600 total reviews\n",
      "Scraping page 7\n",
      "   ---> 700 total reviews\n",
      "Scraping page 8\n",
      "   ---> 800 total reviews\n",
      "Scraping page 9\n",
      "   ---> 900 total reviews\n",
      "Scraping page 10\n",
      "   ---> 1000 total reviews\n",
      "Scraping page 11\n",
      "   ---> 1100 total reviews\n",
      "Scraping page 12\n",
      "   ---> 1200 total reviews\n",
      "Scraping page 13\n",
      "   ---> 1300 total reviews\n",
      "Scraping page 14\n",
      "   ---> 1400 total reviews\n",
      "Scraping page 15\n",
      "   ---> 1500 total reviews\n",
      "Scraping page 16\n",
      "   ---> 1600 total reviews\n",
      "Scraping page 17\n",
      "   ---> 1700 total reviews\n",
      "Scraping page 18\n",
      "   ---> 1800 total reviews\n",
      "Scraping page 19\n",
      "   ---> 1900 total reviews\n",
      "Scraping page 20\n",
      "   ---> 2000 total reviews\n"
     ]
    }
   ],
   "source": [
    "# Set the URL of the paginated webpage that you want to scrape\n",
    "url = \"https://www.airlinequality.com/airline-reviews/british-airways\"\n",
    "\n",
    "# Initialize an empty list to store the data that you scrape\n",
    "data = []\n",
    "\n",
    "# Setting the initial page number and the increment that you want to use to paginate through the webpage\n",
    "page_num = 1\n",
    "page_incr = 1\n",
    "page_size = 100\n",
    "# maximum number of pages to be scraped\n",
    "max_pages = 20\n",
    "\n",
    "# Set the URL of the webpage to be scraped \n",
    "paginated_url = f\"{url}/page/{page_num}/?sortby=post_date%3ADesc&pagesize={page_size}\"\n",
    "\n",
    "# A while loop to paginate through the webpage and scrape the data\n",
    "while page_num <= max_pages:\n",
    "\n",
    "    print(f\"Scraping page {page_num}\")\n",
    "\n",
    "    # A GET request to the paginated URL\n",
    "    response = requests.get(paginated_url)\n",
    "\n",
    "    # Parsing the response using BeautifulSoup\n",
    "    parsed_content = BeautifulSoup(response.text, \"html.parser\")\n",
    "\n",
    "    # Finding all the elements on the page that contain the data to be scraped\n",
    "    elements = parsed_content.find_all(\"div\",class_ = \"body\")\n",
    "\n",
    "    # Looping through the elements and extract the data that you want to scrape\n",
    "    for element in elements:\n",
    "        header = element.find(\"h2\",class_ = \"text_header\").text.replace(\"\\n\", \" \")\n",
    "        sub_header = element.find(\"h3\",class_ = \"text_sub_header\").text.replace(\"\\n\", \" \")\n",
    "        content = element.find(\"div\",class_ = \"text_content\").text.replace(\"\\n\", \" \")\n",
    "        \n",
    "        data.append([header,sub_header,content])\n",
    "\n",
    "    # Increasing the page number and setting the paginated URL to the new page\n",
    "    page_num += page_incr\n",
    "    paginated_url = f\"{url}/page/{page_num}/?sortby=post_date%3ADesc&pagesize={page_size}\"\n",
    "\n",
    "    print(f\"   ---> {len(data)} total reviews\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%% md\n"
    }
   },
   "source": [
    "The loops above collected 2000 reviews by iterating through the paginated pages on the website.\n",
    "\n",
    " The next thing that you should do is clean this data to remove any unnecessary text from each of the rows. For example, \"✅ Trip Verified\" can be removed from each row if it exists, as it's not relevant to what we want to investigate."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": "                                               REVIEW  \\\n0                 \"appallingly uncomfortable flights\"   \n1                              \"Cabin crew very good\"   \n2                                  \"late and delayed\"   \n3          \"learned my lesson about late BA upgrades\"   \n4     \"No curtain between business and economy class\"   \n...                                               ...   \n1995                        \"treated incredibly well\"   \n1996                 \"service in a very discreet way\"   \n1997                   \"let down by inferior product\"   \n1998                           \"boarding worked well\"   \n1999             \"no different than a budget airline\"   \n\n                                          PERSONAL INFO  \\\n0       Jeremy Archdale (United Kingdom) 12th Decemb...   \n1       Andrew Mortimer (New Zealand) 12th December ...   \n2            C King (United Kingdom) 10th December 2022   \n3           R Vines (United Kingdom) 10th December 2022   \n4         A Lavochil (Czech Republic) 9th December 2022   \n...                                                 ...   \n1995       Walter Mythen (United Kingdom) 20th May 2016   \n1996    20 reviews    J Hugo (United Kingdom) 19th M...   \n1997    20 reviews    J Hugo (United Kingdom) 19th M...   \n1998        Leslie Percy (United Kingdom) 19th May 2016   \n1999    20 reviews    J Hugo (United Kingdom) 19th M...   \n\n                                                CONTENT  \n0     Stupidly tried BA again after a five year gap ...  \n1     Not Verified |  Seat horribly narrow; 3-4-3 on...  \n2     Glasgow to London delayed by 1 hour. My wife a...  \n3     When I tried to check in online, I was offered...  \n4     I flew from Prague to LHR. Excellent service, ...  \n...                                                 ...  \n1995  ✅ Verified Review |  We were treated incredibl...  \n1996  ✅ Verified Review |  I was scheduled to be on ...  \n1997  ✅ Verified Review |  British Airways from Lond...  \n1998  We flew British Airways BA2678 from Gatwick to...  \n1999  ✅ Verified Review |  Flew British Airways Lond...  \n\n[2000 rows x 3 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>REVIEW</th>\n      <th>PERSONAL INFO</th>\n      <th>CONTENT</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>\"appallingly uncomfortable flights\"</td>\n      <td>Jeremy Archdale (United Kingdom) 12th Decemb...</td>\n      <td>Stupidly tried BA again after a five year gap ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>\"Cabin crew very good\"</td>\n      <td>Andrew Mortimer (New Zealand) 12th December ...</td>\n      <td>Not Verified |  Seat horribly narrow; 3-4-3 on...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>\"late and delayed\"</td>\n      <td>C King (United Kingdom) 10th December 2022</td>\n      <td>Glasgow to London delayed by 1 hour. My wife a...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>\"learned my lesson about late BA upgrades\"</td>\n      <td>R Vines (United Kingdom) 10th December 2022</td>\n      <td>When I tried to check in online, I was offered...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>\"No curtain between business and economy class\"</td>\n      <td>A Lavochil (Czech Republic) 9th December 2022</td>\n      <td>I flew from Prague to LHR. Excellent service, ...</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>1995</th>\n      <td>\"treated incredibly well\"</td>\n      <td>Walter Mythen (United Kingdom) 20th May 2016</td>\n      <td>✅ Verified Review |  We were treated incredibl...</td>\n    </tr>\n    <tr>\n      <th>1996</th>\n      <td>\"service in a very discreet way\"</td>\n      <td>20 reviews    J Hugo (United Kingdom) 19th M...</td>\n      <td>✅ Verified Review |  I was scheduled to be on ...</td>\n    </tr>\n    <tr>\n      <th>1997</th>\n      <td>\"let down by inferior product\"</td>\n      <td>20 reviews    J Hugo (United Kingdom) 19th M...</td>\n      <td>✅ Verified Review |  British Airways from Lond...</td>\n    </tr>\n    <tr>\n      <th>1998</th>\n      <td>\"boarding worked well\"</td>\n      <td>Leslie Percy (United Kingdom) 19th May 2016</td>\n      <td>We flew British Airways BA2678 from Gatwick to...</td>\n    </tr>\n    <tr>\n      <th>1999</th>\n      <td>\"no different than a budget airline\"</td>\n      <td>20 reviews    J Hugo (United Kingdom) 19th M...</td>\n      <td>✅ Verified Review |  Flew British Airways Lond...</td>\n    </tr>\n  </tbody>\n</table>\n<p>2000 rows × 3 columns</p>\n</div>"
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "#Coverting the list data into a dataframe\n",
    "df = pd.DataFrame(data)\n",
    "df.columns = [\"REVIEW\",\"PERSONAL INFO\",\"CONTENT\"]\n",
    "\n",
    "#Removing unwanted text(first text preprocessing)\n",
    "df.replace(re.compile(r'\\s*✅ Trip Verified \\|\\s*'), '', inplace=True)\n",
    "df\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [],
   "source": [
    "#Saving data into a csv\n",
    "df.to_csv(r\"C:\\Users\\Mr.Hassan\\Desktop\\100DAYSOFML\\webscraping\\BA_reviews.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   },
   "outputs": [
    {
     "data": {
      "text/plain": "                                                CONTENT\n0     Stupidly tried BA again after a five year gap ...\n1     Not Verified |  Seat horribly narrow; 3-4-3 on...\n2     Glasgow to London delayed by 1 hour. My wife a...\n3     When I tried to check in online, I was offered...\n4     I flew from Prague to LHR. Excellent service, ...\n...                                                 ...\n1995  We were treated incredibly well and staff made...\n1996  I was scheduled to be on BA 114 from JFK to LH...\n1997  British Airways from London Heathrow to New Yo...\n1998  We flew British Airways BA2678 from Gatwick to...\n1999  Flew British Airways London Heathrow to Warsaw...\n\n[2000 rows x 1 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>CONTENT</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Stupidly tried BA again after a five year gap ...</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Not Verified |  Seat horribly narrow; 3-4-3 on...</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Glasgow to London delayed by 1 hour. My wife a...</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>When I tried to check in online, I was offered...</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>I flew from Prague to LHR. Excellent service, ...</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>1995</th>\n      <td>We were treated incredibly well and staff made...</td>\n    </tr>\n    <tr>\n      <th>1996</th>\n      <td>I was scheduled to be on BA 114 from JFK to LH...</td>\n    </tr>\n    <tr>\n      <th>1997</th>\n      <td>British Airways from London Heathrow to New Yo...</td>\n    </tr>\n    <tr>\n      <th>1998</th>\n      <td>We flew British Airways BA2678 from Gatwick to...</td>\n    </tr>\n    <tr>\n      <th>1999</th>\n      <td>Flew British Airways London Heathrow to Warsaw...</td>\n    </tr>\n  </tbody>\n</table>\n<p>2000 rows × 1 columns</p>\n</div>"
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sentiment_analysis_df = df.drop([\"REVIEW\",\"PERSONAL INFO\"], axis=1)\n",
    "sentiment_analysis_df.replace(re.compile(r'\\s*✅ Verified Review \\|\\s*'), '', inplace=True)\n",
    "sentiment_analysis_df"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "outputs": [],
   "source": [
    "# Save the DataFrame to a CSV file\n",
    "sentiment_analysis_df.to_csv(\"sentiment_content.csv\", index=False)"
   ],
   "metadata": {
    "collapsed": false,
    "pycharm": {
     "name": "#%%\n"
    }
   }
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3.8.4 64-bit",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.4"
  },
  "orig_nbformat": 4,
  "vscode": {
   "interpreter": {
    "hash": "99af3e640fd4e474a589badeb2298da6e916bd1eef1cfdb1d4d8a7dbe00511e5"
   }
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}