0a_data_prep_reviews.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "009fe68e",
   "metadata": {},
   "source": [
    "# Part 0 - Data preparation\n",
    "\n",
    "In this notebook we will download the Amazon Review dataset and save it to S3. We will also do some light data preprocessing by only keeping the columns we need, filtering out reviews that are too short, and limiting the size of the datasets.\n",
    "\n",
    "To read more, please check out https://towardsdatascience.com/setting-up-a-text-summarisation-project-introduction-526622eea4a8."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "23df895e",
   "metadata": {},
   "source": [
    "## Data download\n",
    "\n",
    "We download the dataset from https://huggingface.co/datasets/amazon_reviews_multi and save it to a Pandas dataframe."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "687ff753",
   "metadata": {},
   "outputs": [],
   "source": [
    "!pip install datasets"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f824c3aa",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from datasets import load_dataset\n",
    "train_ds = load_dataset(\"amazon_reviews_multi\", \"en\", split='train')\n",
    "val_ds = load_dataset(\"amazon_reviews_multi\", \"en\", split='validation')\n",
    "test_ds = load_dataset(\"amazon_reviews_multi\", \"en\", split='test')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5aa10c9d",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df_train = pd.DataFrame(train_ds)\n",
    "df_val = pd.DataFrame(val_ds)\n",
    "df_test = pd.DataFrame(test_ds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "d4ba9364",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f429804d",
   "metadata": {},
   "source": [
    "## Filtering the dataset\n",
    "\n",
    "We want to discard reviews and titles that are too short, so that our model can produce more interesting summaries."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3534fb00",
   "metadata": {},
   "outputs": [],
   "source": [
    "cutoff_title = 5\n",
    "cutoff_body = 20"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "e32d7087",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df_train[(df_train['review_title'].apply(lambda x: len(x.split()) >= cutoff_title)) & (df_train['review_body'].apply(lambda x: len(x.split()) >= cutoff_body))]\n",
    "df_val = df_val[(df_val['review_title'].apply(lambda x: len(x.split()) >= cutoff_title)) & (df_val['review_body'].apply(lambda x: len(x.split()) >= cutoff_body))]\n",
    "df_test = df_test[(df_test['review_title'].apply(lambda x: len(x.split()) >= cutoff_title)) & (df_test['review_body'].apply(lambda x: len(x.split()) >= cutoff_body))]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "541dff26",
   "metadata": {},
   "source": [
    "## Limiting the size of the datasets\n",
    "\n",
    "We want to limit the size of the datasets so that training of the model can finish in a reasonable amount of time. This is a decision that we might want to revisit in the experimentation phase if we want to increase the performance of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4e5228ea",
   "metadata": {},
   "outputs": [],
   "source": [
    "print(len(df_train), len(df_val), len(df_test))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c058601a",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train = df_train.sample(20000, random_state=42)\n",
    "df_train = df_train.rename(columns={\"review_body\": \"text\", \"review_title\": \"summary\"})\n",
    "\n",
    "df_val = df_val.sample(1000, random_state=42)\n",
    "df_val = df_val.rename(columns={\"review_body\": \"text\", \"review_title\": \"summary\"})\n",
    "\n",
    "df_test = df_test.sample(1000, random_state=42)\n",
    "df_test = df_test.rename(columns={\"review_body\": \"text\", \"review_title\": \"summary\"})"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6326cf7",
   "metadata": {},
   "source": [
    "## Save the data as CSV files and upload them to S3\n",
    "\n",
    "We need to upload the data to S3 in order to train the model at a later point."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "4537321c",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_train.to_csv('data/train.csv', index=False, columns=['text', 'summary'])\n",
    "df_val.to_csv('data/val.csv', index=False, columns=['text', 'summary'])\n",
    "df_test.to_csv('data/test.csv', index=False, columns=['text', 'summary'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "9ab2a01b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import sagemaker\n",
    "sess = sagemaker.Session()\n",
    "bucket = sess.default_bucket()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "92aab613",
   "metadata": {},
   "outputs": [],
   "source": [
    "!aws s3 cp data/train.csv s3://$bucket/summarization/data/train.csv\n",
    "!aws s3 cp data/val.csv s3://$bucket/summarization/data/val.csv\n",
    "!aws s3 cp data/test.csv s3://$bucket/summarization/data/test.csv"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_latest_p37",
   "language": "python",
   "name": "conda_pytorch_latest_p37"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}