4a_model_testing_deployed.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "d55f5dfe",
   "metadata": {},
   "source": [
    "# Part 4a - Model testing with model deployed\n",
    "\n",
    "In this notebook we will deploy the model that we have trained to a Sagemaker Endpoint. This allows us to have the model live and running and create summaries at any time. IT also allows us to access the model via http requests, if we wanted to."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0712eb40",
   "metadata": {},
   "source": [
    "First, we define some variables which we need for our Sagemaker setup"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "dd89904c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "role = sagemaker.get_execution_role()\n",
    "bucket = sess.default_bucket()\n",
    "client = boto3.client('sagemaker')\n",
    "\n",
    "print(f\"IAM role arn used for running training: {role}\")\n",
    "print(f\"S3 bucket used for storing artifacts: {sess.default_bucket()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dd8e59d5",
   "metadata": {},
   "source": [
    "This code below allows us to access the details of the last training job. In particular we are interested in the S3 loaction of the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3b406a1e",
   "metadata": {},
   "outputs": [],
   "source": [
    "training_job = client.list_training_jobs()['TrainingJobSummaries'][0]['TrainingJobName']\n",
    "model_data = sess.describe_training_job(training_job)['ModelArtifacts']['S3ModelArtifacts']\n",
    "model_data"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "15666829",
   "metadata": {},
   "source": [
    "Now we can deploy the model to the Sagemaker endpoint. Note that we use our own inference code for this example, as it allows us to finetune the summaries better."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3e196312",
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.huggingface import HuggingFaceModel\n",
    "\n",
    "model_for_deployment = HuggingFaceModel(entry_point='inference.py',\n",
    "                                        source_dir='inference_code',\n",
    "                                        model_data=model_data,\n",
    "                                        role=role,\n",
    "                                        pytorch_version='1.7.1',\n",
    "                                        py_version='py36',\n",
    "                                        transformers_version='4.6.1',\n",
    "                                        )"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8dc3b70d",
   "metadata": {},
   "outputs": [],
   "source": [
    "predictor = model_for_deployment.deploy(initial_instance_count=1,\n",
    "                                        instance_type='ml.g4dn.xlarge',\n",
    "                                        serializer=sagemaker.serializers.JSONSerializer(),\n",
    "                                        deserializer=sagemaker.deserializers.JSONDeserializer()\n",
    "                                        )"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e1ed8d5",
   "metadata": {},
   "source": [
    "Now it's time to test the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b9d3e50b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "df_test = pd.read_csv('data/test.csv')\n",
    "ref_summaries = list(df_test['summary'])\n",
    "texts = list(df_test['text'])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "f5a03fef",
   "metadata": {},
   "outputs": [],
   "source": [
    "data = {\"inputs\":texts[0], \"parameters_list\":[{\"min_length\": 5, \"max_length\": 20}]}\n",
    "predictor.predict(data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6a5e77b3",
   "metadata": {},
   "outputs": [],
   "source": [
    "ref_summaries[0]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8f3ee9bb",
   "metadata": {},
   "outputs": [],
   "source": [
    "candidate_summaries = []\n",
    "\n",
    "for i, text in enumerate(texts):\n",
    "    if i % 100 == 0:\n",
    "        print(i)\n",
    "    data = {\"inputs\":text, \"parameters_list\":[{\"min_length\": 5, \"max_length\": 20}]}\n",
    "    candidate = predictor.predict(data)\n",
    "    candidate_summaries.append(candidate[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5681aaf9",
   "metadata": {},
   "outputs": [],
   "source": [
    "file = open(\"summaries/model-summaries.txt\", \"w\")\n",
    "for s in candidate_summaries:\n",
    "    file.write(s + \"\\n\")\n",
    "file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "268f4723",
   "metadata": {},
   "outputs": [],
   "source": [
    "from datasets import load_metric\n",
    "metric = load_metric(\"rouge\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a837312c",
   "metadata": {},
   "outputs": [],
   "source": [
    "def calc_rouge_scores(candidates, references):\n",
    "    result = metric.compute(predictions=candidates, references=references, use_stemmer=True)\n",
    "    result = {key: round(value.mid.fmeasure * 100, 1) for key, value in result.items()}\n",
    "    return result"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3da89900",
   "metadata": {},
   "outputs": [],
   "source": [
    "calc_rouge_scores(candidate_summaries, ref_summaries)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8e225d2a",
   "metadata": {},
   "source": [
    "As mentioned above, we can also fine-tune the summaries better using certain parameters. You can learn more about it in this blog post: https://huggingface.co/blog/how-to-generate. Let's try it out:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c3ae6b65",
   "metadata": {},
   "outputs": [],
   "source": [
    "candidate_summaries_refined = []\n",
    "\n",
    "for i, text in enumerate(texts):\n",
    "    if i % 100 == 0:\n",
    "        print(i)\n",
    "    data = {\"inputs\":text, \"parameters_list\":[{\"min_length\": 5, \"max_length\": 20, \"num_beams\": 50, \"top_p\": 0.9, \"do_sample\": True}]}\n",
    "    candidate = predictor.predict(data)\n",
    "    candidate_summaries_refined.append(candidate[0][0])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "b703936c",
   "metadata": {},
   "outputs": [],
   "source": [
    "file = open(\"summaries/model-summaries_refined.txt\", \"w\")\n",
    "for s in candidate_summaries_refined:\n",
    "    file.write(s + \"\\n\")\n",
    "file.close()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "58db8e6b",
   "metadata": {},
   "outputs": [],
   "source": [
    "calc_rouge_scores(candidate_summaries_refined, ref_summaries)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_latest_p37",
   "language": "python",
   "name": "conda_pytorch_latest_p37"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}