Demo3-Semantic-search-with-pretrain-model-fullstack.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "680b50d0",
   "metadata": {},
   "source": [
    "# Deploying Semantic Search with Amazon OpenSearch Service "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ea073792",
   "metadata": {},
   "source": [
    "#### Note, we are currently testing the arthitecture using the semantic search with pretrained model. The deployment architecture is as flow. \n",
    "The deployment architecture of semantic search includes: \n",
    "- Choose a pretrain BERT model, here we use all-MiniLM-L6-v2 model\n",
    "- Save the ML models in S3 bucket\n",
    "- Host the ML models using SageMaker endpoints \n",
    "- Create Vector index and load data into the index \n",
    "- Create API gateway handels queries from web applications and pass it to lambda \n",
    "- Create a Lambda function to call SageMaker endpoints to generate embeddings from user query, and send the query results back to API gateway \n",
    "- API gateway sends the search results to frontend, and return search results to the users \n",
    "\n",
    "![Semantic_search_pretrain_fullstack](image/Semantic_search_pretrain_fullstack.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "38edc33d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2.1.0\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "print(torch.__version__)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "4307be8a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "no stored variable or alias #df_en\n"
     ]
    }
   ],
   "source": [
    "%store -r #df_en"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2efbbb9f",
   "metadata": {},
   "source": [
    "### 1. Initialize boto3\n",
    "\n",
    "We will use boto3 to interact with other AWS services.\n",
    "\n",
    "Note: You can ignore any PythonDeprecationWarning warnings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "e9a1c255",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import boto3\n",
    "import re\n",
    "import time\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "s3_resource = boto3.resource(\"s3\")\n",
    "s3 = boto3.client('s3')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c2ef698c",
   "metadata": {},
   "source": [
    "### 2. Save pre-trained all-MiniLM-L6-v2 model to S3\n",
    "\n",
    "First off, we will host a pretrained ['all-MiniLM-L6-v2'](https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2) model in a SageMaker Pytorch model server to generate 384x1 dimension fixed length sentence embedding from [sentence-transformers](https://github.com/UKPLab/sentence-transformers) using HuggingFace Transformers\n",
    "\n",
    "This SageMaker endpoint will be called by the application to generate vector for the search query. \n",
    "\n",
    "First we'll get a pre-trained model and upload to S3\n",
    "By using the model.save() method provided by the Sentence Transformers library, both the model and its associated tokenizer are saved together in the directory specified by saved_model_dir. \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "aa4a6cb2",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ec2-user/anaconda3/envs/pytorch_p310/lib/python3.10/site-packages/huggingface_hub/file_download.py:1132: FutureWarning: `resume_download` is deprecated and will be removed in version 1.0.0. Downloads always resume when possible. If you want to force a new download, use `force_download=True`.\n",
      "  warnings.warn(\n"
     ]
    }
   ],
   "source": [
    "#!pip install -U sentence-transformers\n",
    "from sentence_transformers import SentenceTransformer, util\n",
    "import numpy as np \n",
    "import os\n",
    "\n",
    "# Load the Sentence Transformer model\n",
    "model_name = \"all-MiniLM-L6-v2\"\n",
    "saved_model_dir = 'model/all-MiniLM-L6-v2'\n",
    "os.makedirs(saved_model_dir, exist_ok=True)\n",
    "\n",
    "# Load the model using the Sentence Transformers library\n",
    "model = SentenceTransformer(model_name)\n",
    "\n",
    "# Save the model (and its tokenizer) to the specified directory\n",
    "model.save(saved_model_dir)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9e0d08f0",
   "metadata": {},
   "source": [
    "Create a SageMaker session and get the execution role to be used later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "f10aac53",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "sagemaker_session = sagemaker.Session()\n",
    "role = sagemaker.get_execution_role()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1b7ee7ff",
   "metadata": {},
   "source": [
    "Pack the model, compresses all files and directories within the current directory (transformer in this case) into a single tarball archive named model.tar.gz. The archive is saved one level up from the current directory, as indicated by ../model.tar.gz."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "35d4eaaa",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1_Pooling/\n",
      "1_Pooling/config.json\n",
      "2_Normalize/\n",
      "config.json\n",
      "config_sentence_transformers.json\n",
      "model.safetensors\n",
      "modules.json\n",
      "README.md\n",
      "sentence_bert_config.json\n",
      "special_tokens_map.json\n",
      "tokenizer_config.json\n",
      "tokenizer.json\n",
      "vocab.txt\n"
     ]
    }
   ],
   "source": [
    "!cd model/all-MiniLM-L6-v2 && tar czvf ../all-MiniLM-L6-v2-pretrain.tar.gz *"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9dec51a1",
   "metadata": {},
   "source": [
    "Upload the model to S3. The method call will upload the model.tar.gz file to the S3 bucket associated with the sagemaker_session, storing it with a key that starts with the specified prefix, effectively organizing it within a folder-like structure in S3 named sentence-transformers-model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 52,
   "id": "1fa14539",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'s3://sagemaker-ca-central-1-759472643633/sentence-transformers-model/all-MiniLM-L6-v2-pretrain.tar.gz'"
      ]
     },
     "execution_count": 52,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "inputs = sagemaker_session.upload_data(path='model/all-MiniLM-L6-v2-pretrain.tar.gz', key_prefix='sentence-transformers-model')\n",
    "inputs"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f037dc39",
   "metadata": {},
   "source": [
    "### 3. Create PyTorch Model Object\n",
    "\n",
    "Next we need to create a PyTorchModel object. The deploy() method on the model object creates an endpoint which serves prediction requests in real-time. If the instance_type is set to a SageMaker instance type (e.g. ml.m5.large) then the model will be deployed on SageMaker. If the instance_type parameter is set to local then it will be deployed locally as a Docker container and ready for testing locally.\n",
    "\n",
    "We need to create a Predictor class to accept TEXT as input and output JSON. The default behaviour is to accept a numpy array."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 53,
   "id": "b60cccbe",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "from sagemaker.pytorch import PyTorch, PyTorchModel\n",
    "from sagemaker.predictor import Predictor\n",
    "from sagemaker import get_execution_role\n",
    "\n",
    "\"\"\"\n",
    "When you deploy a PyTorch model on SageMaker and want it to handle raw text input directly, you can use this StringPredictor class to create a predictor for the deployed endpoint. \n",
    "This setup is especially useful for NLP models or any model where the input is text.\n",
    "\"\"\"\n",
    "class StringPredictor(Predictor):\n",
    "    def __init__(self, endpoint_name, sagemaker_session):\n",
    "        super(StringPredictor, self).__init__(endpoint_name, sagemaker_session, content_type='text/plain')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "756b70b2",
   "metadata": {},
   "source": [
    "### 4. Deploy the sentence transformer model to SageMaker Endpoint\n",
    "Now that we have the predictor class, let's deploy a SageMaker endpoint for our application to invoke.\n",
    "\n",
    "#### Note: This process will take about 5 minutes to complete.\n",
    "\n",
    "You can ignore the \"content_type is a no-op in sagemaker>=2\" warning.\n",
    "\n",
    "This example assumes you have an inference script (inference.py) that defines how to load the model and process inputs and outputs. The framework_version should match the PyTorch version used for training the model."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 59,
   "id": "1054fd15",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "---------!"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "content_type is a no-op in sagemaker>=2.\n",
      "See: https://sagemaker.readthedocs.io/en/stable/v2.html for details.\n"
     ]
    }
   ],
   "source": [
    "pytorch_model = PyTorchModel(model_data = inputs, #s3 path to the model.tar.gz\n",
    "                             role=role, \n",
    "                             entry_point ='inference.py', ##script to process inputs and outputs\n",
    "                             source_dir = './deployment/pytorch/code',\n",
    "                             py_version = 'py310', \n",
    "                             framework_version = '2.0.1', # The PyTorch version you're using\n",
    "                             predictor_cls=StringPredictor)\n",
    "\n",
    "predictor = pytorch_model.deploy(instance_type='ml.t2.medium', #https://aws.amazon.com/sagemaker/pricing/ or ml.m5d.large\n",
    "                                 initial_instance_count=1, \n",
    "                                 endpoint_name = f'semantic-search-pretrain-all-MiniLM-L6-v2-{int(time.time())}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "839fe2af",
   "metadata": {},
   "source": [
    "### 5. Test the SageMaker Endpoint.\n",
    "\n",
    "Now that the endpoint is created, let's quickly test it out."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "3271765e-8fdb-4f7a-8fd8-dd725a90d20d",
   "metadata": {},
   "outputs": [],
   "source": [
    "#Test the predictor function for text embedding\n",
    "import json\n",
    "original_payload = 'Riverice events in ottawa'\n",
    "features = predictor.predict(original_payload)\n",
    "vector_data = json.loads(features)\n",
    "#vector"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 66,
   "id": "b70fa363",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Initialize a boto3 client for SageMaker\n",
    "import boto3\n",
    "import sagemaker\n",
    "from sagemaker import get_execution_role\n",
    "from sagemaker.huggingface.model import HuggingFaceModel\n",
    "\n",
    "# Initialize a boto3 client for SageMaker\n",
    "sagemaker_client = boto3.client('sagemaker', region_name='ca-central-1')  # Specify the AWS region\n",
    "def list_sagemaker_endpoints():\n",
    "    \"\"\"List all SageMaker endpoints\"\"\"\n",
    "    try:\n",
    "        # Get the list of all SageMaker endpoints\n",
    "        response = sagemaker_client.list_endpoints(SortBy='Name')\n",
    "        print(\"Listing SageMaker Endpoints:\")\n",
    "        for endpoint in response['Endpoints']:\n",
    "            print(f\"Endpoint Name: {endpoint['EndpointName']}, Status: {endpoint['EndpointStatus']}\")\n",
    "    except Exception as e:\n",
    "        print(f\"Error listing SageMaker endpoints: {e}\")\n",
    "\n",
    "def invoke_sagemaker_endpoint_ft(endpoint_name, payload):\n",
    "    \"\"\"Invoke a SageMaker endpoint to get predictions with ContentType='application/json'.\"\"\"\n",
    "    # Initialize the runtime SageMaker client\n",
    "    runtime_client = boto3.client('runtime.sagemaker', region_name='ca-central-1')  \n",
    "    try:\n",
    "        \"\"\"\n",
    "        if not isinstance(payload, str):\n",
    "            payload = str(payload)\n",
    "        \"\"\"\n",
    "        # Invoke the SageMaker endpoint\n",
    "        response = runtime_client.invoke_endpoint(\n",
    "            EndpointName=endpoint_name,\n",
    "            ContentType='application/json',\n",
    "            Body=json.dumps(payload)\n",
    "        )\n",
    "        # Decode the response\n",
    "        result = json.loads(response['Body'].read().decode())\n",
    "        return (result)\n",
    "        #print(f\"Prediction from {endpoint_name}: {result}\")\n",
    "    except Exception as e:\n",
    "        print(f\"Error invoking SageMaker endpoint {endpoint_name}: {e}\")\n",
    "\n",
    "def invoke_sagemaker_endpoint_pretrain(endpoint_name, payload):\n",
    "    \"\"\"Invoke a SageMaker endpoint to get predictions with ContentType='text/plain'.\"\"\"\n",
    "    # Initialize the runtime SageMaker client\n",
    "    runtime_client = boto3.client('runtime.sagemaker', region_name='ca-central-1')  \n",
    "\n",
    "    try:\n",
    "        # Ensure payload is a string, since ContentType is 'text/plain'\n",
    "        if not isinstance(payload, str):\n",
    "            payload = str(payload)\n",
    "        \n",
    "        # Invoke the SageMaker endpoint\n",
    "        response = runtime_client.invoke_endpoint(\n",
    "            EndpointName=endpoint_name,\n",
    "            ContentType='text/plain',\n",
    "            Body=payload\n",
    "        )\n",
    "        \n",
    "        # Decode the response\n",
    "        result = json.loads(response['Body'].read().decode())\n",
    "        return (result)\n",
    "        #print(f\"Prediction from {endpoint_name}: {result}\")\n",
    "    except Exception as e:\n",
    "        print(f\"Error invoking SageMaker endpoint {endpoint_name}: {e}\")\n",
    "    "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "15654d73-dbf6-447c-b5ba-45395365e922",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Listing SageMaker Endpoints:\n",
      "Endpoint Name: semantic-search-pretrain-all-MiniLM-L6-v2-1719456491, Status: InService\n"
     ]
    }
   ],
   "source": [
    "list_sagemaker_endpoints()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 68,
   "id": "5c400db5-9471-414d-9ec8-a1ceb4d474b7",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "384"
      ]
     },
     "execution_count": 68,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "endpoint_name = 'semantic-search-pretrain-all-MiniLM-L6-v2-1719456491'\n",
    "payload = \"This is an example of how to invoke SageMaker endpoints!\"\n",
    "vector = invoke_sagemaker_endpoint_pretrain(endpoint_name, payload)\n",
    "len(vector)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "ae5bceb0-2aaa-4775-8311-3d7344512fcf",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_pytorch_p310",
   "language": "python",
   "name": "conda_pytorch_p310"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}