97_Retrieval_Augmented_Generation_QandA.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "5f951f46-b561-44d0-b0e8-e94e59f74666",
   "metadata": {},
   "source": [
    "# Retrieval Augmented Generation with SageMaker\n",
    "\n",
    "Text to text Generative AI models have a well documented problem which is the issue of having only information up to the date for which they were trained. This notebook shows how to use retrieval augmented generation (RAG), otherwise known as data augmented generation, to help suppliment text generation models with up to date information via document search. We will use two different models to do this. First, we will use the HuggingFace FLAN T5 for document and question embedding. "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1a8a2d2c-dfdb-4c64-bfcb-06ac368cc5d8",
   "metadata": {},
   "source": [
    "# The Hallucination Issue\n",
    "\n",
    "Now that we have an endpoint up and running, the example below shows how the model confidently \"hallucinates\" that Switzerland won the 2022 world cup.  The actual fact is that Argentina won in 2022 and  France won in 2018. Here in lies the problem we need to fix with RAG."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 69,
   "id": "e0db0b0c-9d66-48ae-acd8-8a32a7959b03",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Some weights of the model checkpoint at google/flan-t5-large were not used when initializing T5EncoderModel: ['decoder.block.16.layer.2.DenseReluDense.wo.weight', 'decoder.block.9.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.0.SelfAttention.v.weight', 'decoder.block.10.layer.1.EncDecAttention.o.weight', 'decoder.block.5.layer.0.SelfAttention.o.weight', 'decoder.block.7.layer.0.SelfAttention.o.weight', 'decoder.block.18.layer.1.EncDecAttention.k.weight', 'decoder.block.13.layer.0.SelfAttention.k.weight', 'decoder.block.12.layer.0.SelfAttention.o.weight', 'decoder.block.10.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.6.layer.0.SelfAttention.q.weight', 'decoder.block.13.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.2.layer_norm.weight', 'decoder.block.18.layer.2.DenseReluDense.wo.weight', 'decoder.block.7.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.0.SelfAttention.v.weight', 'decoder.block.5.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.2.layer.1.EncDecAttention.q.weight', 'decoder.block.11.layer.1.EncDecAttention.v.weight', 'decoder.block.13.layer.2.layer_norm.weight', 'decoder.block.23.layer.2.layer_norm.weight', 'decoder.block.8.layer.0.SelfAttention.o.weight', 'decoder.block.17.layer.0.SelfAttention.o.weight', 'decoder.block.8.layer.1.layer_norm.weight', 'decoder.block.0.layer.1.EncDecAttention.o.weight', 'decoder.block.20.layer.1.EncDecAttention.k.weight', 'decoder.block.4.layer.0.SelfAttention.k.weight', 'decoder.block.4.layer.1.EncDecAttention.v.weight', 'decoder.block.11.layer.0.SelfAttention.o.weight', 'decoder.block.18.layer.1.EncDecAttention.q.weight', 'decoder.block.10.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.0.SelfAttention.k.weight', 'decoder.block.19.layer.1.EncDecAttention.k.weight', 'decoder.block.0.layer.0.SelfAttention.k.weight', 'decoder.block.14.layer.0.SelfAttention.k.weight', 'decoder.block.9.layer.2.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.k.weight', 'decoder.block.10.layer.1.EncDecAttention.q.weight', 'decoder.block.14.layer.1.EncDecAttention.q.weight', 'decoder.block.19.layer.0.SelfAttention.q.weight', 'decoder.block.5.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.2.layer_norm.weight', 'decoder.block.20.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.2.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.15.layer.2.layer_norm.weight', 'decoder.block.0.layer.0.SelfAttention.q.weight', 'decoder.block.9.layer.0.SelfAttention.k.weight', 'decoder.block.13.layer.2.DenseReluDense.wo.weight', 'decoder.block.19.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.0.SelfAttention.o.weight', 'decoder.block.21.layer.0.layer_norm.weight', 'decoder.block.10.layer.0.SelfAttention.q.weight', 'decoder.block.8.layer.0.SelfAttention.q.weight', 'decoder.block.22.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.1.EncDecAttention.q.weight', 'decoder.block.7.layer.1.layer_norm.weight', 'decoder.block.1.layer.1.EncDecAttention.q.weight', 'decoder.block.12.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.21.layer.2.layer_norm.weight', 'decoder.block.12.layer.1.layer_norm.weight', 'decoder.block.1.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.k.weight', 'decoder.block.19.layer.0.layer_norm.weight', 'decoder.block.23.layer.0.SelfAttention.k.weight', 'decoder.block.11.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.3.layer.0.layer_norm.weight', 'decoder.block.2.layer.0.SelfAttention.o.weight', 'decoder.block.18.layer.0.layer_norm.weight', 'decoder.block.6.layer.0.SelfAttention.k.weight', 'decoder.block.20.layer.0.SelfAttention.v.weight', 'decoder.block.19.layer.0.SelfAttention.k.weight', 'decoder.block.8.layer.0.SelfAttention.v.weight', 'decoder.block.9.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.0.SelfAttention.q.weight', 'decoder.block.8.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.6.layer.0.layer_norm.weight', 'decoder.block.16.layer.1.EncDecAttention.v.weight', 'decoder.block.9.layer.0.SelfAttention.o.weight', 'decoder.block.1.layer.0.SelfAttention.v.weight', 'decoder.block.13.layer.0.layer_norm.weight', 'decoder.block.16.layer.1.EncDecAttention.o.weight', 'decoder.block.19.layer.1.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.o.weight', 'decoder.block.14.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.22.layer.1.EncDecAttention.q.weight', 'decoder.block.16.layer.0.SelfAttention.k.weight', 'decoder.block.18.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.10.layer.1.layer_norm.weight', 'decoder.block.23.layer.0.SelfAttention.q.weight', 'decoder.block.3.layer.2.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.o.weight', 'decoder.block.19.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.16.layer.0.SelfAttention.v.weight', 'decoder.block.9.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.17.layer.0.SelfAttention.q.weight', 'decoder.block.11.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.1.EncDecAttention.v.weight', 'decoder.block.0.layer.0.SelfAttention.v.weight', 'decoder.block.23.layer.0.SelfAttention.o.weight', 'decoder.block.13.layer.0.SelfAttention.o.weight', 'lm_head.weight', 'decoder.block.6.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.19.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.13.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.4.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.12.layer.1.EncDecAttention.o.weight', 'decoder.block.7.layer.0.layer_norm.weight', 'decoder.block.11.layer.0.layer_norm.weight', 'decoder.block.6.layer.1.EncDecAttention.k.weight', 'decoder.block.6.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.q.weight', 'decoder.block.4.layer.1.EncDecAttention.o.weight', 'decoder.block.16.layer.1.EncDecAttention.k.weight', 'decoder.block.3.layer.1.EncDecAttention.k.weight', 'decoder.block.12.layer.1.EncDecAttention.q.weight', 'decoder.block.13.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.3.layer.1.layer_norm.weight', 'decoder.block.20.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.18.layer.0.SelfAttention.k.weight', 'decoder.block.12.layer.0.SelfAttention.k.weight', 'decoder.block.17.layer.1.EncDecAttention.v.weight', 'decoder.block.14.layer.1.EncDecAttention.k.weight', 'decoder.block.3.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.22.layer.1.EncDecAttention.k.weight', 'decoder.block.13.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.2.layer_norm.weight', 'decoder.block.17.layer.0.layer_norm.weight', 'decoder.block.21.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.2.layer.0.SelfAttention.k.weight', 'decoder.block.10.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.10.layer.1.EncDecAttention.k.weight', 'decoder.block.14.layer.0.SelfAttention.v.weight', 'decoder.block.11.layer.1.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.v.weight', 'decoder.block.23.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.21.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.o.weight', 'decoder.block.22.layer.0.layer_norm.weight', 'decoder.block.3.layer.0.SelfAttention.q.weight', 'decoder.block.10.layer.0.SelfAttention.v.weight', 'decoder.block.7.layer.2.DenseReluDense.wo.weight', 'decoder.block.8.layer.2.layer_norm.weight', 'decoder.block.11.layer.2.layer_norm.weight', 'decoder.block.22.layer.0.SelfAttention.q.weight', 'decoder.block.20.layer.1.EncDecAttention.o.weight', 'decoder.block.16.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.14.layer.1.layer_norm.weight', 'decoder.block.4.layer.0.SelfAttention.v.weight', 'decoder.block.5.layer.2.layer_norm.weight', 'decoder.block.14.layer.0.layer_norm.weight', 'decoder.block.23.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.7.layer.0.SelfAttention.k.weight', 'decoder.block.6.layer.1.layer_norm.weight', 'decoder.block.16.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.0.layer_norm.weight', 'decoder.block.15.layer.1.EncDecAttention.k.weight', 'decoder.block.2.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.1.layer_norm.weight', 'decoder.block.21.layer.1.EncDecAttention.q.weight', 'decoder.block.21.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.13.layer.0.SelfAttention.v.weight', 'decoder.block.9.layer.0.SelfAttention.q.weight', 'decoder.block.2.layer.1.EncDecAttention.k.weight', 'decoder.block.15.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.19.layer.0.SelfAttention.v.weight', 'decoder.block.11.layer.2.DenseReluDense.wo.weight', 'decoder.block.4.layer.1.EncDecAttention.k.weight', 'decoder.block.9.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.18.layer.2.layer_norm.weight', 'decoder.block.11.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.0.SelfAttention.k.weight', 'decoder.block.8.layer.1.EncDecAttention.o.weight', 'decoder.block.12.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.8.layer.0.SelfAttention.k.weight', 'decoder.block.3.layer.0.SelfAttention.k.weight', 'decoder.block.15.layer.0.layer_norm.weight', 'decoder.block.8.layer.0.layer_norm.weight', 'decoder.block.10.layer.0.SelfAttention.k.weight', 'decoder.block.17.layer.2.layer_norm.weight', 'decoder.block.17.layer.0.SelfAttention.k.weight', 'decoder.embed_tokens.weight', 'decoder.block.18.layer.1.layer_norm.weight', 'decoder.block.2.layer.2.DenseReluDense.wo.weight', 'decoder.block.4.layer.2.layer_norm.weight', 'decoder.block.22.layer.2.layer_norm.weight', 'decoder.block.5.layer.1.EncDecAttention.v.weight', 'decoder.block.19.layer.2.DenseReluDense.wo.weight', 'decoder.block.15.layer.0.SelfAttention.v.weight', 'decoder.block.14.layer.1.EncDecAttention.o.weight', 'decoder.final_layer_norm.weight', 'decoder.block.21.layer.2.DenseReluDense.wo.weight', 'decoder.block.6.layer.2.layer_norm.weight', 'decoder.block.18.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.1.EncDecAttention.q.weight', 'decoder.block.10.layer.2.layer_norm.weight', 'decoder.block.7.layer.1.EncDecAttention.v.weight', 'decoder.block.3.layer.1.EncDecAttention.o.weight', 'decoder.block.22.layer.1.layer_norm.weight', 'decoder.block.12.layer.0.SelfAttention.v.weight', 'decoder.block.14.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.20.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.2.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.17.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.16.layer.1.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.q.weight', 'decoder.block.8.layer.2.DenseReluDense.wo.weight', 'decoder.block.12.layer.1.EncDecAttention.k.weight', 'decoder.block.11.layer.1.EncDecAttention.q.weight', 'decoder.block.11.layer.0.SelfAttention.v.weight', 'decoder.block.23.layer.0.SelfAttention.v.weight', 'decoder.block.14.layer.0.SelfAttention.o.weight', 'decoder.block.4.layer.2.DenseReluDense.wo.weight', 'decoder.block.22.layer.0.SelfAttention.v.weight', 'decoder.block.17.layer.1.EncDecAttention.k.weight', 'decoder.block.15.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.3.layer.0.SelfAttention.o.weight', 'decoder.block.8.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.23.layer.1.EncDecAttention.q.weight', 'decoder.block.0.layer.1.EncDecAttention.q.weight', 'decoder.block.6.layer.0.SelfAttention.o.weight', 'decoder.block.15.layer.0.SelfAttention.q.weight', 'decoder.block.20.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.11.layer.1.EncDecAttention.k.weight', 'decoder.block.4.layer.1.layer_norm.weight', 'decoder.block.21.layer.0.SelfAttention.o.weight', 'decoder.block.18.layer.0.SelfAttention.q.weight', 'decoder.block.13.layer.0.SelfAttention.q.weight', 'decoder.block.1.layer.0.SelfAttention.q.weight', 'decoder.block.23.layer.1.layer_norm.weight', 'decoder.block.9.layer.1.EncDecAttention.v.weight', 'decoder.block.20.layer.0.layer_norm.weight', 'decoder.block.1.layer.1.layer_norm.weight', 'decoder.block.18.layer.0.SelfAttention.o.weight', 'decoder.block.0.layer.1.EncDecAttention.v.weight', 'decoder.block.22.layer.1.EncDecAttention.v.weight', 'decoder.block.8.layer.1.EncDecAttention.v.weight', 'decoder.block.4.layer.0.SelfAttention.q.weight', 'decoder.block.15.layer.0.SelfAttention.k.weight', 'decoder.block.1.layer.2.DenseReluDense.wo.weight', 'decoder.block.11.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.12.layer.2.DenseReluDense.wo.weight', 'decoder.block.6.layer.1.EncDecAttention.q.weight', 'decoder.block.1.layer.1.EncDecAttention.o.weight', 'decoder.block.14.layer.0.SelfAttention.q.weight', 'decoder.block.16.layer.2.layer_norm.weight', 'decoder.block.23.layer.1.EncDecAttention.o.weight', 'decoder.block.13.layer.1.layer_norm.weight', 'decoder.block.5.layer.0.SelfAttention.v.weight', 'decoder.block.7.layer.1.EncDecAttention.o.weight', 'decoder.block.2.layer.0.SelfAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.k.weight', 'decoder.block.1.layer.0.layer_norm.weight', 'decoder.block.17.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.9.layer.1.layer_norm.weight', 'decoder.block.19.layer.1.EncDecAttention.v.weight', 'decoder.block.23.layer.0.layer_norm.weight', 'decoder.block.10.layer.2.DenseReluDense.wo.weight', 'decoder.block.3.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.2.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.1.layer.1.EncDecAttention.v.weight', 'decoder.block.5.layer.0.SelfAttention.q.weight', 'decoder.block.3.layer.1.EncDecAttention.q.weight', 'decoder.block.3.layer.0.SelfAttention.v.weight', 'decoder.block.5.layer.2.DenseReluDense.wo.weight', 'decoder.block.20.layer.2.layer_norm.weight', 'decoder.block.1.layer.1.EncDecAttention.k.weight', 'decoder.block.9.layer.0.layer_norm.weight', 'decoder.block.18.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.0.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.2.layer.1.layer_norm.weight', 'decoder.block.22.layer.2.DenseReluDense.wo.weight', 'decoder.block.19.layer.2.layer_norm.weight', 'decoder.block.8.layer.1.EncDecAttention.k.weight', 'decoder.block.5.layer.1.EncDecAttention.k.weight', 'decoder.block.12.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.1.EncDecAttention.k.weight', 'decoder.block.18.layer.0.SelfAttention.v.weight', 'decoder.block.2.layer.0.SelfAttention.v.weight', 'decoder.block.12.layer.1.EncDecAttention.v.weight', 'decoder.block.17.layer.2.DenseReluDense.wo.weight', 'decoder.block.0.layer.2.DenseReluDense.wo.weight', 'decoder.block.4.layer.0.layer_norm.weight', 'decoder.block.18.layer.1.EncDecAttention.o.weight', 'decoder.block.6.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight', 'decoder.block.16.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.22.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.17.layer.1.layer_norm.weight', 'decoder.block.22.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.20.layer.1.EncDecAttention.v.weight', 'decoder.block.16.layer.0.layer_norm.weight', 'decoder.block.2.layer.0.layer_norm.weight', 'decoder.block.23.layer.2.DenseReluDense.wo.weight', 'decoder.block.10.layer.0.layer_norm.weight', 'decoder.block.14.layer.1.EncDecAttention.v.weight', 'decoder.block.21.layer.0.SelfAttention.q.weight', 'decoder.block.7.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.21.layer.1.EncDecAttention.o.weight', 'decoder.block.7.layer.2.layer_norm.weight', 'decoder.block.1.layer.2.DenseReluDense.wi_0.weight', 'decoder.block.15.layer.1.EncDecAttention.q.weight', 'decoder.block.17.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.2.DenseReluDense.wo.weight', 'decoder.block.7.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.5.layer.1.layer_norm.weight', 'decoder.block.14.layer.2.layer_norm.weight', 'decoder.block.22.layer.0.SelfAttention.k.weight', 'decoder.block.5.layer.1.EncDecAttention.o.weight', 'decoder.block.21.layer.1.EncDecAttention.k.weight', 'decoder.block.1.layer.2.DenseReluDense.wi_1.weight', 'decoder.block.20.layer.0.SelfAttention.q.weight', 'decoder.block.13.layer.1.EncDecAttention.q.weight', 'decoder.block.6.layer.2.DenseReluDense.wo.weight', 'decoder.block.16.layer.0.SelfAttention.o.weight', 'decoder.block.19.layer.0.SelfAttention.o.weight', 'decoder.block.7.layer.1.EncDecAttention.q.weight', 'decoder.block.15.layer.1.layer_norm.weight', 'decoder.block.12.layer.0.layer_norm.weight', 'decoder.block.2.layer.1.EncDecAttention.v.weight', 'decoder.block.10.layer.0.SelfAttention.o.weight', 'decoder.block.20.layer.1.layer_norm.weight', 'decoder.block.0.layer.1.layer_norm.weight', 'decoder.block.14.layer.2.DenseReluDense.wo.weight', 'decoder.block.5.layer.1.EncDecAttention.q.weight', 'decoder.block.8.layer.1.EncDecAttention.q.weight', 'decoder.block.23.layer.1.EncDecAttention.v.weight', 'decoder.block.9.layer.0.SelfAttention.v.weight', 'decoder.block.3.layer.2.DenseReluDense.wo.weight', 'decoder.block.6.layer.0.SelfAttention.v.weight', 'decoder.block.15.layer.0.SelfAttention.o.weight', 'decoder.block.5.layer.0.layer_norm.weight', 'decoder.block.16.layer.0.SelfAttention.q.weight', 'decoder.block.22.layer.1.EncDecAttention.o.weight', 'decoder.block.13.layer.1.EncDecAttention.v.weight', 'decoder.block.6.layer.1.EncDecAttention.o.weight', 'decoder.block.5.layer.2.DenseReluDense.wi_1.weight']\n",
      "- This IS expected if you are initializing T5EncoderModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n",
      "- This IS NOT expected if you are initializing T5EncoderModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n"
     ]
    }
   ],
   "source": [
    "import transformers\n",
    "import torch\n",
    "import pandas as pd\n",
    "import numpy as np\n",
    "\n",
    "import json\n",
    "import boto3\n",
    "from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, T5EncoderModel\n",
    "\n",
    "# Note:  we need to use at least t5-large, otherwise the embeddings do not match the right document later.\n",
    "#        it might be good to actually show this, maybe?  and show the similarity search results later?\n",
    "\n",
    "tokenizer = AutoTokenizer.from_pretrained(\"google/flan-t5-large\")\n",
    "model = AutoModelForSeq2SeqLM.from_pretrained(\"google/flan-t5-large\")\n",
    "encoder_model = T5EncoderModel.from_pretrained(\"google/flan-t5-large\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7e1d8e41-4607-403e-828c-f2eca2babfcc",
   "metadata": {
    "tags": []
   },
   "source": [
    "# IGNORE THE WARNING ^^ ABOVE ^^"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "602c7fee-0386-410a-9c1c-dfdbe13f60dc",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer the following question.\n",
      "Question: Who won the 2022 world cup?\n",
      "Answer:\n",
      "argentina\n"
     ]
    }
   ],
   "source": [
    "prompt = f'''Answer the following question.\n",
    "Question: Who won the 2022 world cup?\n",
    "Answer:\n",
    "'''\n",
    "inputs = tokenizer(prompt, return_tensors='pt')\n",
    "output = tokenizer.decode(\n",
    "    model.generate(\n",
    "        input_ids=inputs[\"input_ids\"], \n",
    "        max_new_tokens=200,\n",
    "    )[0], \n",
    "    skip_special_tokens=True\n",
    ")\n",
    "\n",
    "\n",
    "print(f'{prompt}{output}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c8fa7a33-0829-49fd-9c23-7900620bf6f3",
   "metadata": {},
   "source": [
    "In this cell promt engineering with adding the line `If you do not have the information to answer the question, say \"I don't know\".` to the prompt produces the answer of \"I don't know\" which is better than producing a wrong answer."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 71,
   "id": "a5edd310-9bf6-4451-81f4-c35e4aaced6d",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Answer the following question. If you do not have the information to answer the question, say \"I don't know\".\n",
      "Question: Who won the 2022 world cup?\n",
      "Answer:\n",
      "I don't know\n"
     ]
    }
   ],
   "source": [
    "prompt = f'''Answer the following question. If you do not have the information to answer the question, say \"I don't know\".\n",
    "Question: Who won the 2022 world cup?\n",
    "Answer:\n",
    "'''\n",
    "inputs = tokenizer(prompt, return_tensors='pt')\n",
    "output = tokenizer.decode(\n",
    "    model.generate(\n",
    "        input_ids=inputs[\"input_ids\"], \n",
    "        max_new_tokens=200,\n",
    "    )[0], \n",
    "    skip_special_tokens=True\n",
    ")\n",
    "\n",
    "print(f'{prompt}{output}')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2b58c05c-dcd9-41fb-bbd8-ea05ee77cb12",
   "metadata": {},
   "source": [
    "# Get a HuggingFace Model for Embeddings\n",
    "\n",
    "Load in the [FLAN T5 large model](https://huggingface.co/google/flan-t5-large) from HuggingFace. This will be the model we use to create our document search embeddings."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "dc2e9d49-5d5a-4d85-832c-d40b9bb8dd90",
   "metadata": {},
   "source": [
    "# Create Embedding Database\n",
    "\n",
    "Now use the HuggingFace model to create embeddings for each of the three documents which have been provided. The documents used here are only illustrative. These documents could be extended to any collection of text to help supplement your use case."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "8c4f13d4-7f74-4302-8f4f-dc694e3c81e5",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "import torch\n",
    "\n",
    "def get_embedding(text, encoder_model, tokenizer):\n",
    "    with torch.no_grad():\n",
    "        input_ids = tokenizer(\n",
    "            text, return_tensors=\"pt\", max_length=512, truncation=True\n",
    "        ).input_ids #.to(DEVICE)\n",
    "        outputs = encoder_model(input_ids=input_ids)\n",
    "        last_hidden_states = outputs.last_hidden_state\n",
    "        e = last_hidden_states.mean(dim=1)\n",
    "    return e\n",
    "\n",
    "def create_doc_database(docs, encoder_model, tokenizer):\n",
    "    database = []\n",
    "    for i in range(docs.shape[0]):\n",
    "        text = docs['title'].values[i] + ' - ' + docs['document'].values[i]\n",
    "        e = get_embedding(text, encoder_model, tokenizer)\n",
    "        database.append(e)\n",
    "    database = torch.cat(database)\n",
    "    return database"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 73,
   "id": "a1494a60-f483-48c7-9b35-26d9d58cf2a5",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>title</th>\n",
       "      <th>document</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>world cup 2022</td>\n",
       "      <td>\"Lionel Messi can finally be called a world ch...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>champions league 2022</td>\n",
       "      <td>'Real Madrid were crowned European champions f...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>ballon dor 2022</td>\n",
       "      <td>'Real Madrid and France\\'s Karim Benzema has w...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                   title                                           document\n",
       "0         world cup 2022  \"Lionel Messi can finally be called a world ch...\n",
       "1  champions league 2022  'Real Madrid were crowned European champions f...\n",
       "2        ballon dor 2022  'Real Madrid and France\\'s Karim Benzema has w..."
      ]
     },
     "execution_count": 73,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "\n",
    "docs = pd.read_csv('document-corpus.txt', delimiter=\"::: \", engine='python')\n",
    "docs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 74,
   "id": "0a915a9c-9cc5-4bdb-92c4-1416fbc396ef",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "database = create_doc_database(docs, encoder_model, tokenizer)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 75,
   "id": "6935bbc8-18d3-4ff1-8fd8-35226e02a055",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "torch.Size([3, 1024])"
      ]
     },
     "execution_count": 75,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "database.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3c316a51-1df2-4a25-805c-f5a323de271b",
   "metadata": {},
   "source": [
    "# Add document-search feature\n",
    "\n",
    "Now that you have a database of embeddings, we can search the database against a text input `\"Who won the 2022 world cup?\"` to see which document is most relevant to the question by looking at the dot product of the embeddings."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 76,
   "id": "6f1ef481-c6e2-4a6a-b2b1-644b1782c163",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def search_database(search_embedding, database):\n",
    "    similarities = []\n",
    "    for i in range(database.shape[0]):\n",
    "        similarities.append(\n",
    "            float(torch.dot(search_embedding[0], database[i]))\n",
    "        )\n",
    "    return np.argmax(similarities), similarities"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 77,
   "id": "20e075b2-591e-4348-b135-1708e6600539",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Input: Who won the 2022 world cup?\n",
      "Was matched with document #0 which is titled \"world cup 2022\"\n"
     ]
    }
   ],
   "source": [
    "search = 'Who won the 2022 world cup?'\n",
    "search_embedding = get_embedding(search, encoder_model, tokenizer)\n",
    "doc_index, similarities = search_database(search_embedding, database)\n",
    "print(f\"Input: {search}\\nWas matched with document #{doc_index} which is titled \\\"{docs.loc[doc_index]['title']}\\\"\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98d6bbf7-adf0-436c-aa76-c900d7d1a8c3",
   "metadata": {},
   "source": [
    "# Dynamically Engineer the Prompt\n",
    "\n",
    "Now that we have a user input matched with a relevant document, we can engineer a prompt which includes both the question and context from the document."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "d78f1d20-2a3c-4a29-a68c-7cfd1b6f0657",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "prompt_eng_base = '''Answer the following question with the following context. If you do not have the information to answer the question, say \"I don't know\".\n",
    "\n",
    "Context: [PLACE DOC HERE]\n",
    "\n",
    "Question: [PLACE QUESTION HERE]\n",
    "Answer:\n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 79,
   "id": "9cd1702d-abec-43ce-a555-d39150c6818e",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "def make_prompt(search, context, prompt_eng_base):\n",
    "    prompt = prompt_eng_base.replace('[PLACE DOC HERE]', context)\n",
    "    prompt = prompt.replace('[PLACE QUESTION HERE]', search)\n",
    "    return prompt"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "cde37f7b-4284-4045-91d4-f17efbd16dda",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'Answer the following question with the following context. If you do not have the information to answer the question, say \"I don\\'t know\".\\n\\nContext: \"Lionel Messi can finally be called a world champion.Messi scored twice in one of the most epic soccer games anyone has ever watched as Argentina won the 2022 FIFA World Cup Final over France on penalties.The climactic match in Qatar finished 3-3 after extra time, with La Albiceleste claiming the shootout by a 4-2 margin.Argentina held a comfortable 2-0 lead until the 80th minute courtesy of a Messi penalty and a sublime team goal finished by Ángel Di María in the first half. However, Kylian Mbappé converted from the spot and finished a sumptuous volley in a span of two minutes to send the game to extra time.Eugene Omoruyi Scores 17 Points vs. Indiana PacersMessi was once again on hand to put Argentina in front in the 108th minute, but Mbappé kept his cool from the penalty spot once more to send the final to a shootout.Messi knocked home the first penalty, then Argentina keeper Emiliano Martínez made a save in the second round from Kingsley Coman\\'s effort. France\\'s Aurélien Tchouaméni sent his spot-kick wide in the third round, leaving Gonzalo Montiel to seal the title in the fourth.✨ Watch more top videos, highlights, and B/R original contentArgentina appeared to have full control of the contest thanks to a pair of first-half goals from Messi and Di María.Messi became the first man to score in all four knockout round games of a single World Cup with his penalty in the 23rd minute after Di María was brought down by Ousmane Dembélé.Di María padded the lead after a scintillating counterattack that started in Argentina\\'s own half.France did not attempt a single shot until the 71st minute, and La Albiceleste appeared to be cruising. However, the game flipped on its head when they conceded a penalty in the 80th minute.Mbappé coolly converted, and the final burst into life once more at the Lusail Stadium.One minute later, Mbappé levelled the contest at two goals apiece with a stunning volley, eventually sending the final to another 30 minutes of game time.Messi and Argentina thought they had the World Cup locked up again in the 108th minute, when the 35-year-old finished off a chaotic sequence in front of Hugo Lloris in the French net.But Mbappé struck again from the penalty spot after a handball from Gonzalo Montiel in the box 10 minutes after Messi\\'s second goal.Mbappé\\'s hat trick pushed him ahead of Messi into first place in the Golden Boot race with seven goals.That was a small consolation for the French superstar after losing in his second World Cup final.Argentina\\'s stars were faultless with their spot-kicks, while France was left to rue Coman\\'s saved effort and Tchouaméni\\'s miss, handing a third world title to Argentina and preventing Les Bleus from defending their 2018 tournament victory in Russia.While a penalty shootout is always a cruel way to lose, Argentina was good value for the win on the balance of play. France finished with just five shots on target to La Albiceleste\\'s 10 and failed to do anything in the final third in the first 70 minutes. Emiliano Martínez made just two saves for Argentina, albeit one was crucial in the 123rd minute when left one-on-one with Randal Kolo Muani.Messi was, perhaps inevitably, Man of the Match for his pair of goal contributions.He has now won a major tournament title in each of the last two years with Argentina, securing his first Copa America in 2021 and now adding a World Cup title to his illustrious resume.\"\\n\\nQuestion: Who won the 2022 world cup?\\nAnswer:\\n'"
      ]
     },
     "execution_count": 80,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "augmented_prompt = make_prompt(search, docs.loc[doc_index]['document'], prompt_eng_base)\n",
    "augmented_prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88b0bdf8-540b-464f-a556-97f26a147d0c",
   "metadata": {},
   "source": [
    "# Wrap the RAG Flow into a Function"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 81,
   "id": "1edb71c1-af16-4e14-b305-ea3f327c3d9c",
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "base_prompt = f'''Answer the following question. If you do not have the information to answer the question, say \"I don't know\".\n",
    "\n",
    "Question: [SEARCH HERE]\n",
    "Answer:\n",
    "'''\n",
    "\n",
    "def rag_demo(search, use_search=True):\n",
    "    search_embedding = get_embedding(search, encoder_model, tokenizer)\n",
    "    print('Returned embedding size {} for input {}'.format(search_embedding.shape, search))\n",
    "    doc_index, similarities = search_database(search_embedding, database)\n",
    "    if use_search:\n",
    "        augmented_prompt = make_prompt(search, docs.loc[doc_index]['document'], prompt_eng_base)\n",
    "    else:\n",
    "        augmented_prompt = base_prompt.replace('[SEARCH HERE]', search)\n",
    "\n",
    "    inputs = tokenizer(augmented_prompt, return_tensors='pt')\n",
    "        \n",
    "    output = tokenizer.decode(\n",
    "        model.generate(\n",
    "            input_ids=inputs[\"input_ids\"], \n",
    "            max_new_tokens=200,\n",
    "        )[0], \n",
    "        skip_special_tokens=True\n",
    "    )\n",
    "    return output"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c4f239b9-1eab-4cc3-a79a-4af44a6f1082",
   "metadata": {},
   "source": [
    "# Example Outputs\n",
    "\n",
    "The outputs below show how you can now get relevant information to the model in order to give informed responses back to the user!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 82,
   "id": "d984d679-a8bf-42af-92b5-a31e95eaa573",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Returned embedding size torch.Size([1, 1024]) for input Who won the 2022 world cup?\n",
      "I don't know\n"
     ]
    }
   ],
   "source": [
    "out = rag_demo(\n",
    "    'Who won the 2022 world cup?', use_search=False\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 83,
   "id": "8c285e66-eb66-423e-9536-db5482268594",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Token indices sequence length is longer than the specified maximum sequence length for this model (956 > 512). Running this sequence through the model will result in indexing errors\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Returned embedding size torch.Size([1, 1024]) for input Who won the 2022 world cup?\n",
      "Argentina\n"
     ]
    }
   ],
   "source": [
    "out = rag_demo(\n",
    "    'Who won the 2022 world cup?', use_search=True\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "f5cf7907-9c2e-43ac-8063-e4fa56599430",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Returned embedding size torch.Size([1, 1024]) for input Who were a 5 of the most important players in the 2022 world cup final?\n",
      "I don't know\n"
     ]
    }
   ],
   "source": [
    "out = rag_demo(\n",
    "    'Who were a 5 of the most important players in the 2022 world cup final?', use_search=False\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 85,
   "id": "29aaf3c4-3dfe-444c-babe-c4d6e936cb39",
   "metadata": {
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Returned embedding size torch.Size([1, 1024]) for input Who were a 5 of the most important players in the 2022 world cup final?\n",
      "Lionel Messi, ngel Di Mara, Kylian Mbappé, Kingsley Coman, and Aurélien Tchouaméni\n"
     ]
    }
   ],
   "source": [
    "out = rag_demo(\n",
    "    'Who were a 5 of the most important players in the 2022 world cup final?', use_search=True\n",
    ")\n",
    "print(out)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8cfa3c01-8dda-4241-8a99-94210220070b",
   "metadata": {},
   "source": [
    "# Suggested Next Steps\n",
    "\n",
    "* Explore libraries which can help with this kind of workflow. See: [LangChain](https://github.com/hwchase17/langchain)\n",
    "* Bring your own documents or information to this workflow to explore creating RAG based systems.\n",
    "* Look into fine tuning your embedding model to produce better searching.\n",
    "* Integrate this RAG flow with integrations to your own search capabilities."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8e72fe76-2671-418f-bee1-4418c25b742e",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "availableInstances": [
   {
    "_defaultOrder": 0,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.t3.medium",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 1,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.t3.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 2,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.t3.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 3,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.t3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 4,
    "_isFastLaunch": true,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 5,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 6,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 7,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 8,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 9,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 10,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 11,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 12,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.m5d.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 13,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.m5d.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 14,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.m5d.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 15,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.m5d.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 16,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.m5d.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 17,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.m5d.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 18,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.m5d.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 19,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.m5d.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 20,
    "_isFastLaunch": false,
    "category": "General purpose",
    "gpuNum": 0,
    "hideHardwareSpecs": true,
    "memoryGiB": 0,
    "name": "ml.geospatial.interactive",
    "supportedImageNames": [
     "sagemaker-geospatial-v1-0"
    ],
    "vcpuNum": 0
   },
   {
    "_defaultOrder": 21,
    "_isFastLaunch": true,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 4,
    "name": "ml.c5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 22,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 8,
    "name": "ml.c5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 23,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.c5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 24,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.c5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 25,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 72,
    "name": "ml.c5.9xlarge",
    "vcpuNum": 36
   },
   {
    "_defaultOrder": 26,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 96,
    "name": "ml.c5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 27,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 144,
    "name": "ml.c5.18xlarge",
    "vcpuNum": 72
   },
   {
    "_defaultOrder": 28,
    "_isFastLaunch": false,
    "category": "Compute optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.c5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 29,
    "_isFastLaunch": true,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g4dn.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 30,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g4dn.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 31,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g4dn.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 32,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g4dn.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 33,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g4dn.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 34,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g4dn.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 35,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 61,
    "name": "ml.p3.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 36,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 244,
    "name": "ml.p3.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 37,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 488,
    "name": "ml.p3.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 38,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.p3dn.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 39,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.r5.large",
    "vcpuNum": 2
   },
   {
    "_defaultOrder": 40,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.r5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 41,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.r5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 42,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.r5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 43,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.r5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 44,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.r5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 45,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 512,
    "name": "ml.r5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 46,
    "_isFastLaunch": false,
    "category": "Memory Optimized",
    "gpuNum": 0,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.r5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 47,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 16,
    "name": "ml.g5.xlarge",
    "vcpuNum": 4
   },
   {
    "_defaultOrder": 48,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 32,
    "name": "ml.g5.2xlarge",
    "vcpuNum": 8
   },
   {
    "_defaultOrder": 49,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 64,
    "name": "ml.g5.4xlarge",
    "vcpuNum": 16
   },
   {
    "_defaultOrder": 50,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 128,
    "name": "ml.g5.8xlarge",
    "vcpuNum": 32
   },
   {
    "_defaultOrder": 51,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 1,
    "hideHardwareSpecs": false,
    "memoryGiB": 256,
    "name": "ml.g5.16xlarge",
    "vcpuNum": 64
   },
   {
    "_defaultOrder": 52,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 192,
    "name": "ml.g5.12xlarge",
    "vcpuNum": 48
   },
   {
    "_defaultOrder": 53,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 4,
    "hideHardwareSpecs": false,
    "memoryGiB": 384,
    "name": "ml.g5.24xlarge",
    "vcpuNum": 96
   },
   {
    "_defaultOrder": 54,
    "_isFastLaunch": false,
    "category": "Accelerated computing",
    "gpuNum": 8,
    "hideHardwareSpecs": false,
    "memoryGiB": 768,
    "name": "ml.g5.48xlarge",
    "vcpuNum": 192
   }
  ],
  "instance_type": "ml.m5.2xlarge",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}