LlamaIndex_Orca-2-13B-RAG.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Orca 2 13B Chat\n",
    "\n",
    "This notebook demonstrates the use of LlamaIndex for Retrieval Augmented Generation in Linux and with Nvidia's CUDA.\n",
    "\n",
    "See the [README.md](README.md) file for help on how to run this."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 1. Prepare Llama Index for use"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/mark/miniconda3/envs/LlamaIndexRAGLinux/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n"
     ]
    }
   ],
   "source": [
    "import logging\n",
    "import sys\n",
    "\n",
    "logging.basicConfig(stream=sys.stdout, level=logging.INFO)\n",
    "logging.getLogger().addHandler(logging.StreamHandler(stream=sys.stdout))\n",
    "\n",
    "from llama_index.core import VectorStoreIndex, SimpleDirectoryReader, ServiceContext"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 2. Load the Word document(s)\n",
    "\n",
    "Note: A fictitious story about Thundertooth a dinosaur who has travelled to the future. Thanks ChatGPT!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "documents = SimpleDirectoryReader(\"./Data/\").load_data()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 3. Instantiate the model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "ggml_init_cublas: GGML_CUDA_FORCE_MMQ:   no\n",
      "ggml_init_cublas: CUDA_USE_TENSOR_CORES: yes\n",
      "ggml_init_cublas: found 1 CUDA devices:\n",
      "  Device 0: NVIDIA GeForce RTX 3090, compute capability 8.6, VMM: yes\n",
      "llama_model_loader: loaded meta data with 22 key-value pairs and 363 tensors from ./Models/orca-2-13b.Q5_K_S.gguf (version GGUF V3 (latest))\n",
      "llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.\n",
      "llama_model_loader: - kv   0:                       general.architecture str              = llama\n",
      "llama_model_loader: - kv   1:                               general.name str              = LLaMA v2\n",
      "llama_model_loader: - kv   2:                       llama.context_length u32              = 4096\n",
      "llama_model_loader: - kv   3:                     llama.embedding_length u32              = 5120\n",
      "llama_model_loader: - kv   4:                          llama.block_count u32              = 40\n",
      "llama_model_loader: - kv   5:                  llama.feed_forward_length u32              = 13824\n",
      "llama_model_loader: - kv   6:                 llama.rope.dimension_count u32              = 128\n",
      "llama_model_loader: - kv   7:                 llama.attention.head_count u32              = 40\n",
      "llama_model_loader: - kv   8:              llama.attention.head_count_kv u32              = 40\n",
      "llama_model_loader: - kv   9:     llama.attention.layer_norm_rms_epsilon f32              = 0.000010\n",
      "llama_model_loader: - kv  10:                       llama.rope.freq_base f32              = 10000.000000\n",
      "llama_model_loader: - kv  11:                          general.file_type u32              = 16\n",
      "llama_model_loader: - kv  12:                       tokenizer.ggml.model str              = llama\n",
      "llama_model_loader: - kv  13:                      tokenizer.ggml.tokens arr[str,32003]   = [\"<unk>\", \"<s>\", \"</s>\", \"<0x00>\", \"<...\n",
      "llama_model_loader: - kv  14:                      tokenizer.ggml.scores arr[f32,32003]   = [0.000000, 0.000000, 0.000000, 0.0000...\n",
      "llama_model_loader: - kv  15:                  tokenizer.ggml.token_type arr[i32,32003]   = [2, 3, 3, 6, 6, 6, 6, 6, 6, 6, 6, 6, ...\n",
      "llama_model_loader: - kv  16:                tokenizer.ggml.bos_token_id u32              = 1\n",
      "llama_model_loader: - kv  17:                tokenizer.ggml.eos_token_id u32              = 2\n",
      "llama_model_loader: - kv  18:            tokenizer.ggml.padding_token_id u32              = 0\n",
      "llama_model_loader: - kv  19:               tokenizer.ggml.add_bos_token bool             = true\n",
      "llama_model_loader: - kv  20:               tokenizer.ggml.add_eos_token bool             = false\n",
      "llama_model_loader: - kv  21:               general.quantization_version u32              = 2\n",
      "llama_model_loader: - type  f32:   81 tensors\n",
      "llama_model_loader: - type q5_K:  281 tensors\n",
      "llama_model_loader: - type q6_K:    1 tensors\n",
      "llm_load_vocab: special tokens definition check successful ( 262/32003 ).\n",
      "llm_load_print_meta: format           = GGUF V3 (latest)\n",
      "llm_load_print_meta: arch             = llama\n",
      "llm_load_print_meta: vocab type       = SPM\n",
      "llm_load_print_meta: n_vocab          = 32003\n",
      "llm_load_print_meta: n_merges         = 0\n",
      "llm_load_print_meta: n_ctx_train      = 4096\n",
      "llm_load_print_meta: n_embd           = 5120\n",
      "llm_load_print_meta: n_head           = 40\n",
      "llm_load_print_meta: n_head_kv        = 40\n",
      "llm_load_print_meta: n_layer          = 40\n",
      "llm_load_print_meta: n_rot            = 128\n",
      "llm_load_print_meta: n_embd_head_k    = 128\n",
      "llm_load_print_meta: n_embd_head_v    = 128\n",
      "llm_load_print_meta: n_gqa            = 1\n",
      "llm_load_print_meta: n_embd_k_gqa     = 5120\n",
      "llm_load_print_meta: n_embd_v_gqa     = 5120\n",
      "llm_load_print_meta: f_norm_eps       = 0.0e+00\n",
      "llm_load_print_meta: f_norm_rms_eps   = 1.0e-05\n",
      "llm_load_print_meta: f_clamp_kqv      = 0.0e+00\n",
      "llm_load_print_meta: f_max_alibi_bias = 0.0e+00\n",
      "llm_load_print_meta: n_ff             = 13824\n",
      "llm_load_print_meta: n_expert         = 0\n",
      "llm_load_print_meta: n_expert_used    = 0\n",
      "llm_load_print_meta: rope scaling     = linear\n",
      "llm_load_print_meta: freq_base_train  = 10000.0\n",
      "llm_load_print_meta: freq_scale_train = 1\n",
      "llm_load_print_meta: n_yarn_orig_ctx  = 4096\n",
      "llm_load_print_meta: rope_finetuned   = unknown\n",
      "llm_load_print_meta: model type       = 13B\n",
      "llm_load_print_meta: model ftype      = Q5_K - Small\n",
      "llm_load_print_meta: model params     = 13.02 B\n",
      "llm_load_print_meta: model size       = 8.36 GiB (5.51 BPW) \n",
      "llm_load_print_meta: general.name     = LLaMA v2\n",
      "llm_load_print_meta: BOS token        = 1 '<s>'\n",
      "llm_load_print_meta: EOS token        = 2 '</s>'\n",
      "llm_load_print_meta: UNK token        = 0 '<unk>'\n",
      "llm_load_print_meta: PAD token        = 0 '<unk>'\n",
      "llm_load_print_meta: LF token         = 13 '<0x0A>'\n",
      "llm_load_tensors: ggml ctx size =    0.28 MiB\n",
      "llm_load_tensors: offloading 40 repeating layers to GPU\n",
      "llm_load_tensors: offloading non-repeating layers to GPU\n",
      "llm_load_tensors: offloaded 41/41 layers to GPU\n",
      "llm_load_tensors:        CPU buffer size =   107.43 MiB\n",
      "llm_load_tensors:      CUDA0 buffer size =  8448.52 MiB\n",
      "....................................................................................................\n",
      "llama_new_context_with_model: n_ctx      = 4096\n",
      "llama_new_context_with_model: freq_base  = 10000.0\n",
      "llama_new_context_with_model: freq_scale = 1\n",
      "llama_kv_cache_init:      CUDA0 KV buffer size =  3200.00 MiB\n",
      "llama_new_context_with_model: KV self size  = 3200.00 MiB, K (f16): 1600.00 MiB, V (f16): 1600.00 MiB\n",
      "llama_new_context_with_model:  CUDA_Host input buffer size   =    19.04 MiB\n",
      "llama_new_context_with_model:      CUDA0 compute buffer size =   368.02 MiB\n",
      "llama_new_context_with_model:  CUDA_Host compute buffer size =    10.00 MiB\n",
      "llama_new_context_with_model: graph splits (measure): 3\n",
      "AVX = 1 | AVX_VNNI = 0 | AVX2 = 1 | AVX512 = 0 | AVX512_VBMI = 0 | AVX512_VNNI = 0 | FMA = 1 | NEON = 0 | ARM_FMA = 0 | F16C = 1 | FP16_VA = 0 | WASM_SIMD = 0 | BLAS = 1 | SSE3 = 1 | SSSE3 = 1 | VSX = 0 | MATMUL_INT8 = 0 | \n",
      "Model metadata: {'tokenizer.ggml.add_eos_token': 'false', 'tokenizer.ggml.padding_token_id': '0', 'tokenizer.ggml.eos_token_id': '2', 'general.architecture': 'llama', 'llama.rope.freq_base': '10000.000000', 'llama.context_length': '4096', 'general.name': 'LLaMA v2', 'tokenizer.ggml.add_bos_token': 'true', 'llama.embedding_length': '5120', 'llama.feed_forward_length': '13824', 'llama.attention.layer_norm_rms_epsilon': '0.000010', 'llama.rope.dimension_count': '128', 'tokenizer.ggml.bos_token_id': '1', 'llama.attention.head_count': '40', 'llama.block_count': '40', 'llama.attention.head_count_kv': '40', 'general.quantization_version': '2', 'tokenizer.ggml.model': 'llama', 'general.file_type': '16'}\n"
     ]
    }
   ],
   "source": [
    "import torch\n",
    "\n",
    "from llama_index.llms.llama_cpp import LlamaCPP\n",
    "from llama_index.llms.llama_cpp.llama_utils import messages_to_prompt, completion_to_prompt\n",
    "llm = LlamaCPP(\n",
    "    model_url=None, # We'll load locally.\n",
    "    model_path='./Models/orca-2-13b.Q5_K_S.gguf',\n",
    "    temperature=0.1,\n",
    "    max_new_tokens=1024, # Increasing to support longer responses\n",
    "    context_window=4096, # Orca 2 4096 context window\n",
    "    generate_kwargs={},\n",
    "    # set to at least 1 to use GPU\n",
    "    model_kwargs={\"n_gpu_layers\": 41}, # 41 was all that was required and that fit within the RTX 3090's 24GB\n",
    "    messages_to_prompt=messages_to_prompt,\n",
    "    completion_to_prompt=completion_to_prompt,\n",
    "    verbose=True\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 4. Checkpoint\n",
    "\n",
    "Are you running on GPU? The above output should include near the top something like:\n",
    "> ggml_init_cublas: found 1 CUDA devices:\n",
    "\n",
    "And in the full text near the bottom should be:\n",
    "> llm_load_tensors: using CUDA for GPU acceleration"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 5. Embeddings\n",
    "\n",
    "Convert your source document text into embeddings.\n",
    "\n",
    "The embedding model is from huggingface, this one performs well.\n",
    "\n",
    "> https://huggingface.co/thenlper/gte-large\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.embeddings.huggingface import HuggingFaceEmbedding\n",
    "\n",
    "embed_model = HuggingFaceEmbedding(model_name=\"thenlper/gte-large\", cache_folder=None)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 6. Prompt Template\n",
    "\n",
    "Prompt template for Orca 2 is the ChatML template:\n",
    "\n",
    "<|im_start|>system<br>\n",
    "{system_message}<|im_end|><br>\n",
    "<|im_start|>user<br>\n",
    "{prompt}<|im_end|><br>\n",
    "<|im_start|>assistant"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Produces a prompt for the model\n",
    "def chatml_prompt(systemmessage, promptmessage):\n",
    "    return f\"<|im_start|>system\\n{systemmessage}<|im_end|>\\n<|im_start|>user\\n{promptmessage}<|im_end|>\\n<|im_start|>assistant\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 7. Service Context\n",
    "\n",
    "For chunking the document into tokens using the embedding model and our LLM"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core import Settings\n",
    "\n",
    "Settings.llm = llm\n",
    "Settings.embed_model = embed_model\n",
    "Settings.chunk_size=256 # Number of tokens in each chunk"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 8. Index documents"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "index = VectorStoreIndex.from_documents(documents)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 9. Query Engine\n",
    "\n",
    "Create a query engine, specifying how many citations we want to get back from the searched text (in this case 3).\n",
    "\n",
    "The DB_DOC_ID_KEY is used to get back the filename of the original document"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [],
   "source": [
    "from llama_index.core.query_engine import CitationQueryEngine\n",
    "query_engine = CitationQueryEngine.from_args(\n",
    "    index,\n",
    "    similarity_top_k=3,\n",
    "    # here we can control how granular citation sources are, the default is 512\n",
    "    citation_chunk_size=256,\n",
    ")\n",
    "\n",
    "# For citations we get the document info\n",
    "DB_DOC_ID_KEY = \"db_document_id\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 10. Prompt and Response function\n",
    "\n",
    "Pass in a question, get a response back.\n",
    "\n",
    "IMPORTANT: The prompt is set here, adjust it to match what you want the LLM to act like and do."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "def RunQuestion(questionText):\n",
    "    systemmessage = \"You are a story teller who likes to elaborate. Answer questions in a positive, helpful and interesting way. If the answer is not in the following context return ONLY 'Sorry, I don't know the answer to that'.\"\n",
    "\n",
    "    queryQuestion = chatml_prompt(systemmessage, questionText)\n",
    "\n",
    "    response = query_engine.query(queryQuestion)\n",
    "\n",
    "    return response"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 11. Questions to test with"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [],
   "source": [
    "TestQuestions = [\n",
    "    \"Summarise the story for me\",\n",
    "    \"Who was the main protagonist?\",\n",
    "    \"Did they have any children? If so, what were their names?\",\n",
    "    \"Did anything eventful happen?\",\n",
    "]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 12. Run Questions through model (this can take a while) and see citations\n",
    "\n",
    "Runs each test question, saves it to a dictionary for output in the last step.\n",
    "\n",
    "Note: Citations are the source documents used and the text the response is based on. This is important for RAG so you can reference these documents for the user, and to ensure it's utilising the right documents."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "1/4: Summarise the story for me\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =     265.66 ms\n",
      "llama_print_timings:      sample time =      23.56 ms /    90 runs   (    0.26 ms per token,  3820.36 tokens per second)\n",
      "llama_print_timings: prompt eval time =     734.69 ms /  1267 tokens (    0.58 ms per token,  1724.53 tokens per second)\n",
      "llama_print_timings:        eval time =    1661.86 ms /    89 runs   (   18.67 ms per token,    53.55 tokens per second)\n",
      "llama_print_timings:       total time =    2577.07 ms /  1356 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/4: |Thundertooth Part 1.docx| Source 1:\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n",
      "\n",
      "\n",
      "Realizing the dinosaur's predicament, Mayor Grace extended an invitation. \"You are welcome in our city, Thundertooth. We can find a way to provide for you without causing harm to anyone. Let us work together to find a solution.\"\n",
      "\n",
      "\n",
      "\n",
      "Grateful for the mayor's hospitality, Thundertooth followed her through the city. Together, they explored the futuristic marketplaces and innovative food labs, eventually discovering a sustainable solution that satisfied the dinosaur's hunger without compromising the well-being of the city's inhabitants.\n",
      "\n",
      "2/4: |Thundertooth Part 1.docx| Source 2:\n",
      "As the news of Thundertooth's arrival spread, the city embraced the talking dinosaur as a symbol of unity between the past and the future. Thundertooth found a new home in the city's park, where holographic flowers bloomed, and the citizens marveled at the beauty of coexistence across time. And so, in this extraordinary city of flying cars and advanced technology, Thundertooth became a beloved figure, a living bridge between eras, teaching the people that understanding and cooperation could overcome even the greatest challenges.\n",
      "\n",
      "3/4: |Thundertooth Part 1.docx| Source 3:\n",
      "Thundertooth\n",
      "\n",
      "\n",
      "\n",
      "Once upon a time, in a prehistoric land filled with dense forests and roaring rivers, there lived a dinosaur named Thundertooth. Thundertooth was no ordinary dinosaur; he possessed the rare ability to speak, a talent that set him apart from his ancient companions. One fateful day, as Thundertooth was basking in the warmth of the sun, a mysterious portal opened before him, and he found himself hurtling through time and space.\n",
      "\n",
      "\n",
      "\n",
      "As the dazzling vortex subsided, Thundertooth opened his eyes to a world unlike anything he had ever seen. The air was filled with the hum of engines, and towering structures reached towards the sky. Thundertooth's surroundings were a blend of metal and glass, and he quickly realized that he had been transported to a future era.\n",
      "\n",
      "4/4: |Thundertooth Part 3.docx| Source 4:\n",
      "The mayor, recognizing Thundertooth's intelligence and resourcefulness, approached him. \"Thundertooth, we need a plan to divert or neutralize the meteor. Our technology can only do so much, but with your unique abilities, perhaps we can find a solution.\"\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "\n",
      "2/4: Who was the main protagonist?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =     265.66 ms\n",
      "llama_print_timings:      sample time =      25.32 ms /    94 runs   (    0.27 ms per token,  3712.48 tokens per second)\n",
      "llama_print_timings: prompt eval time =     709.89 ms /  1119 tokens (    0.63 ms per token,  1576.29 tokens per second)\n",
      "llama_print_timings:        eval time =    1750.00 ms /    93 runs   (   18.82 ms per token,    53.14 tokens per second)\n",
      "llama_print_timings:       total time =    2659.78 ms /  1212 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/4: |Thundertooth Part 3.docx| Source 1:\n",
      "The mayor, recognizing Thundertooth's intelligence and resourcefulness, approached him. \"Thundertooth, we need a plan to divert or neutralize the meteor. Our technology can only do so much, but with your unique abilities, perhaps we can find a solution.\"\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n",
      "2/4: |Thundertooth Part 3.docx| Source 2:\n",
      "Thundertooth, ever the protector of his newfound home, wasted no time. With a determined gleam in his eyes, he gathered his family and hurried to the city's command center, where Mayor Grace and the leading scientists were coordinating the evacuation efforts.\n",
      "\n",
      "\n",
      "\n",
      "The mayor, recognizing Thundertooth's intelligence and resourcefulness, approached him. \"Thundertooth, we need a plan to divert or neutralize the meteor. Our technology can only do so much, but with your unique abilities, perhaps we can find a solution.\"\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "3/4: |Thundertooth Part 1.docx| Source 3:\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n",
      "\n",
      "\n",
      "Realizing the dinosaur's predicament, Mayor Grace extended an invitation. \"You are welcome in our city, Thundertooth. We can find a way to provide for you without causing harm to anyone. Let us work together to find a solution.\"\n",
      "\n",
      "\n",
      "\n",
      "Grateful for the mayor's hospitality, Thundertooth followed her through the city. Together, they explored the futuristic marketplaces and innovative food labs, eventually discovering a sustainable solution that satisfied the dinosaur's hunger without compromising the well-being of the city's inhabitants.\n",
      "\n",
      "4/4: |Thundertooth Part 1.docx| Source 4:\n",
      "As the news of Thundertooth's arrival spread, the city embraced the talking dinosaur as a symbol of unity between the past and the future. Thundertooth found a new home in the city's park, where holographic flowers bloomed, and the citizens marveled at the beauty of coexistence across time. And so, in this extraordinary city of flying cars and advanced technology, Thundertooth became a beloved figure, a living bridge between eras, teaching the people that understanding and cooperation could overcome even the greatest challenges.\n",
      "\n",
      "\n",
      "3/4: Did they have any children? If so, what were their names?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =     265.66 ms\n",
      "llama_print_timings:      sample time =      24.79 ms /    92 runs   (    0.27 ms per token,  3711.02 tokens per second)\n",
      "llama_print_timings: prompt eval time =     574.48 ms /   975 tokens (    0.59 ms per token,  1697.19 tokens per second)\n",
      "llama_print_timings:        eval time =    1700.67 ms /    91 runs   (   18.69 ms per token,    53.51 tokens per second)\n",
      "llama_print_timings:       total time =    2465.87 ms /  1066 tokens\n",
      "Llama.generate: prefix-match hit\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/4: |Thundertooth Part 1.docx| Source 1:\n",
      "Thundertooth\n",
      "\n",
      "\n",
      "\n",
      "Once upon a time, in a prehistoric land filled with dense forests and roaring rivers, there lived a dinosaur named Thundertooth. Thundertooth was no ordinary dinosaur; he possessed the rare ability to speak, a talent that set him apart from his ancient companions. One fateful day, as Thundertooth was basking in the warmth of the sun, a mysterious portal opened before him, and he found himself hurtling through time and space.\n",
      "\n",
      "\n",
      "\n",
      "As the dazzling vortex subsided, Thundertooth opened his eyes to a world unlike anything he had ever seen. The air was filled with the hum of engines, and towering structures reached towards the sky. Thundertooth's surroundings were a blend of metal and glass, and he quickly realized that he had been transported to a future era.\n",
      "\n",
      "2/4: |Thundertooth Part 2.docx| Source 2:\n",
      "As the years passed, Thundertooth's life took a heartwarming turn. He met a kind and intelligent dinosaur named Seraphina, and together they started a family. Thundertooth and Seraphina were blessed with four children, each with unique characteristics that mirrored the diversity of their modern world.\n",
      "\n",
      "\n",
      "\n",
      "Lumina: The eldest of Thundertooth's children, Lumina inherited her mother's intelligence and her father's sense of wonder. With sparkling scales that emitted a soft glow, Lumina had the ability to generate light at will. She became fascinated with technology, often spending hours tinkering with gadgets and inventing new ways to enhance the widgets produced in the family's factory.\n",
      "\n",
      "\n",
      "\n",
      "Echo: The second-born, Echo, had a gift for mimicry. He could perfectly replicate any sound or voice he heard, providing entertainment to the entire city. His playful nature and ability to bring joy to those around him made him a favorite among the neighborhood children.\n",
      "\n",
      "3/4: |Thundertooth Part 1.docx| Source 3:\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n",
      "\n",
      "\n",
      "Realizing the dinosaur's predicament, Mayor Grace extended an invitation. \"You are welcome in our city, Thundertooth. We can find a way to provide for you without causing harm to anyone. Let us work together to find a solution.\"\n",
      "\n",
      "\n",
      "\n",
      "Grateful for the mayor's hospitality, Thundertooth followed her through the city. Together, they explored the futuristic marketplaces and innovative food labs, eventually discovering a sustainable solution that satisfied the dinosaur's hunger without compromising the well-being of the city's inhabitants.\n",
      "\n",
      "4/4: |Thundertooth Part 1.docx| Source 4:\n",
      "As the news of Thundertooth's arrival spread, the city embraced the talking dinosaur as a symbol of unity between the past and the future. Thundertooth found a new home in the city's park, where holographic flowers bloomed, and the citizens marveled at the beauty of coexistence across time. And so, in this extraordinary city of flying cars and advanced technology, Thundertooth became a beloved figure, a living bridge between eras, teaching the people that understanding and cooperation could overcome even the greatest challenges.\n",
      "\n",
      "\n",
      "4/4: Did anything eventful happen?\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "\n",
      "llama_print_timings:        load time =     265.66 ms\n",
      "llama_print_timings:      sample time =      24.96 ms /    93 runs   (    0.27 ms per token,  3725.22 tokens per second)\n",
      "llama_print_timings: prompt eval time =     584.22 ms /  1024 tokens (    0.57 ms per token,  1752.76 tokens per second)\n",
      "llama_print_timings:        eval time =    1738.83 ms /    93 runs   (   18.70 ms per token,    53.48 tokens per second)\n",
      "llama_print_timings:       total time =    2514.64 ms /  1117 tokens\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/4: |Thundertooth Part 1.docx| Source 1:\n",
      "\"Hello there, majestic creature. What brings you to our time?\" Mayor Grace inquired, her voice calm and reassuring.\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth, though initially startled, found comfort in the mayor's soothing tone. In broken sentences, he explained his journey through time, the strange portal, and his hunger dilemma. Mayor Grace listened intently, her eyes widening with amazement at the tale of the prehistoric dinosaur navigating the future.\n",
      "\n",
      "\n",
      "\n",
      "Realizing the dinosaur's predicament, Mayor Grace extended an invitation. \"You are welcome in our city, Thundertooth. We can find a way to provide for you without causing harm to anyone. Let us work together to find a solution.\"\n",
      "\n",
      "\n",
      "\n",
      "Grateful for the mayor's hospitality, Thundertooth followed her through the city. Together, they explored the futuristic marketplaces and innovative food labs, eventually discovering a sustainable solution that satisfied the dinosaur's hunger without compromising the well-being of the city's inhabitants.\n",
      "\n",
      "2/4: |Thundertooth Part 1.docx| Source 2:\n",
      "As the news of Thundertooth's arrival spread, the city embraced the talking dinosaur as a symbol of unity between the past and the future. Thundertooth found a new home in the city's park, where holographic flowers bloomed, and the citizens marveled at the beauty of coexistence across time. And so, in this extraordinary city of flying cars and advanced technology, Thundertooth became a beloved figure, a living bridge between eras, teaching the people that understanding and cooperation could overcome even the greatest challenges.\n",
      "\n",
      "3/4: |Thundertooth Part 1.docx| Source 3:\n",
      "Thundertooth\n",
      "\n",
      "\n",
      "\n",
      "Once upon a time, in a prehistoric land filled with dense forests and roaring rivers, there lived a dinosaur named Thundertooth. Thundertooth was no ordinary dinosaur; he possessed the rare ability to speak, a talent that set him apart from his ancient companions. One fateful day, as Thundertooth was basking in the warmth of the sun, a mysterious portal opened before him, and he found himself hurtling through time and space.\n",
      "\n",
      "\n",
      "\n",
      "As the dazzling vortex subsided, Thundertooth opened his eyes to a world unlike anything he had ever seen. The air was filled with the hum of engines, and towering structures reached towards the sky. Thundertooth's surroundings were a blend of metal and glass, and he quickly realized that he had been transported to a future era.\n",
      "\n",
      "4/4: |Thundertooth Part 3.docx| Source 4:\n",
      "The mayor, recognizing Thundertooth's intelligence and resourcefulness, approached him. \"Thundertooth, we need a plan to divert or neutralize the meteor. Our technology can only do so much, but with your unique abilities, perhaps we can find a solution.\"\n",
      "\n",
      "\n",
      "\n",
      "Thundertooth nodded, understanding the gravity of the situation. He gathered Lumina, Echo, Sapphire, and Ignis, explaining the urgency and the role each of them would play in the impending crisis.\n",
      "\n",
      "\n",
      "\n",
      "1. **Lumina**: Utilizing her deep understanding of technology, Lumina would enhance the city's energy systems to generate a powerful force field, providing a protective barrier against the meteor's impact.\n",
      "\n",
      "\n",
      "\n",
      "2. **Echo**: With his extraordinary mimicry abilities, Echo would amplify the emergency signals, ensuring that every citizen received timely warnings and instructions for evacuation.\n",
      "\n",
      "\n",
      "\n",
      "3. **Sapphire**: Harnessing her calming and healing powers, Sapphire would assist in calming the panicked masses, ensuring an orderly and efficient evacuation.\n",
      "\n",
      "\n",
      "\n",
      "4. **Ignis**: Drawing upon his fiery talents, Ignis would create controlled bursts of heat, attempting to alter the meteor's trajectory and reduce its destructive force.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "qa_pairs = []\n",
    "\n",
    "for index, question in enumerate(TestQuestions, start=1):\n",
    "    question = question.strip() # Clean up\n",
    "\n",
    "    print(f\"\\n{index}/{len(TestQuestions)}: {question}\")\n",
    "\n",
    "    response = RunQuestion(question) # Query and get  response\n",
    "\n",
    "    qa_pairs.append((question.strip(), str(response).strip())) # Add to our output array\n",
    "\n",
    "    # Displays the citations\n",
    "    for index, node in enumerate(response.source_nodes, start=1):\n",
    "        print(f\"{index}/{len(response.source_nodes)}: |{node.node.metadata['file_name']}| {node.node.get_text()}\")\n",
    "\n",
    "    # Uncomment the following line if you want to test just the first question\n",
    "    # break "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### 13. Output responses"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "1/4 Summarise the story for me\n",
      "\n",
      "Possible summary:\n",
      "\n",
      "The story is about Thundertooth, a talking dinosaur who travels through time and ends up in a futuristic city. He meets Mayor Grace, who helps him find food and a home in the city's park. Thundertooth becomes a symbol of unity between the past and the future, and he joins forces with other talking animals to save the city from a meteor.\n",
      "\n",
      "--------\n",
      "\n",
      "2/4 Who was the main protagonist?\n",
      "\n",
      "The main protagonist of the story was Thundertooth, a prehistoric dinosaur who traveled through time and became a hero in the futuristic city. He was a kind and intelligent creature who used his abilities to help the people and the environment. He was welcomed by Mayor Grace and his friends Lumina, Echo, Sapphire, and Ignis, who also played important roles in the story.\n",
      "\n",
      "--------\n",
      "\n",
      "3/4 Did they have any children? If so, what were their names?\n",
      "\n",
      "Yes, Thundertooth and Seraphina had four children. Their names were Lumina, Echo, Rumble, and Glimmer. Lumina had the ability to generate light with her scales, Echo could mimic any sound he heard, Rumble was a strong and brave dinosaur who loved to explore, and Glimmer was a graceful and elegant creature who could create beautiful illusions.\n",
      "\n",
      "--------\n",
      "\n",
      "4/4 Did anything eventful happen?\n",
      "\n",
      "Yes, something eventful happened. A prehistoric dinosaur named Thundertooth was transported to the future by a mysterious portal and met a mayor who welcomed him to the city. He also found a way to satisfy his hunger without harming anyone and became a symbol of unity between the past and the future. Additionally, he helped his friends save the city from a meteor threat by using their unique abilities.\n",
      "\n",
      "--------\n",
      "\n"
     ]
    },
    {
     "ename": "",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31mThe Kernel crashed while executing code in the current cell or a previous cell. \n",
      "\u001b[1;31mPlease review the code in the cell(s) to identify a possible cause of the failure. \n",
      "\u001b[1;31mClick <a href='https://aka.ms/vscodeJupyterKernelCrash'>here</a> for more info. \n",
      "\u001b[1;31mView Jupyter <a href='command:jupyter.viewOutput'>log</a> for further details."
     ]
    }
   ],
   "source": [
    "for index, (question, answer) in enumerate(qa_pairs, start=1):\n",
    "    print(f\"{index}/{len(qa_pairs)} {question}\\n\\n{answer}\\n\\n--------\\n\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "llamaindexgeneric",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.13"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}