distilGPT2_tune.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Story Generation based on a Given Prompt using DistilGpt-2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Author : Kamlesh Kumar\n",
    "### CMS : 348395\n",
    "### Class : BSCS-10C\n",
    "#### https://www.linkedin.com/in/kamlesh-kumar-389847224/"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*installing additional libraries required*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%capture\n",
    "!pip install -q kaggle\n",
    "!pip install -q transformers\n",
    "!pip install accelerate -U\n",
    "!pip install tensorboard"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Text Generation Via Transformers:\n",
    "Story Generation based on a Given Prompt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data Exploration:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Warning: Your Kaggle API key is readable by other users on this system! To fix this, you can run 'chmod 600 /home/kamlesh_kumar/.kaggle/kaggle.json'\n",
      "Downloading writing-prompts.zip to /home/kamlesh_kumar/FYP/GPT2 Fine tunning\n",
      "100%|███████████████████████████████████████▉| 369M/370M [00:38<00:00, 10.7MB/s]\n",
      "100%|████████████████████████████████████████| 370M/370M [00:38<00:00, 10.1MB/s]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "UsageError: Line magic function `%unzip` not found.\n"
     ]
    }
   ],
   "source": [
    "# Store your Kaggle api file kaggle.json in the folder where this notebook is located\n",
    "\n",
    "!mkdir ~/.kaggle\n",
    "!cp kaggle.json ~/.kaggle/\n",
    "!chmod 600 ~/.kaggle/kaggle.json\n",
    "!kaggle datasets download -d ratthachat/writing-prompts"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Archive:  writing-prompts.zip\n",
      "replace writingPrompts/README? [y]es, [n]o, [A]ll, [N]one, [r]ename: ^C\n"
     ]
    }
   ],
   "source": [
    "# Unzip the data writting prompts datset. \n",
    "!unzip writing-prompts.zip"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " You 've finally managed to discover the secret to immortality . Suddenly , Death appears before you , hands you a business card , and says , `` When you realize living forever sucks , call this number , I 've got a job offer for you . ''\n",
      " The moon is actually a giant egg , and it has just started to hatch .\n",
      " You find a rip in time walking through the alleys . You enter it to find yourself on a metal table with surgical instruments on a chair next to you .\n",
      " For years in your youth the same imaginary character appears in your dreams , you are good friends . Years later , when adult , you meet her in real life , she clearly recognises you and tries to avoid you , and you want answers .\n",
      " You glance at your watch 10:34 am , roughly 10 seconds later your plane explodes over the Pacific Ocean . Your eyes open as you jolt awake . The familiar hum of the planes engine remains . Checking your watch it is 9:35\n",
      " Through Iron And Flame\n",
      " You live in a world where there has never been sickness , and you are the first to have ever experienced being sick .\n"
     ]
    }
   ],
   "source": [
    "# Set this variable to the path where you have stored the dataset\n",
    "\n",
    "Dataset_path = \"/home/kamlesh_kumar/FYP/GPT2Finetunning/writingPrompts/\"\n",
    "# !head {Dataset_path}train.wp_source\n",
    "!head -n 7 {Dataset_path}train.wp_source"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "So many times have I walked on ruins , the remainings of places that I loved and got used to.. At first I was scared , each time I could feel my city , my current generation collapse , break into the black hole that thrives within it , I could feel humanity , the way I 'm able to feel my body.. After a few hundred years , the pattern became obvious , no longer the war and damage that would devastate me over and over again in the far past was effecting me so dominantly . <newline> It 's funny , but I felt as if after gaining what I desired so long , what I have lived for my entire life , only then , when I achieved immortality I started truly aging . <newline> <newline> 5 world wars have passed , and now they feel like a simple sickeness that would pass by every so often , I could no longer evaluate the individual human as a being of its own , the importance of mortals is merely the same as the importance of my skin cells ; They are a part of a mechanism so much more advanced , a mechanism that is so dear to my fallen heart a mechanism that I have seen fall and rise so many times , a mechanism that when lost all of which it had , had me loosing my will to live , for the first time in all of my thousands years of existence . <newline> <newline> Acceptance , something so important . a skill that has proved itself worthy dozens of times , an ability that looks so easy to achieve , a gift , that I was n't able to aquire in all my years , until now . When the ashes on the ground flew into the now empty air upon humanity 's fall , I felt as if all of it 's weight was crushing me . Ignorance took over and I searched years for a hope , a sign of the very same patterns that I used to watch reappear every hundred years , the very core of my will to exist that was now no more that I so strongly wish was . <newline> <newline> If you have ever wondered if silence can drive people crazy , it can.. <newline> I ca n't feel my legs , I have walked for days , just to hear the sound of gravel , crushed bones , crushed buildings and crushed civilizations under my steps to keep my sanity.. until I remembered , the day in my far past . The day of my rebirth , I took out of my pocket a small plastic box , with nine buttons and a small glass window . I could n't believe this was our past , I could n't believe how far we have been able to progress and yet , be destroyed by our own violence . <newline> I slowly dialed the number I was given , exactly 1729 years ago . <newline> <newline> I dropped a tear , a tear that was too slow to hit the ground as I got sucked into the darkness that emerged around me . <newline> <newline> A chill went through my spine as I saw my destiny rise above me , I could see the white teeth under the dark cloack ... <newline> <newline> `` You have finally arrived '' He projected into my mind , with the most chilling cold and unhuman voice . <newline> <newline> `` I 'm ready to obey '' I answered . I knew who was sitting infront of me , and it was time for me to obey him , after all these years of playing god , even I came to it . <newline> <newline> Funny is n't it ? Even by achieving immortality , death , is inescapable .\n"
     ]
    }
   ],
   "source": [
    "# printing 4 lines of the target file i.e. the story\n",
    "!head -n 1 {Dataset_path}train.wp_target"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [],
   "source": [
    "import subprocess\n",
    "\n",
    "# For removing the extra brackets from the source file. \n",
    "def remove_brackets_from_file(file_path):\n",
    "    sed_command = r\"sed -i 's/\\[.*\\]//g' {}\"\n",
    "    subprocess.call(sed_command.format(file_path), shell=True)\n",
    "\n",
    "# For replacing some extra punctuations and spaces. \n",
    "def clean_punctuation(text):\n",
    "    for p in '!,.:;?':\n",
    "        text = text.replace(' ' + p, p)\n",
    "    replacements = [\n",
    "        (\" n't\", \"n't\"), (\" 's\", \"'s\"), (\" 're\", \"'re\"), (\" 've\", \"'ve\"),\n",
    "        (\" 'll\", \"'ll\"), (\" 'am\", \"'am\"), (\" 'm\", \"'m\"), (\" ' m\", \"'m\"),\n",
    "        (\" 've\", \"'ve\"), (\" ' s\", \"'s\"), (\"<newline>\", \"\\n\")\n",
    "    ]\n",
    "    for old, new in replacements:\n",
    "        text = text.replace(old, new)\n",
    "    return text\n",
    "\n",
    "def process_file(file_path):\n",
    "    with open(file_path, 'r') as file:\n",
    "        lines = file.readlines()\n",
    "    cleaned_lines = [clean_punctuation(line) for line in lines]\n",
    "    with open(file_path, 'w') as file:\n",
    "        file.writelines(cleaned_lines)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Custom Dataset class for loading and encoding the dataset on fly, while training\n",
    "\n",
    "from torch.utils.data import Dataset\n",
    "\n",
    "class CustomDataset(Dataset):\n",
    "    # def __init__(self, prompts_file, stories_file, tokenizer):\n",
    "    #     self.prompts = open(prompts_file, 'r').readlines()\n",
    "    #     self.stories = open(stories_file, 'r').readlines()\n",
    "    #     self.tokenizer = tokenizer\n",
    "\n",
    "    def __init__(self, prompts_file, stories_file, tokenizer, max_samples=30000):\n",
    "        with open(prompts_file, 'r') as file:\n",
    "            self.prompts = [next(file).strip() for _ in range(max_samples)]\n",
    "\n",
    "        with open(stories_file, 'r') as file:\n",
    "            self.stories = [next(file).strip() for _ in range(max_samples)]\n",
    "\n",
    "        # Ensuring that both lists have the same length\n",
    "        min_length = min(len(self.prompts), len(self.stories))\n",
    "        self.prompts = self.prompts[:min_length]\n",
    "        self.stories = self.stories[:min_length]\n",
    "\n",
    "        self.tokenizer = tokenizer\n",
    "\n",
    "    def __len__(self):\n",
    "        return len(self.prompts)\n",
    "\n",
    "    def __getitem__(self, idx):\n",
    "        prompt = self.prompts[idx].strip()\n",
    "        story = self.stories[idx].strip()\n",
    "\n",
    "        # Encode the prompt and story using the tokenizer\n",
    "        combined_text = f\"\"\"prompt :\n",
    "{prompt}\n",
    "story: \n",
    "{story}\n",
    "\"\"\"\n",
    "        combined_text = clean_punctuation(combined_text)\n",
    "        encoding = self.tokenizer(combined_text, return_tensors=\"pt\", max_length=512, truncation=True)\n",
    "\n",
    "        return {\n",
    "            \"input_ids\": encoding[\"input_ids\"].squeeze(),\n",
    "            \"attention_mask\": encoding[\"attention_mask\"].squeeze(),\n",
    "        }"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Additional code for loading tokenized data directly into memory before start of training.\n",
    "\n",
    "# class CustomDataset(Dataset):\n",
    "#     def __init__(self, encoded_data):\n",
    "#         self.encoded_data = encoded_data\n",
    "\n",
    "#     def __len__(self):\n",
    "#         return len(self.encoded_data)\n",
    "\n",
    "#     def __getitem__(self, idx):\n",
    "#         return self.encoded_data[idx]\n",
    "\n",
    "# def encode_dataset(prompts_file, stories_file, tokenizer, max_length):\n",
    "#     prompts = open(prompts_file, 'r').readlines()\n",
    "#     stories = open(stories_file, 'r').readlines()\n",
    "    \n",
    "#     encoded_data = []\n",
    "#     for prompt, story in zip(prompts, stories):\n",
    "#         encoding = tokenizer(prompt.strip(), story.strip(), max_length=max_length, padding=\"max_length\", truncation=True)\n",
    "#         encoded_data.append({\n",
    "#             \"input_ids\": encoding[\"input_ids\"],\n",
    "#             \"attention_mask\": encoding[\"attention_mask\"],\n",
    "#         })\n",
    "    \n",
    "#     return encoded_data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Preprocessing"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Define your list of source and target file paths to make it easier to acess.\n",
    "\n",
    "target_file_paths = [Dataset_path + \"train.wp_target\", Dataset_path + \"valid.wp_target\", Dataset_path + \"test.wp_target\"]\n",
    "\n",
    "source_file_paths = [Dataset_path + \"train.wp_source\", Dataset_path + \"valid.wp_source\", Dataset_path + \"test.wp_source\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "## Code for encoding and loading the train and val dataset into memory.\n",
    " \n",
    "# encoded_train_data = encode_dataset(source_file_paths[0], target_file_paths[0], tokenizer, max_length=512)\n",
    "# encoded_val_data = encode_dataset(source_file_paths[1], target_file_paths[1], tokenizer, max_length=512)\n",
    "\n",
    "# train_dataset = CustomDataset(encoded_train_data)\n",
    "# val_dataset = CustomDataset(encoded_val_data)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Processing complete for all files.\n"
     ]
    }
   ],
   "source": [
    "\n",
    "# remove_brackets_from_file defined above\n",
    "\n",
    "# Process each file\n",
    "for file_path in source_file_paths:\n",
    "    remove_brackets_from_file(file_path)\n",
    "    # process_file(file_path)\n",
    "\n",
    "print(\"Processing complete for all files.\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Model Selection:\n",
    "Model used is distilgpt2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "from transformers import GPT2Tokenizer, GPT2LMHeadModel\n",
    "\n",
    "model_name = \"distilgpt2\"\n",
    "tokenizer = GPT2Tokenizer.from_pretrained(model_name, pad_token='pad')\n",
    "model = GPT2LMHeadModel.from_pretrained(model_name)\n",
    "\n",
    "# tokenizer.pad_token = tokenizer.eos_token\n",
    "\n",
    "from transformers import DataCollatorForLanguageModeling\n",
    "data_collator = DataCollatorForLanguageModeling(\n",
    "    tokenizer=tokenizer, mlm=False,\n",
    ")\n",
    "tokenizer.pad_token = tokenizer.eos_token"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "GPT2LMHeadModel(\n",
       "  (transformer): GPT2Model(\n",
       "    (wte): Embedding(50257, 768)\n",
       "    (wpe): Embedding(1024, 768)\n",
       "    (drop): Dropout(p=0.1, inplace=False)\n",
       "    (h): ModuleList(\n",
       "      (0-5): 6 x GPT2Block(\n",
       "        (ln_1): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (attn): GPT2Attention(\n",
       "          (c_attn): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (attn_dropout): Dropout(p=0.1, inplace=False)\n",
       "          (resid_dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "        (ln_2): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "        (mlp): GPT2MLP(\n",
       "          (c_fc): Conv1D()\n",
       "          (c_proj): Conv1D()\n",
       "          (act): NewGELUActivation()\n",
       "          (dropout): Dropout(p=0.1, inplace=False)\n",
       "        )\n",
       "      )\n",
       "    )\n",
       "    (ln_f): LayerNorm((768,), eps=1e-05, elementwise_affine=True)\n",
       "  )\n",
       "  (lm_head): Linear(in_features=768, out_features=50257, bias=False)\n",
       ")"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# moving model to GPU\n",
    "\n",
    "import torch\n",
    "device = torch.device(\"cuda\" if torch.cuda.is_available() else \"cpu\")\n",
    "model.to(device)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Fine tunning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘experiments’: File exists\n"
     ]
    }
   ],
   "source": [
    "!mkdir experiments"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Creating the data loaders for training and validation dataset.\n",
    "# Using 30000 samples for training and 10000 samples for validation\n",
    "\n",
    "train_dataset = CustomDataset(source_file_paths[0], target_file_paths[0], tokenizer, max_samples = 30000)\n",
    "val_dataset = CustomDataset(source_file_paths[1], target_file_paths[1], tokenizer, max_samples = 10000)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "\n",
      "\n",
      "\n",
      "prompt:\n",
      "Through Iron And Flame\n",
      "story: \n",
      "The mountain stood still and large beneath the Warrior. It had not trembled in the days since the people chose him as their Champion. `` Perhaps the Dragon slumbers yet, '' he thought. `` I shall wake him, soon enough. '' He strode forward, heavily, in his iron raiment, shining dully in the morning sun. The shrouded peak drew slowly nearer as the sun climbed the sky, mirroring the Warrior's toil this day as the last. The Dragon slept silently in its lair amidst the clouds. \n",
      " \n",
      " For generations the people had not seen it wake. But their stories, told by night and hearth fire, still remained hushed in reverence of its deadly wrath. Legends told of it waking and the destruction that followed. But that was before. Before the people had tamed iron and bent it to their will. Now they had weapons that could fight the myth. That could quench the fire. \n",
      " \n",
      " The Warrior stopped at the edge of the cloud bank, just as the sun had stopped there hours before. He breathed. Deliberately he pulled the heavy iron axe from his belt and gripped it between strong hands. A glow had appeared within the clouds. The mountain rumbled. \n",
      " \n",
      " `` I have come to wake you, beast, and I have come to show you the people no longer fear you. '' The ground rolled and thundered. He roared in defiance and raised his axe, charging into the fog. He half ran and climbed in muffled beats as his bloodlust for the Dragon pounded in his ears. The glow grew brighter and the ground shook violently. \n",
      " \n",
      " A gout of flame split the clouds in front of the Warrior and he saw his destination. The rock ended in a ridge just above him and the Dragons flames beat and splashed beyond. Without breaking stride, the iron Warrior crested the ridge and leapt, axe raised high, into a pit of Dragon fire.\n",
      "\n",
      "\n",
      "\n",
      "\n",
      " dict_keys(['input_ids', 'attention_mask'])\n"
     ]
    }
   ],
   "source": [
    "# testing dataset loader class.\n",
    "\n",
    "output = train_dataset.__getitem__(5)\n",
    "\n",
    "decoded_text = tokenizer.decode(output['input_ids'], skip_special_tokens=True)\n",
    "print(\"\\n\\n\\n\")\n",
    "print(decoded_text)\n",
    "\n",
    "print(\"\\n\\n\\n\", output.keys())"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "30000"
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_dataset.__len__()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Detected kernel version 5.4.0, which is below the recommended minimum of 5.5.0; this can cause the process to hang. It is recommended to upgrade the kernel to the minimum version or higher.\n"
     ]
    }
   ],
   "source": [
    "# For setting up the trainer and training arguments\n",
    "\n",
    "from transformers import Trainer, TrainingArguments\n",
    "\n",
    "#setting up trainer arguments\n",
    "training_args = TrainingArguments(\n",
    "    output_dir=\"/home/kamlesh_kumar/FYP/GPT2Finetunning/experiments\", #The output directory for saved model\n",
    "    overwrite_output_dir=True, #overwrite the content of the output directory\n",
    "    num_train_epochs=10, # number of training epochs\n",
    "    gradient_accumulation_steps=4,\n",
    "    learning_rate=2e-3,\n",
    "    lr_scheduler_type=\"cosine\",\n",
    "    save_total_limit=3,\n",
    "\n",
    "    per_device_train_batch_size=8, # batch size for training\n",
    "    per_device_eval_batch_size=8,  # batch size for evaluation\n",
    "    eval_steps = 500, # Number of update steps between two evaluations.\n",
    "    warmup_ratio=0.05,\n",
    "    # fp16=True, #whether to use floating point 16 for training\n",
    "    # fp16_opt_level=\"O1\", #see apex AMP optimization level for detail\n",
    "    # prediction_loss_only=True,\n",
    "\n",
    "    logging_dir='/home/kamlesh_kumar/FYP/GPT2Finetunning/logs_dir',  # Directory for storing logs\n",
    "    logging_strategy=\"steps\",           # Log every 'logging_steps'\n",
    "    logging_steps=500,                  # Log every 500 steps\n",
    "\n",
    "    )\n",
    "\n",
    "#instantiating a Trainer class object\n",
    "trainer = Trainer(\n",
    "    model=model,\n",
    "    args=training_args,\n",
    "    data_collator=data_collator,\n",
    "    train_dataset=train_dataset,\n",
    "    eval_dataset=val_dataset\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "4 hours of training for 1 epoch and 8.2GBs of GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/kamlesh_kumar/.venv/lib/python3.10/site-packages/transformers/optimization.py:411: FutureWarning: This implementation of AdamW is deprecated and will be removed in a future version. Use the PyTorch implementation torch.optim.AdamW instead, or set `no_deprecation_warning=True` to disable this warning\n",
      "  warnings.warn(\n"
     ]
    },
    {
     "data": {
      "text/html": [
       "\n",
       "    <div>\n",
       "      \n",
       "      <progress value='103' max='9370' style='width:300px; height:20px; vertical-align: middle;'></progress>\n",
       "      [ 103/9370 02:28 < 3:46:24, 0.68 it/s, Epoch 0.11/10]\n",
       "    </div>\n",
       "    <table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       " <tr style=\"text-align: left;\">\n",
       "      <th>Step</th>\n",
       "      <th>Training Loss</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table><p>"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#conduct training\n",
    "trainer.train()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Saving Model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "!mkdir final_model"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "saved_model_path = \"/home/kamlesh_kumar/FYP/GPT2Finetunning/final_model\"\n",
    "model.save_pretrained(saved_model_path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Evaluation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "1. Results of tensorboard are analyzed in the report\n",
    "2. Also I have added example for comparing two different models using human evaluation.\n",
    "    1. model with 1 epoch on all the dataset\n",
    "    2. model with 10 epoch with 30k examples."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TensorFlow installation not found - running with reduced feature set.\n",
      "Serving TensorBoard on localhost; to expose to the network, use a proxy or pass --bind_all\n",
      "TensorBoard 2.15.1 at http://localhost:6006/ (Press CTRL+C to quit)\n",
      "^C\n"
     ]
    }
   ],
   "source": [
    "!tensorboard --logdir=/home/kamlesh_kumar/FYP/GPT2Finetunning/logs_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Story Generation or Inference"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Stopping the prevoius code and inferencing by loading the both the pretrained and fie tunned models separately"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Inference on GPU\n",
    "\n",
    "from transformers import GPT2LMHeadModel, GPT2Tokenizer\n",
    "from transformers import set_seed    \n",
    "\n",
    "# Load the fine-tuned model\n",
    "\n",
    "# Path for model which has been trained for 1 epoch with 273000 examples\n",
    "# saved_model_path = \"/home/kamlesh_kumar/FYP/GPT2Finetunning/1_epoch_final_model\"\n",
    "\n",
    "# path for model which has been trained for 10 epochs with 30000 samples\n",
    "saved_model_path = \"/home/kamlesh_kumar/FYP/GPT2Finetunning/final_model\"\n",
    "\n",
    "pretrained_model = GPT2LMHeadModel.from_pretrained(\"distilgpt2\")\n",
    "\n",
    "# Load the model from the checkpoint\n",
    "fine_tuned_model = GPT2LMHeadModel.from_pretrained(saved_model_path).to('cuda')\n",
    "\n",
    "# Load tokenizer (same for both models)\n",
    "tokenizer = GPT2Tokenizer.from_pretrained(\"distilgpt2\", pad_token='pad')\n",
    "\n",
    "tokenizer.pad_token = tokenizer.eos_token\n",
    "\n",
    "\n",
    "def generate_text(model, prompt):\n",
    "    model_inputs = tokenizer(prompt, return_tensors=\"pt\").to('cuda')\n",
    "\n",
    "#     # Generate text\n",
    "#     output = model.generate(**encoding, max_new_tokens=512)\n",
    "    sample_output = model.generate(\n",
    "        **model_inputs,\n",
    "        max_new_tokens=300,\n",
    "        do_sample=True,\n",
    "        top_k=0\n",
    "    )\n",
    "\n",
    "    return sample_output[0]\n",
    "\n",
    "# prompt = \"\"\"prompt :\n",
    "# and how? :\n",
    "# story:\n",
    "# \"\"\"\n",
    "\n",
    "# # Generate text with pre-trained model\n",
    "# # print(\"\\nPre-trained Model:\")\n",
    "# # print(generate_text(pretrained_model, prompt))\n",
    "\n",
    "# set seed to reproduce results. Feel free to change the seed though to get different results\n",
    "set_seed(42)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "You've finally managed to discover the secret to immortality. Suddenly, Death appears before you, hands you a business card, and says, `` When you realize living forever sucks, call this number, I've got a job offer for you. ''\n",
      "story:\n",
      "`` you really aren't expecting this, are you? '' Death asked me quietly, not really caring if my answer was actually going to snowball. \n",
      " \n",
      " `` I guess that means you're fucking *here*. '' I replied, a bit sheepish as I was trying to rationalize.Things like that usually happened when someone died, or some mundane at that moment suddenly different, in an instant or two. I was right, I was done. I had actually been offered the opportunity. \n",
      " \n",
      " `` Yes. Well, I have actually got this deal for you, and as such you must understand I've lost quite a bit of your previous choice. Great deal. '' He started to shiver, and for a second I was filled with dread. I can't even pin down on what I thought of his choice. `` Am I either going to die like Romeo or not there are... '' I began hoarsely for a second, the painful memory was reliving the moment and acceptance almost overwhelming me in the process. Everyone seemed to me, a few acquaintances who had been laying about their profits in cancer rooms and drug dealing, others with debilitating cancer tormented souls who, at this very moment, feared they would find a way to do it. `` So I'll grant this to you for, like me or not, death? '' I replied sternly. \n",
      " \n",
      " He looked at me disinterestedly and turned away, I considered all this for a\n"
     ]
    }
   ],
   "source": [
    "## 1st story\n",
    "prompt = \"\"\"prompt :\n",
    "You 've finally managed to discover the secret to immortality . Suddenly , Death appears before you , hands you a business card , and says , `` When you realize living forever sucks , call this number , I 've got a job offer for you . ''\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=True))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "You discover a hidden door in your basement that leads to an ancient library filled with unknown books.\n",
      "\n",
      "story:\n",
      "I did it with confidence. \n",
      " \n",
      " I ’ ve heard of this place before, the nice folks that live all over it, live by it ’ s gold. Nevertheless, I have several books unlaced within my basement. The first of them, which was lordly and elegant, contained 300 gold bottles. The second, the ones that have an ornately designed lock with two revolving doors. The third, and the fourth. Each filled with gold, containing what I can only guess was bolen when someone knocked on the door. The truths, as my brother and I learned in college, were written away in those sealed rooms. Some believe that that number spoke for free, others one would die of that theft. It is hard to tell and to have access to something close to books, not only by carefully looking in that order, but by slowly turning them down. \n",
      " \n",
      " It was a dark and stormy night, one that had turned into a stormy night. In the distance, I heard that one door had fallen and was not guarded by the mighty forces of darkness, although I tried to act. I retraced my steps inside the house, utilizing the lock mechanism for my attacks. Maybe three people have been open in that house for the last eight hours, after which I could point them in the right direction and swing the locks on the left. \n",
      " \n",
      " Maybe I should have raised the sword, hidden in my basement. Maybe also\n"
     ]
    }
   ],
   "source": [
    "# 2nd story\n",
    "prompt = \"\"\"prompt :\n",
    "You discover a hidden door in your basement that leads to an ancient library filled with unknown books.\n",
    "\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "Every night, you dream of a mysterious city. One day, you find a map leading to it in real life.\n",
      "\n",
      "story:\n",
      "A city kid rests outside his room playing video games in his room. To his surprise, the city has a map. From there, it starts to glow brighter and richer, until a certain explosion occurs when the player removes his glasses. The kid notices that one of his game characters is glowing brighter, and is able to take the city as a whole. He begins to yell around the room, with a mix of terror and frustration in his mind. As he goes around the room, more small explosions erupt in the city. Several people try to run down walls; some shout “ Medic! ” The kid screams, but his mom just blocks the walls, causing a woman to run and run over with a baby. Behind her, a bellowing amount of gunfire erupts, as lightning cracks in the city, and everyone just collapses out.\n",
      " \n",
      " When the sound of gunshots erupts, another kid looks up. He can't tell what kind of danger he is in, but when people are running, he can feel a flood of bullets go by him. It ’ s terrifying, actually. \n",
      "\n",
      " \n",
      " “ Hello soldier. ” As the shot of bullets flies through the air, he sees he can ’ t be sure. He screams to his mom, and his dad. He runs down the street, scrambling up and down the stairs towards his room. The person with the blue and purple uniform says, “ What am I thinking? ” The\n"
     ]
    }
   ],
   "source": [
    "# 2nd story\n",
    "prompt = \"\"\"prompt :\n",
    "Every night, you dream of a mysterious city. One day, you find a map leading to it in real life.\n",
    "\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "You inherit an old watch that allows you to time travel, but only to moments of personal significance.\"\n",
      "\n",
      "story:\n",
      "I hate needles. At least I hate them. They're like chalk. \n",
      " \n",
      " There are things that are freakin'in my stomach everyday. I've seen them like it on a coworkers'office supply market. Watch them bleed out to pieces, dye them- red and: just pieces of shit. It's not the freakin'misiest thing, and yeah, most certainly not with *this* goddamn standard watch. There are things that will never look past me. \n",
      " \n",
      " You see, it's not the freakin'misest thing I vision, the exception just because the people with the bright purple watch you 'd've known are the true elite, the true elite, the true prize. \n",
      " \n",
      " The problem is, they should all die right? It gets a little late though: wake up, there's hair on the body, still on the sheets. Tannhose is an understatement, those idiots sleep right in. And if you're *really* taking these little shits, there's an internal bleeding problem: you have to use a body to change, somehow. Here's just how important it is, really. \n",
      " \n",
      " Well, screw that up. \n",
      " \n",
      " I *use* to be fully aware, and are constantly switching shifts whenever possible, so nobody knows where or what to do in my body to go, so I'm on a cellular level +1 in a flying, rusty, malfunction\n"
     ]
    }
   ],
   "source": [
    "# 3rd story\n",
    "prompt = \"\"\"prompt :\n",
    "You inherit an old watch that allows you to time travel, but only to moments of personal significance.\"\n",
    "\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "One morning, you wake up in a world where everyone can hear each other's thoughts.\n",
      "\n",
      "story:\n",
      "What the fuck kind of person is this? Everybody is so fucking goddamn gullible that they can see the color streamed from their phones while I was eating my cereal, and this is the only time I live in my apartment, so I had to call Diane over to make sure everything went fine, and no good lies are about to spread across the world before you try to tell them to stay away from me. I'll tell you what you realize is my greatest fear. \n",
      " \n",
      " As I walked out of the door I saw you all dolled up to the sky. \n",
      " \n",
      " I saw no one there, but I sluggishly looked over and saw you all dolled up, because what? Who are you and what's your name? I guess I'm about 12 or ''. My girlfriend's been dead for two weeks... no ONE said she 'd seen me fall to her death, like everyone else. \n",
      " \n",
      " Still, enough of these thoughts, to drive me crazy. Eventually I got about an hour after you showed up. I was so scared of the dark and asked you, `` Desmond, please call the Father from the Far East. '' Then the phone rang. `` No '', I tried to speak up, `` I don't have the man to talk to. '' \n",
      " \n",
      " `` He has an important job, \n",
      " He has Joshua who needs to be with his parents since the moment we were born. '' You seemed to\n"
     ]
    }
   ],
   "source": [
    "# 4th story\n",
    "prompt = \"\"\"prompt :\n",
    "One morning, you wake up in a world where everyone can hear each other's thoughts.\n",
    "\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "A mysterious figure leaves a puzzle box on your doorstep every year on your birthday.\n",
      "\n",
      "story:\n",
      "The presents arrived precisely at 11:00 this day. I no longer heard Mr. Christopher Tolkien himself talk excitedly after saving the day. I was turned to the side because I didn't know if he knew I would wake up this morning with an actual skeleton in my closet. Unaware of his actions, I had opened the box to curiosity and addressed it to whoever I was supposed to be dead. \n",
      " \n",
      " `` Christopher... um... my birthday... am I dead? '' \n",
      " \n",
      " I've got a few confused looks as I read the newspaper. `` MIREARY NEEDED TO BE FREE!!! '' \n",
      " \n",
      " Wadday then? This is certainly not a real birthday wish. I could wish for unlimited knowledge for everyone. Wouldn't he know I didn't wish for unlimited knowledge? What do I know now? What does that have to do with me? \n",
      " \n",
      " Mr. Christopher Tolkien seems to have awoken. I am wishing I could get a new one and the world would be okay. Hmm... This isn't a job I'm in love with. The only reason I'm here is because it's my job! Surprised by the gift this is a job I didn't receive long before the government put me into this position. The reason I felt sad when I was offered this position, more than anyone would ever assume. \n",
      " \n",
      " `` Excuse me handsome. I'm Sir Christopher `` I look at the picture of himself in\n"
     ]
    }
   ],
   "source": [
    "# 5th story\n",
    "prompt = \"\"\"prompt :\n",
    "A mysterious figure leaves a puzzle box on your doorstep every year on your birthday.\n",
    "\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "You find a strange plant in your garden that grows objects instead of flowers.\n",
      "\n",
      "story:\n",
      "What bugger, what am I going to do with anything anyway? The back--style clock reminding me of quitting early and the sooner it comes to 8am already I can tell the neighbours what is going on. The neighbours counter, stocked up with alcohol and the nearest dog to the door with 8 am standing patiently, patiently waiting to be sniffed up and walked off with the rest of the empty sky. \n",
      " \n",
      " The empty spot on the front garden was alarming, wide enough for the whole neighbourhood to sit in. I had installed something along the market in an attempt to add to my monthly squabble, and while it would have come to an end why not, a sip of lemonade was going to help and help fill the house with the energy needed for the evening. \n",
      " \n",
      " It was always the case that things got messy when I got rid of something I needed to clean, I had to leave the bar to find other people to enjoy the evening whilst I waited for the house to close and I pulled over `` I want to live on my own '' as an ad system for a local company hiring people to schedule maintenance. I scoured the abandoned grey stone, curious what that might mean for them - shadows, observing the terrain laid out for miles wasn't one of the reasons I expected, nothing other than marrusted unpainted flowers and plain white wallpaper, no dead leaves, no one was wilted in my haste to find more than two-by\n"
     ]
    }
   ],
   "source": [
    "# 6th story\n",
    "prompt = \"\"\"prompt :\n",
    "You find a strange plant in your garden that grows objects instead of flowers.\n",
    "\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "A diary from the future falls into your hands, describing events yet to happen in your life..\n",
      "\n",
      "story:\n",
      "I wake up now, confused and exhausted. Emergency call lead to proportions of pockets, coins and other odds. Gangs greet me from the doorway, they yell at me as if it is not the right way in the world. I reach into my pockets to find my purse. Is there something inside me? I hesitate, I left it in my pocket somewhere to get a sewing kit and now it contains what I consider to be an ordinary circular object. I feel confused, the components have become unimportant, such as a pocket watch. An old part of me has been spending the last 18 years on the casing for years. Whenever I was of age I still remember. We joked, made jokes, made conversations. We drank coffee, made love. We sat at the pastel beds, people around us laughed and talked. We made love. We kicked young men until their 90's. We were married, we wanted a kid with a pocket watch and no home. We moved in, moved to a new neighborhood. I don't forget because I was young, they were there when the earthquake hit.\n",
      " \n",
      " I grew up in a harsh neighborhood, had a okay home when my grandfather was out. I lived a quiet neighborhood, always taking advantage of the resulting stress of leaving home for work, so I always prepared myself to move to a nicer neighborhood. Though I could have been walking past the street for a second too long, I never had any luck with my work. I remember\n"
     ]
    }
   ],
   "source": [
    "# 7th story\n",
    "prompt = \"\"\"prompt :\n",
    "A diary from the future falls into your hands, describing events yet to happen in your life..\n",
    "\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "You receive a letter from your future self warning you about a decision you're about to make.\n",
      "story:\n",
      "A change of clothing is the most important decision, to make sure your `` Jew '' speech is not actually over. This beautiful woman has made her experience known. She has to wear a special dress for a special of a special occasion. Her favorite piece of clothing is the one that hints at the physical beauty of a person's appearance. It is: a burgundy red dress that reveals the choices of manufacturing components having been purchased for months. \n",
      " \n",
      " She has to wear a specially fitted suit that would fit the description of any gifted person in the world. It is ridiculous, she states. Her voice sounds monotonous.. unless used in a heard from her vicinity, she will say something like: \n",
      " \n",
      " `` Please absolve your sins with whatever weapon you can find on earth. If you will like to make copies of your dress you will need to send them to the future, as great as possible. What you will need must be told from your future, by great philosophers, than from your future self. '' \n",
      " \n",
      " GOD, she will tell you the intricacies of her body that everyone else in the world thought about her as a Jew. The way her body is curly, she is in the shape of a fox, possibly bigger than your average arm. More specifically, the fox she studies over the years with a 19th century lint in her hand. \n",
      " \n",
      " `` Should I remind you the important thing everyone needs in their lives, they\n"
     ]
    }
   ],
   "source": [
    "# 8th story\n",
    "prompt = \"\"\"prompt :\n",
    "You receive a letter from your future self warning you about a decision you're about to make.\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "The moon is actually a giant egg, and it has just started to hatch.\n",
      "story:\n",
      "An oily click clings to the chirp as Gene slams shutters his funeral parlor. Dave screams across the concrete floor, his breath rattling in his ears. \n",
      " \n",
      " `` Fucking tin foil. '' Gene mutters. He turns around and stares at Risley, the last of his kind maintained seated at the foot of his bed, watching closely as Risley absorbs the egg into his hands. \n",
      " \n",
      " Risley adjusts her collar for reassurance she doesn't storm in and when she does the dreaded silence of the funeral she jumps unceremoniously. \n",
      " \n",
      " watches Ollivander shuffle over to the table and Risley begins to sob, rubbing his eyes. Her death cries above her become desperate screams. \n",
      " \n",
      " Risley puts on her best red startle and backflips Risley puts her fingers up to her throat. She hears a clattering of metal and wood. Her eyes are just as stunned as Risley, thrusting her arms around her to stop her screams. \n",
      " \n",
      " The hatch opens slowly, engulfing Risley and the space between the soft concrete floor and the glittering glass. \n",
      " \n",
      " `` The moon's been really going crazy recently, '' Risley shouts from the other side, touching the head of the large egg as she squints and whines to see a small blue circle marking the base of the moon. \n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 9th story\n",
    "prompt = \"\"\"prompt :\n",
    "The moon is actually a giant egg , and it has just started to hatch .\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=False))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Setting `pad_token_id` to `eos_token_id`:50256 for open-end generation.\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Output:\n",
      "----------------------------------------------------------------------------------------------------\n",
      "prompt :\n",
      "You discover you can communicate with animals.\n",
      "story:\n",
      "He was in the middle of a famine zone when a man migrated to Israel. He had started fleeing for food the food court brought upon and began getting whipped. Everyone wanted to be there for him. That was until he met a girl he loved who was a tribal girl. They kept tabs on him and kept tabs on him. Eventually he got arrested and he was probed three times and thrusted. Luckily for him, he had a beautiful girlfriend who recently returned from a large hunting trip. He still loves her, and he enjoyed living there. Until the day he began to wonder. I have a secret. I have never trusted someone who really loves me. The first time I truly got around to them I discovered that they have no love to share. After many months, he seems to become obsessed with me. Suddenly, researching and restraining me on some land a few countries have in the middle east. After some time I figured I would learn to live amongst humans. I even mastered my best tricks of rope and rope. \n",
      " \n",
      " And then on a good night at the farm. I work on animals. There was no match in sight for a farmer or ant. I then came to help in the field with antelope. It was the last time I ever tasted something alive. It might have been delicious food though.\n",
      " \n",
      " Everything seemed to be working just right. He still showed up incessantly twiddling his thumbs. Somehow I didn't notice my humans or anything.\n"
     ]
    }
   ],
   "source": [
    "# 10th story\n",
    "prompt = \"\"\"prompt :\n",
    "You discover you can communicate with animals.\n",
    "story:\n",
    "\"\"\"\n",
    "\n",
    "output_ids_generation = generate_text(fine_tuned_model, prompt)\n",
    "\n",
    "print(\"Output:\\n\" + 100 * '-')\n",
    "print(tokenizer.decode( output_ids_generation , skip_special_tokens=False))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## THE END"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Rerences:\n",
    "\n",
    "https://huggingface.co/blog/how-to-generate\n",
    "https://www.kaggle.com/code/ratthachat/writingprompts-gpt2-lm-fine-tune/notebook\n",
    "\n",
    "\n"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}