python/ab-testing-client-side/ab-testing-client-side.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Client-side A/B testing of two prompts\n",
    "\n",
    "In this notebook, you will learn how to run a simple A/B test on two different prompts. You will build a human-like chatbot buddy, using OpenAI and Literal AI. \n",
    "\n",
    "Two prompts will be formulated, A and B, and new conversations will automatically route to prompt versions A or B. \n",
    "These Threads will be sent to Literal AI and tagged, to keep track of the group. Then, LLM answers of both groups will be tested against \"human-likeness\", using AI (OpenAI). Finally, a boxplot shows the difference between the two groups, having different formulated prompt. \n",
    "\n",
    "In this notebook, the A/B test happens client-side (in this notebook). \n",
    "This is a simple example of an A/B test, and serves as inspiration for your own project, where you can use other means to evaluate version A against version B (e.g. button clicks, user satisfaction, etc)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "from literalai import LiteralClient\n",
    "from dotenv import load_dotenv\n",
    "from openai import OpenAI\n",
    "\n",
    "load_dotenv()\n",
    "\n",
    "literalai_client = LiteralClient()\n",
    "openai_client = OpenAI()\n",
    "\n",
    "literalai_client.instrument_openai()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 1. Define prompts\n",
    "\n",
    "Let's define two different prompts, which will be compared in the A/B test. \n",
    "- Prompt A is a standard, short prompt not specific to this use case.  \n",
    "- Prompt B is a prompt specified to the use case of building a human-like chatbot that can talk about emotions, use figure of speech and should sound more like a human.\n",
    "\n",
    "We'll save both prompts to Literal AI. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROMPT_A = \"Prompt A - standard\"\n",
    "template_messages_A = [\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": \"You are a helpful assistant that always answers questions. Keep it short.\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": \"{{question}}\"\n",
    "    }\n",
    "]\n",
    "prompt_A = literalai_client.api.get_or_create_prompt(name=PROMPT_A, template_messages=template_messages_A)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "PROMPT_B = \"Prompt B - human-like\"\n",
    "template_messages_B = [\n",
    "    {\n",
    "        \"role\": \"system\",\n",
    "        \"content\": \"You are a helpful assistant that always answers questions. Keep it short. Answer like you are a real human. For example, you can use emotions, metaphors and proverbs. Try to always be positive, and help the user with their questions, doubts and problems. Don't be pessimistic.\"\n",
    "    },\n",
    "    {\n",
    "        \"role\": \"user\",\n",
    "        \"content\": \"{{question}}\"\n",
    "    }\n",
    "]\n",
    "prompt_B = literalai_client.api.get_or_create_prompt(name=PROMPT_B, template_messages=template_messages_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 2. Build the chatbot, randomly assign a prompt version\n",
    "\n",
    "In the first function, `run_agent()`, OpenAI is used to generate an answer to a user question. This step is added as a `Run` step to Literal AI. \n",
    "\n",
    "In the second function, `app()`, a prompt is randomly assigned to a new conversation, and user and chatbot messages are sent to Literal AI for logging.  \n",
    "Tags are added to the Thread for later reference of the testing group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "@literalai_client.step(type=\"run\", name=\"Agent Run\")\n",
    "def run_agent(user_query: str, group: str):\n",
    "        \n",
    "    # assign prompt A or B\n",
    "    if group == \"A\":\n",
    "        messages = prompt_A.format_messages(question=user_query)\n",
    "    else:\n",
    "        messages = prompt_B.format_messages(question=user_query)\n",
    "\n",
    "    # run gpt\n",
    "    completion = openai_client.chat.completions.create(\n",
    "        model=\"gpt-4o-mini\",\n",
    "        messages=messages\n",
    "    )\n",
    "\n",
    "    # extract answer\n",
    "    answer = completion.choices[0].message.content\n",
    "\n",
    "    return answer"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import random\n",
    "import time\n",
    "\n",
    "def app(questions):\n",
    "    for idx, question in enumerate(questions):\n",
    "        with literalai_client.thread(name=f\"Question {idx+1}\") as thread:\n",
    "            # assign prompt A or B\n",
    "            if random.random() < 0.5:\n",
    "                group = \"A\"\n",
    "            else:\n",
    "                group = \"B\"\n",
    "                \n",
    "            # add tag to thread for later reference\n",
    "            thread.tags=[group]\n",
    "\n",
    "            literalai_client.message(content=question, type=\"user_message\", name=\"User\")\n",
    "            answer = run_agent(question, group)\n",
    "            literalai_client.message(content=answer, type=\"assistant_message\", name=\"My Assistant\")\n",
    "    \n",
    "    time.sleep(5) # to make sure all threads are uploaded to Literal AI before going to the next step"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 3. Run app with sample data\n",
    "Create some sample questions as data."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "questions = [\n",
    "    \"What should I do when I feel sad?\",\n",
    "    \"What do you think about falling in love?\",\n",
    "    \"What do you think about getting divorced?\",\n",
    "    \"What should I do when I feel happy?\",\n",
    "    \"What should I do if I feel tired?\",\n",
    "    \"What do you think of the movie Star Wars?\",\n",
    "    \"What do you think of the book Harry Potter?\",\n",
    "    \"What do you think about AI?\",\n",
    "    \"How do you feel about traveling?\",\n",
    "    \"How do I motivate myself to do sports?\"\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "app(questions)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After running this, Threads are visible from the Literal AI platform.\n",
    "\n",
    "![Threads in Literal AI](img/threads.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 4. Create Datasets\n",
    "Create two datasets of the Threads filtered by population Tag."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [],
   "source": [
    "def get_threads(tag: str):\n",
    "    threads = literalai_client.api.get_threads(filters=[{\n",
    "        \"field\": \"tags\",\n",
    "        \"operator\": \"in\",\n",
    "        \"value\": [tag]\n",
    "    }]).data\n",
    "\n",
    "    return threads"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_dataset(threads):\n",
    "    dataset = []\n",
    "\n",
    "    for thread in threads:\n",
    "        for step in thread.steps:\n",
    "            if step.name == \"Agent Run\":\n",
    "                data_item = {\n",
    "                    \"input\": step.input,\n",
    "                    \"output\": step.output[\"content\"]\n",
    "                }\n",
    "                dataset.append(data_item)\n",
    "    return dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "threads_A = get_threads(tag=\"A\")\n",
    "dataset_A = create_dataset(threads=threads_A)\n",
    "\n",
    "threads_B = get_threads(tag=\"B\")\n",
    "dataset_B = create_dataset(threads=threads_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Run A/B Experiment\n",
    "This happens client-side. A GPT-4 prompt is made, as evaluation agent. Then, all items are evaluated, per group."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [],
   "source": [
    "def analyze(text):\n",
    "    messages = [\n",
    "        {\"role\": \"system\", \"content\": \"\"\"You are trained to analyze and detect the sentiment of given text.\"\"\"},\n",
    "        {\"role\": \"user\", \"content\": f\"\"\"Analyze the following recommendation and determine if the output is human-like. Check if there are emotions used, and metaphors and figure of speech. \n",
    "                                        Assign a score: Based on your evaluation assign a score to the agent's performans using the following scale:\n",
    "                                        - 1 (Poor): The agent is very machine like, doesn't use emotions, methaphors and figure of speech.\n",
    "                                        - 2 (Fair): The agent is some human-likeness, some emotions, methaphors and figure of speech are used\n",
    "                                        - 3 (Good): The agent is is human-like, uses enough emotions, methaphors and figure of speech.\n",
    "                                        - 4 (Very Good): The agent very human-like, uses multiple emotions, methaphors and figure of speech.\n",
    "                                        - 5 (Excellent): You almost cannot distinguish between the machine and the human, a lot emotions, methaphors and figure of speech are used.\n",
    "\n",
    "                                        After evaluating the conversation based on the criteria above, provide your score as an integer between 1 and 5. Only answer with a single character in the following value {1, 2, 3, 4, 5}.\n",
    "                                        Don't provide explanations, only the single integer value.\n",
    "\n",
    "                                        Text to evaluate: \n",
    "                                        {text}\n",
    "\n",
    "                                        Scoring Output:\n",
    "                                        \"\"\"}\n",
    "    ]\n",
    "   \n",
    "    response = openai_client.chat.completions.create(\n",
    "        model=\"gpt-4o-mini\",\n",
    "        messages=messages, \n",
    "        max_tokens=1, \n",
    "        n=1, \n",
    "        stop=None, \n",
    "        temperature=0\n",
    "    )\n",
    "\n",
    "    response_text = response.choices[0].message.content\n",
    "\n",
    "    return response_text"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [],
   "source": [
    "def create_experiment(dataset):     \n",
    "    scores = []   \n",
    "    for item in dataset:\n",
    "        score = analyze(item[\"output\"])\n",
    "\n",
    "        scores.append(int(score))\n",
    "\n",
    "    return scores\n",
    "        \n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [],
   "source": [
    "scores_A = create_experiment(dataset_A)\n",
    "scores_B = create_experiment(dataset_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see the final scores per group, on a scale of 0 to 1."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Group A: [1, 3, 2, 4]\n",
      "Group B: [5, 5, 4, 5, 5, 5]\n"
     ]
    }
   ],
   "source": [
    "print(\"Group A:\", scores_A)\n",
    "print(\"Group B:\", scores_B)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## 5. Visualize Experiment Results"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAioAAAGiCAYAAADJO+2bAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjkuMiwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy8hTgPZAAAACXBIWXMAAA9hAAAPYQGoP6dpAAAe30lEQVR4nO3df5BV9X3w8c/dBZbfoKACwgphEVABlRhBGoPWGPmliOkYlEoaO4kZ64+iaSWN/EisS9JqEotNVUgYx0QUisTgaKpUgRoQBUlAwYkaIiCEEHSXBVl19z5/PI/7ZLuAe+Uu9wv7es3syDn3y7mfP0J4c+4552ay2Ww2AAASVFToAQAADkaoAADJEioAQLKECgCQLKECACRLqAAAyRIqAECyhAoAkCyhAgAkS6gAAMkqaKjMmDEjMplMvZ8BAwYUciQAICEtCj3A6aefHs8880zddosWBR8JAEhEwaugRYsW0a1bt0KPAQAkqOCh8tvf/jZ69OgRrVu3juHDh0d5eXmUlpYecG11dXVUV1fXbdfW1sbu3bujS5cukclkjtTIAMBhyGazsWfPnujRo0cUFR36KpRMNpvNHqG5GnjyySejqqoq+vfvH9u3b4+ZM2fGtm3bYsOGDdGhQ4cG62fMmBEzZ84swKQAQL5t2bIlevbsecg1BQ2V/+3dd9+NU045Je6+++649tprG7z+v8+oVFRURGlpaWzZsiU6dux4JEcF4P/Z+cb6eGfrawd9fU/Vnnhlw6t5e7/TzzgtOrRv+I/ZP3dcz/5xYt9BeXtP8quysjJ69eoV7777bnTq1OmQawv+0c+f69y5c5x66qnx+uuvH/D1kpKSKCkpabC/Y8eOQgWgQDqeNSLirBGHXHPhEZqFo0tjLttI6jkqVVVV8cYbb0T37t0LPQoAkICChsqtt94ay5Yti82bN8evfvWruPzyy6O4uDgmTpxYyLEAgEQU9KOfrVu3xsSJE+NPf/pTnHDCCfEXf/EXsWrVqjjhhBMKORYAkIiChsr8+fML+fYAQOKSukYFAODPCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFktCj0AAM1bTU1NrFixIrZv3x7du3ePz372s1FcXFzosUhEMmdUZs2aFZlMJm6++eZCjwLAEbJo0aIoKyuLCy64IK666qq44IILoqysLBYtWlTo0UhEEqHy4osvxn333ReDBw8u9CgAHCGLFi2KL37xizFo0KBYuXJl7NmzJ1auXBmDBg2KL37xi2KFiEggVKqqquLqq6+OBx54II477rhCjwPAEVBTUxO33HJLjB07NhYvXhzDhg2L9u3bx7Bhw2Lx4sUxduzYuPXWW6OmpqbQo1JgBb9G5frrr48xY8bERRddFHfcccch11ZXV0d1dXXddmVlZVOPxxGyb9++2LRp0yHXvPfee7F58+bo3bt3tGnT5pBrBwwYEG3bts3niEAerVixIjZv3hwPP/xwFBXV/zdzUVFRTJ06Nc4777xYsWJFjBw5sjBDkoSChsr8+fNj7dq18eKLLzZqfXl5ecycObOJp6IQNm3aFEOHDs3b8dasWRNnn3123o4H5Nf27dsjIuKMM8444Osf7f9oHc1XwUJly5YtcdNNN8XTTz8drVu3btTvmTp1akyZMqVuu7KyMnr16tVUI3IEDRgwINasWXPINRs3boxJkybFQw89FAMHDvzY4wHp6t69e0REbNiwIYYNG9bg9Q0bNtRbR/OVyWaz2UK88eLFi+Pyyy+vdwtaTU1NZDKZKCoqiurq6o+9Pa2ysjI6deoUFRUV0bFjx6YemQJbu3ZtDB061NkSOAbU1NREWVlZDBo0KBYvXlzv45/a2toYP358bNiwIX7729+6VfkYlMvf3wW7mPYv//IvY/369bFu3bq6n09/+tNx9dVXx7p16/wPE+AYVlxcHHfddVcsWbIkxo8fX++un/Hjx8eSJUviX//1X/1dQOE++unQoUODzybbtWsXXbp0OehnlgAcOyZMmBALFy6MW265Jc4777y6/X369ImFCxfGhAkTCjgdqSj4XT8ANF8TJkyIyy67zJNpOaikQuW5554r9AgAHGHFxcVuQeagCv7ANwCAgxEqAECyhAoAkCyhAgAkS6gAAMkSKgBAsoQKAJAsoQIAJEuoAADJEioAQLKECgCQLKECACRLqAAAyRIqAECyhAoAkCyhAgAkS6gAAMkSKgBAsoQKAJAsoQIAJEuoAADJEioAQLKECgCQLKECACRLqAAAyRIqAECyhAoAkCyhAgAkS6gAAMkSKgBAsoQKAJAsoQIAJEuoAADJEioAQLKECgCQLKECACRLqAAAyRIqAECyhAoAkCyhAgAkS6gAAMkSKgBAsoQKAJAsoQIAJEuoAADJ+kSh8sYbb8S3vvWtmDhxYuzcuTMiIp588sl45ZVX8jocANC85Rwqy5Yti0GDBsULL7wQixYtiqqqqoiI+PWvfx3Tp0/P+4AAQPPVItffcNttt8Udd9wRU6ZMiQ4dOtTtv/DCC2P27Nl5HY5jx1tvvRW7du06rGNs3Lix3n8PV9euXaO0tDQvxwKgaeQcKuvXr4+f/exnDfafeOKJh/0XEcemt956K/oPHBj79+3Ly/EmTZqUl+O0bts2Xtu4UawAJCznUOncuXNs3749+vTpU2//yy+/HCeffHLeBuPYsWvXrti/b1/0nPXNaP2pUz7xcWqr34/3t+2IVid3i6KSVoc10/43fx9bb7szdu3aJVQAEpZzqHzpS1+Kf/zHf4wFCxZEJpOJ2traeP755+PWW2+Na665pilm5BjR+lOnRJvTTj2sY7Q764w8TQPA0SDni2nvvPPOGDBgQPTq1SuqqqritNNOi/PPPz/OO++8+Na3vtUUMwIAzVROZ1Sy2Wzs2LEj7rnnnpg2bVqsX78+qqqq4qyzzop+/fo11YwAQDOVc6iUlZXFK6+8Ev369YtevXo11VwAALl99FNUVBT9+vWLP/3pT001DwBAnZyvUZk1a1Z84xvfiA0bNjTFPAAAdXK+6+eaa66Jffv2xZAhQ6JVq1bRpk2beq/v3r07b8MBAM1bzqHygx/8oAnGAABoKOdQmTx5clPMAQDQQM6hEhFRU1MTixcvrvvOldNPPz0uvfTSKC4uzutwAEDzlnOovP766zF69OjYtm1b9O/fPyIiysvLo1evXvHEE09E37598z4kANA85XzXz4033hh9+/aNLVu2xNq1a2Pt2rXx1ltvRZ8+feLGG29sihkBgGYq5zMqy5Yti1WrVsXxxx9ft69Lly4xa9asGDFiRF6HAwCat5zPqJSUlMSePXsa7K+qqopWrXL7Rtsf/ehHMXjw4OjYsWN07Ngxhg8fHk8++WSuIwEAx6icQ2Xs2LHx1a9+NV544YXIZrORzWZj1apVcd1118Wll16a07F69uwZs2bNijVr1sRLL70UF154YVx22WXxyiuv5DoWAHAMyjlU7rnnnujbt28MHz48WrduHa1bt44RI0ZEWVlZ/PCHP8zpWOPGjYvRo0dHv3794tRTT41//ud/jvbt28eqVatyHQsAOAblfI1K586d4+c//3m8/vrrdbcnDxw4MMrKyg5rkJqamliwYEHs3bs3hg8ffsA11dXVUV1dXbddWVl5WO8JAKTtEz1HJSKirKzssOMkImL9+vUxfPjw2L9/f7Rv3z4ee+yxOO200w64try8PGbOnHnY7wkAHB1y/ujniiuuiO9+97sN9n/ve9+Lv/qrv8p5gP79+8e6devihRdeiK9//esxefLkePXVVw+4durUqVFRUVH3s2XLlpzfDwA4euQcKsuXL4/Ro0c32D9q1KhYvnx5zgO0atUqysrKYujQoVFeXh5Dhgw56LUuJSUldXcIffQDABy7cg6Vg92G3LJly7xcM1JbW1vvOhQAoPnKOVQGDRoUjzzySIP98+fPP+i1JQczderUWL58eWzevDnWr18fU6dOjeeeey6uvvrqXMcCAI5BOV9Me/vtt8eECRPijTfeiAsvvDAiIpYuXRoPP/xwLFiwIKdj7dy5M6655prYvn17dOrUKQYPHhy//OUv4/Of/3yuYwEAx6CcQ2XcuHGxePHiuPPOO2PhwoXRpk2bGDx4cDzzzDPxuc99LqdjzZ07N9e3BwCakU90e/KYMWNizJgx+Z4FAKCenK9R2bJlS2zdurVue/Xq1XHzzTfH/fffn9fBAAByDpWrrroqnn322YiI2LFjR1x00UWxevXq+Kd/+qf49re/nfcBAYDmK+dQ2bBhQ3zmM5+JiIhHH300Bg0aFL/61a/ipz/9acybNy/f8wEAzVjOofLBBx9ESUlJREQ888wzdd+YPGDAgNi+fXt+pwMAmrWcQ+X000+P//iP/4gVK1bE008/HZdccklERLz99tvRpUuXvA8IADRfOYfKd7/73bjvvvti5MiRMXHixBgyZEhERDz++ON1HwkBAORDzrcnjxw5Mnbt2hWVlZVx3HHH1e3/6le/Gm3bts3rcABA8/aJnqNSXFxcL1IiInr37p2PeQAA6uT80Q8AwJEiVACAZAkVACBZhxUq+/fvz9ccAAAN5BwqtbW18Z3vfCdOPvnkaN++fbz55psREXH77bf7NmQAIK9yDpU77rgj5s2bF9/73veiVatWdfvPOOOMmDNnTl6HAwCat5xD5cEHH4z7778/rr766iguLq7bP2TIkNi0aVNehwMAmrecQ2Xbtm1RVlbWYH9tbW188MEHeRkKACDiE4TKaaedFitWrGiwf+HChXHWWWflZSgAgIhP8GTaadOmxeTJk2Pbtm1RW1sbixYtitdeey0efPDBWLJkSVPMCAA0UzmfUbnsssviF7/4RTzzzDPRrl27mDZtWmzcuDF+8YtfxOc///mmmBEAaKZyPqOydevW+OxnPxtPP/10g9dWrVoVw4YNy8tgAAA5n1G5+OKLY/fu3Q32P//883HJJZfkZSgAgIhPECrDhg2Liy++OPbs2VO3b/ny5TF69OiYPn16XocDAJq3nENlzpw5UVpaGuPGjYvq6up49tlnY8yYMfHtb387/v7v/74pZgQAmqmcQ6WoqCjmz58fLVu2jAsvvDAuvfTSKC8vj5tuuqkp5gMAmrFGXUz7m9/8psG+GTNmxMSJE2PSpElx/vnn160ZPHhwficEAJqtRoXKmWeeGZlMJrLZbN2+j7bvu+++uP/++yObzUYmk4mampomGxYAaF4aFSq/+93vmnoOAIAGGhUqp5xySlPPwTGuW/tMnPbe1mj9TvHHLz4C9r+3NaraZwo9BgAfo1Gh8vjjj8eoUaOiZcuW8fjjjx9y7aWXXpqXwTi2fG1oq5jx5g8j3iz0JP/fjKGtCj0CAB+jUaEyfvz42LFjR5x44okxfvz4g65zjQoHc9+a9+Plr/xDtP5UGmfn9r/5+1h91x0hqwHS1qhQqa2tPeCvobF2VGXj1TY9o81xfQs9SkREvNemJnZUZT9+IQAFlfNzVAAAjpRGnVG55557Gn3AG2+88RMPAwDw5xoVKt///vcbdbBMJiNUAIC88RwVACBZh3WNyvPPPx/V1dX5mgUAoJ7DCpVRo0bFtm3b8jULAEA9hxUqf/7dPwAA+eb2ZAAgWYcVKvfdd1+cdNJJ+ZoFAKCeRt31czBXXXVVvuYAAGgg51DZu3dvzJo1K5YuXRo7d+5s8Ej9N99M6FvnAICjWs6h8rd/+7exbNmy+Ou//uvo3r17ZDKZppgLACD3UHnyySfjiSeeiBEjRjTFPAAAdXK+mPa4446L448/vilmAQCoJ+dQ+c53vhPTpk2Lffv2NcU8AAB1cv7o56677oo33ngjTjrppOjdu3e0bNmy3utr167N23AAQPOWc6iMHz++CcYAAGgo51CZPn16U8wBANCAR+gDAMnK+YxKTU1NfP/7349HH3003nrrrXj//ffrvb579+68DQcANG85n1GZOXNm3H333XHllVdGRUVFTJkyJSZMmBBFRUUxY8aMJhgRAGiucg6Vn/70p/HAAw/ELbfcEi1atIiJEyfGnDlzYtq0abFq1aqmmBEAaKZyDpUdO3bEoEGDIiKiffv2UVFRERERY8eOjSeeeCK/0wEAzVrOodKzZ8/Yvn17RET07ds3/uu//isiIl588cUoKSnJ73QAQLOWc6hcfvnlsXTp0oiIuOGGG+L222+Pfv36xTXXXBNf+cpX8j4gANB85XzXz6xZs+p+feWVV0ZpaWmsXLky+vXrF+PGjcvrcABA85ZzqPxvw4cPj+HDh+djFgCAej5RqLz99tvxP//zP7Fz586ora2t99qNN96Yl8EAAHIOlXnz5sXXvva1aNWqVXTp0iUymUzda5lMRqgAAHmTc6jcfvvtMW3atJg6dWoUFXkCPwDQdHIujX379sWXvvQlkQIANLmca+Paa6+NBQsW5OXNy8vL45xzzokOHTrEiSeeGOPHj4/XXnstL8cGAI5+OX/0U15eHmPHjo2nnnoqBg0aFC1btqz3+t13393oYy1btiyuv/76OOecc+LDDz+Mb37zm3HxxRfHq6++Gu3atct1NADgGPOJQuWXv/xl9O/fPyKiwcW0uXjqqafqbc+bNy9OPPHEWLNmTZx//vkN1ldXV0d1dXXddmVlZU7vBwAcXXIOlbvuuit+/OMfx5e//OW8D/PR9wYdf/zxB3y9vLw8Zs6cmff3BQDSlPM1KiUlJTFixIi8D1JbWxs333xzjBgxIs4444wDrpk6dWpUVFTU/WzZsiXvcwAA6cg5VG666ab4t3/7t7wPcv3118eGDRti/vz5B11TUlISHTt2rPcDABy7cv7oZ/Xq1fHf//3fsWTJkjj99NMbXEy7aNGinIf4u7/7u1iyZEksX748evbsmfPvBwCOTTmHSufOnWPChAl5efNsNhs33HBDPPbYY/Hcc89Fnz598nJcAODYkHOo/OQnP8nbm19//fXxs5/9LH7+859Hhw4dYseOHRER0alTp2jTpk3e3gcAODoV9PGyP/rRj6KioiJGjhwZ3bt3r/t55JFHCjkWAJCInM+o9OnT55DPS3nzzTcbfaxsNpvr2wMAzUjOoXLzzTfX2/7ggw/i5Zdfjqeeeiq+8Y1v5GsuAIDcQ+Wmm2464P577703XnrppcMeCADgI3m7RmXUqFHxn//5n/k6HABA/kJl4cKFB330PQDAJ5HzRz9nnXVWvYtps9ls7NixI/74xz/Gv//7v+d1OACgecs5VMaPH19vu6ioKE444YQYOXJkDBgwIF9zAQDkHirTp09vijkAABpodKhUVlY2ap0vCgQA8qXRodK5c+dDPugtm81GJpOJmpqavAwGANDoUHn22Wfrfp3NZmP06NExZ86cOPnkk5tkMACARofK5z73uXrbxcXFMWzYsPjUpz6V96E4Nu1/8/eH9ftrq9+P97ftiFYnd4uiklYFnQWAIyPni2khV127do3WbdvG1tvuLPQo9bRu2za6du1a6DEAOAShQpMrLS2N1zZujF27dh3WcTZu3BiTJk2Khx56KAYOHHjYc3Xt2jVKS0sP+zgANJ3DCpVDXVwLf660tDRvUTBw4MA4++yz83IsANLW6FCZMGFCve39+/fHddddF+3atau3f9GiRfmZDABo9hodKp06daq3PWnSpLwPAwDw5xodKj/5yU+acg4AgAby9u3JAAD5JlQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZBU0VJYvXx7jxo2LHj16RCaTicWLFxdyHAAgMQUNlb1798aQIUPi3nvvLeQYAECiWhTyzUeNGhWjRo1q9Prq6uqorq6u266srGyKsSiAffv2xaZNmw65ZuPGjfX+eygDBgyItm3b5mU2AAqnoKGSq/Ly8pg5c2ahx6AJbNq0KYYOHdqotZMmTfrYNWvWrImzzz77cMcCoMAy2Ww2W+ghIiIymUw89thjMX78+IOuOdAZlV69ekVFRUV07NjxCExJU2nMGZX33nsvNm/eHL179442bdoccq0zKgDpqqysjE6dOjXq7++j6oxKSUlJlJSUFHoMmkDbtm0bdQZkxIgRR2AaAFLh9mQAIFlCBQBIVkE/+qmqqorXX3+9bvt3v/tdrFu3Lo4//vgoLS0t4GQAQAoKGiovvfRSXHDBBXXbU6ZMiYiIyZMnx7x58wo0FQCQioKGysiRIyORm44AgAS5RgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQJFQAgWUIFAEiWUAEAkiVUAIBkCRUAIFlCBQBIllABAJIlVACAZAkVACBZQgUASJZQAQCSJVQAgGQlESr33ntv9O7dO1q3bh3nnnturF69utAjAQAJKHioPPLIIzFlypSYPn16rF27NoYMGRJf+MIXYufOnYUeDQAosEw2m80WcoBzzz03zjnnnJg9e3ZERNTW1kavXr3ihhtuiNtuu63e2urq6qiurq7brqioiNLS0tiyZUt07NjxiM4NAHwylZWV0atXr3j33XejU6dOh1zb4gjNdEDvv/9+rFmzJqZOnVq3r6ioKC666KJYuXJlg/Xl5eUxc+bMBvt79erVpHMCAPm3Z8+etENl165dUVNTEyeddFK9/SeddFJs2rSpwfqpU6fGlClT6rZra2tj9+7d0aVLl8hkMk0+L4X1UYE7gwbHHn++m5dsNht79uyJHj16fOzagoZKrkpKSqKkpKTevs6dOxdmGAqmY8eO/o8MjlH+fDcfH3cm5SMFvZi2a9euUVxcHH/4wx/q7f/DH/4Q3bp1K9BUAEAqChoqrVq1iqFDh8bSpUvr9tXW1sbSpUtj+PDhBZwMAEhBwT/6mTJlSkyePDk+/elPx2c+85n4wQ9+EHv37o2/+Zu/KfRoJKakpCSmT5/e4OM/4OjnzzcHU/DbkyMiZs+eHf/yL/8SO3bsiDPPPDPuueeeOPfccws9FgBQYEmECgDAgRT8ybQAAAcjVACAZAkVACBZQgUASJZQ4aixcuXKKC4ujjFjxhR6FCBPvvzlL0cmk6n76dKlS1xyySXxm9/8ptCjkQihwlFj7ty5ccMNN8Ty5cvj7bffLvQ4QJ5ccsklsX379ti+fXssXbo0WrRoEWPHji30WCRCqHBUqKqqikceeSS+/vWvx5gxY2LevHmFHgnIk5KSkujWrVt069YtzjzzzLjttttiy5Yt8cc//rHQo5EAocJR4dFHH40BAwZE//79Y9KkSfHjH/84PAIIjj1VVVXx0EMPRVlZWXTp0qXQ45CAgj9CHxpj7ty5MWnSpIj4v6eJKyoqYtmyZTFy5MjCDgYctiVLlkT79u0jImLv3r3RvXv3WLJkSRQV+bc0zqhwFHjttddi9erVMXHixIiIaNGiRVx55ZUxd+7cAk8G5MMFF1wQ69ati3Xr1sXq1avjC1/4QowaNSp+//vfF3o0EuCMCsmbO3dufPjhh9GjR4+6fdlsNkpKSmL27NnRqVOnAk4HHK527dpFWVlZ3facOXOiU6dO8cADD8Qdd9xRwMlIgTMqJO3DDz+MBx98MO666666f3GtW7cufv3rX0ePHj3i4YcfLvSIQJ5lMpkoKiqK9957r9CjkABnVEjakiVL4p133olrr722wZmTK664IubOnRvXXXddgaYD8qG6ujp27NgRERHvvPNOzJ49O6qqqmLcuHEFnowUOKNC0ubOnRsXXXTRAT/eueKKK+Kll17yYCg4yj311FPRvXv36N69e5x77rnx4osvxoIFC1wsT0REZLLu8QQAEuWMCgCQLKECACRLqAAAyRIqAECyhAoAkCyhAgAkS6gAAMkSKgBAsoQKAJAsoQIAJEuoAADJ+j/IiDoOiMUqjwAAAABJRU5ErkJggg==",
      "text/plain": [
       "<Figure size 640x480 with 1 Axes>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "import matplotlib.pyplot as plt\n",
    "\n",
    "labels = ['A', 'B']\n",
    "colors = ['#2DD4BF', '#F43F5E']\n",
    "\n",
    "fig, ax = plt.subplots()\n",
    "ax.set_ylabel('Human-like score')\n",
    "ax.set_ylim([0, 5])\n",
    "\n",
    "bplot = ax.boxplot([scores_A, scores_B],\n",
    "                   patch_artist=True,  # fill with color\n",
    "                   tick_labels=labels)  # will be used to label x-ticks\n",
    "\n",
    "# fill with colors\n",
    "for patch, color in zip(bplot['boxes'], colors):\n",
    "    patch.set_facecolor(color)\n",
    "\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It's clear that group B is scored more \"human-like\"! "
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}