thematic_analysis.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instruction\n",
    "\n",
    "The is a notebook for conducting thematic analysis. The prompts can be found in the paper. \n",
    "\n",
    "Before you start using this notebook, please make sure you have your OpenAI API key in `chat_modes.py`.\n",
    "\n",
    "There are four steps.\n",
    "1. Extracted the codes\n",
    "2. Grouped the codes the themes\n",
    "3. Label the responses\n",
    "4. Evaluation\n",
    "\n",
    "\n",
    "\n",
    "Please adhere to the instructions provided in each step. \n",
    "\n",
    "Possible Errors:\n",
    "1. Format Error: Due to the GPT may output the format of the result differently, sometimes you may encounter a format error. Please re-run the cell to tackle this issue.\n",
    "2. maximum context length: If you have a large number of free-text responses, the total token count may exceed the maximum context length (16k for GPT-3.5 and 32k for GPT-4). If you encounter this issue, please consider the solution we implemented in our work on the password manager dataset. You can find detailed information in our paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import os\n",
    "import json\n",
    "import chat_mods"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 1. Extracted the codes\n",
    "\n",
    "We provided 4 exemplars to ChatGPT.\n",
    "ChatGPT will generated the codes.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<u>Please provide some exemplars (under Exemplars)in the prompt. We suggested 4-8 examplars will be good.</u>\n",
    "\n",
    "Here is the example:\n",
    "```\n",
    "Task: The aim of this project is to investigate which aspects of the music listening experience are prioritized by people listening to music on their personal devices.\n",
    "\n",
    "Here are 4 examples for how to generate the codes. For each example, you will see one response with the codes step by step.\n",
    "\n",
    "These responses are the answer of the question: What’s the first thing that comes into your mind about this track?\n",
    "\n",
    "Each generated code have the format like:\n",
    "'quote' refers to /mentions 'definition of the code'. Therefore, we got a code: 'code'. \n",
    "\n",
    "Examplars:\n",
    "\n",
    "(1)\n",
    "Response: Nostalgia, friends, sing along. \n",
    "\n",
    "Action: Let's code the response.\n",
    "\"Nostalgia, friends, sing along\" mentions Nostalgia. Therefore, we got a code: Nostalgia.\n",
    "\n",
    "(2)\n",
    "Response: I don't actually know or remember it.\n",
    "Action: Let's code the response.\n",
    "\"I don't actually know or remember it\" refers to something not known or recognized. Therefore, we got a code: unfamiliar.\n",
    "\n",
    "\n",
    "(3)\n",
    "Response: Emotion I like the song and I really like the singers voice.\n",
    "\n",
    "Action: Let's code the response.\n",
    "\"I like the song and I really like the singers\" refers to “constructive, optimistic, or confident”. Therefore, we got a code: Positive.\n",
    "\"I like the song and I really like the singers voice\" refers to the singer’s voice. Therefore, we got a code: vocals.\n",
    "\n",
    "\n",
    "(4)\n",
    "Response: I remember really liking this song when it came out and dancing to it.\n",
    "\n",
    "Action: Let's code the response.\n",
    "“I remember really liking this song when it came out and dancing to it” refers to “constructive, optimistic, or confident”. Therefore, we got a code: Positive\n",
    "“I remember really liking this song when it came out and dancing to it” refers to “memory”. Therefore, we got a code: Memory\n",
    "“I remember really liking this song when it came out and dancing to it” mentions “dancing”. Therefore, we got a code: Dancing\n",
    "\n",
    "You need to generate the codes with the format:\n",
    "'quote' refers to /mentions 'definition of the code'. Therefore, we got a code: 'code'. \n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [],
   "source": [
    "exemplars_for_codes = '''Task: [Put the your task description here]\n",
    "\n",
    "Here are 4 examples for how to generate the codes. For each example, you will see one response with the codes step by step.\n",
    "\n",
    "These responses are the answer of the question: What’s the first thing that comes into your mind about this track?\n",
    "\n",
    "Each generated code have the format like:\n",
    "'quote' refers to /mentions 'definition of the code'. Therefore, we got a code: 'code'. \n",
    "\n",
    "Exemplars:\n",
    "[Put your exemplars here]\n",
    "\n",
    "You need to generate the codes with the format:\n",
    "'quote' refers to /mentions 'definition of the code'. Therefore, we got a code: 'code'. \n",
    "'''"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompt_for_codes, text_for_machine = chat_mods.load_data(\"your_free_text_responses.txt\")\n",
    "\n",
    "init_codes, msgs, content = chat_mods.init_code_gen(exemplars_for_codes, prompt_for_codes)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# The cell is for generating the summary of the result. It might take a while to finish\n",
    "sum_codes, codes = chat_mods.sum_code_gen(init_codes, msgs)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Codes Review\n",
    "You can review the generated codes below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "print(codes)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 2 Group the codes to themes\n",
    "\n",
    "The result will be the grouped themes. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "question = \"\" # Put the question of the survey here. e.g. How do you manage your password?\n",
    "machine_themes, msgs_theme = chat_mods.codes_to_themes(sum_codes, codes, question)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The themes have been stored in the file called <b>revised_theme.json</b>. Please find and edit the file.\n",
    "\n",
    "If you find that the results are unsatisfactory, it is recommended to restart the kernel and begin again from Step 1. This will ensure a fresh start and potentially improve the outcome.\n",
    "\n",
    "You have the ability to edit the content, including changing the theme name, adding/deleting/merging themes, and relocating the code to another theme.\n",
    "#### Please remember to save the file as \"revised_theme.json\" once you have finished.\n",
    "\n",
    "\n",
    "You need to write down your actions in action_description (in the next cell)\n",
    "Here are the examples.\n",
    "```\n",
    "action_description = \"\"\"\n",
    "1. I merged the \"Relatability and Connection\" into \"Personal Connection\" since they are similar.\n",
    "2. I deleted \"Specific Elements\" since it is too specific.\n",
    "3. I merged \"Sensual and Sexual\", \"Catchiness and Appeal\", and \"Dream-like and Ethereal\" into \"Emotional Response\" since these are the emotional response.\n",
    "4. I deleted \"Political Activism\" since it is too specific.\n",
    "5. I merged \"Association and Context\" into \"Familiarity and Memory\" since it belongs to that theme.\n",
    "\"\"\"\n",
    "```\n",
    "\n",
    "Your task is to provide feedback to the machine coder. The loop will continue until saturation is reached. The end result will be the code book."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##### Discussion Starting Point"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "action_description = \"\"\"\n",
    "\n",
    "\"\"\"\n",
    "### Write your actions above."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After you edited the themes and codes in revised_theme, please run the cell below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "msgs_theme = chat_mods.discussion(machine_themes, msgs_theme, action_description)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Continue or Complete?\n",
    "#### -Continue\n",
    "\n",
    "To initiate a new round of discussion, execute the cell below.\n",
    "Return to the <b>Discussion Starting Point</b> to repeat the process.\n",
    "\n",
    "\n",
    "If you believe that saturation has been reached, please proceed to the <b>-Complete</b> step."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### -Complete\n",
    "Final Code book Generation\n",
    "\n",
    "Run the cell below"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "machine_theme, msgs_theme = chat_mods.diss_process(msgs_theme)\n",
    "final_code_book = chat_mods.final_codebook_gen(\"codebook.json\", msgs_theme)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 3. label the responses based on the final code book.\n",
    "Please code the free-text responses based on the final code book <u>codebook.json</u>.\n",
    "\n",
    "In the the codebook.json file, each code is labeled with a number. Please label the responses using the corresponding code numbers. If a response contains multiple codes, please separate them with a <b>space</b>.\n",
    "\n",
    "For instance:\n",
    "```\n",
    "{\n",
    "\"3\": {\n",
    "    \"cultural\": \"the association of the song with a specific country\"\n",
    "     }\n",
    "},\n",
    "{\n",
    "\"4\": {\n",
    "    \"music genre\": \"the association of the song with a specific music genre\"\n",
    "     }\n",
    "```\n",
    "\n",
    "If you would like to label the response as \"culture\" and \"music genre,\" please assign them the labels 3 and 4, respectively."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "code_result = chat_mods.label(final_code_book, text_for_machine)\n",
    "print(\"Completed\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Step 4. Evaluation\n",
    "To calculate the Inner-Annotator-Agreement (IAA), you can refer to the NLTK package (https://www.nltk.org/_modules/nltk/metrics/agreement.html)\n",
    "The machine coded result is stored as coded_result.json"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Finished\n",
    "You should get the following files after you complete the whole process.\n",
    "\n",
    "* codebook: codebook.json\n",
    "* machine coded result: coded_result.json\n",
    "* (revised_theme.json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python (llm-ta)",
   "language": "python",
   "name": "llm-ta"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.16"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}