syntax_cleaning.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "a112ddf7",
   "metadata": {},
   "source": [
    "# Data Cleaning Task\n",
    "#### 2/24/23\n",
    "https://docs.google.com/document/d/1JbR3mLBW1kZCSrqBYMhQs4AC-4Hk5OOO2UCLflP76eY/edit"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e44fecf",
   "metadata": {},
   "source": [
    "## Understanding the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "538efa2b",
   "metadata": {},
   "outputs": [],
   "source": [
    "import numpy as np # Used once for np.nan\n",
    "import pandas as pd\n",
    "from scipy.stats import zscore\n",
    "import datetime # Used to get datetime from unix timestamp\n",
    "\n",
    "pd.options.mode.chained_assignment = None # Avoids an unnecessary warning"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7a3bdd24",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(15150, 26)"
      ]
     },
     "execution_count": 2,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = pd.read_csv(\"results_0221.csv\") # Import the original results dataset\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a9de7706",
   "metadata": {},
   "source": [
    "### Quick notes and observations\n",
    "\n",
    "(I'm looking at the dataset on Numbers, since jupyterhub can't easily display the whole table.) \n",
    "\n",
    "The original dataset has 15,150 rows and 26 columns. It includes all the data received from the experiment divided into separate user entries. Each entry has information on the answers to the consent survey, the experiment (as well as the practice trials), and the background survey.\n",
    "\n",
    "Maho pointed out that it seems like some users participated more than once. Since the desired effect might be different on familiar users, it's important to identify these users and remove their repeated entries.\n",
    " \n",
    "Column info:\n",
    "1. The time the results were received (Unix timestamp) - all of these will be the same for each row in a given entry.\n",
    "2. IP address hash. Likewise, all of these will be the same for each row in a given entry. They will also be the same for a given user, so it can be used to find the users who participated more than once.\n",
    "3. Only says \"PennController\", so not useful.  \n",
    "4. The items' order number. These are all grouped in chunks of 5. Not sure what the \"item\" refers to.\n",
    "5. Only says 0, so not useful.\n",
    "6. The type of question (consent, exercise, experiment-filler, experiment-critical, or background)\n",
    "7. Only says NULL, so not useful.\n",
    "8. The type of element displayed (\"Scale\" occurs every 5 rows).\n",
    "9. The name given to the above.\n",
    "10. \"Parameter\" - unclear to me.\n",
    "11. Value - seems to include the choice that the user picks on the scale (Important!).\n",
    "12. \"Event\" time - probably not useful.\n",
    "13. Only says undefined, so not useful.\n",
    "\n",
    "Columns 14-21 include answers to the background surveys.\n",
    "\n",
    "22. The list that the stimulus was taken from (always the same for a given participant's experiment trials).\n",
    "23. The stimulus sentence's order number.\n",
    "24. The condition (either practice, good, intermediate, bad, or one of the 4 critical conditions).\n",
    "25. The actual sentence.\n",
    "26. The comments (seem to be mostly empty, but worth checking before deleting)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "384c5519",
   "metadata": {},
   "source": [
    "## Step 1: Extract submission time"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "566f2ea0",
   "metadata": {},
   "source": [
    "Instead of extracting the submission time first, I'll extract it later during Step 6 by converting the Unix timestamps into date times."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "6665346e",
   "metadata": {},
   "source": [
    "## Step 2: Extract relevant rows"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "7a3198d1",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(12337, 26)"
      ]
     },
     "execution_count": 3,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Only include rows that say \"experiment-filler\", \"experiment-critical\", or \"background\" under column V6.\n",
    "df = df[(df[\"V6\"] == \"experiment-filler\") | \n",
    "        (df[\"V6\"] == \"experiment-critical\") | \n",
    "        (df[\"V6\"] == \"background\")]\n",
    "df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "a8cf0cc7",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test: Are there only three trial types in V6?\n",
    "assert df[\"V6\"].nunique() == 3"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f5fcbec5",
   "metadata": {},
   "source": [
    "## Step 3: Identify and recode unique subject IDs"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "d3dabd79",
   "metadata": {
    "scrolled": false
   },
   "outputs": [],
   "source": [
    "# Change column V1 from Unix time to an ordinal number, starting at 1.\n",
    "df[\"V2\"] = df.groupby(\"V2\", sort=False).ngroup() + 1\n",
    "df = df.rename(columns={\"V2\": \"subj_id\"})"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "d0f82269",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test: is the list of users equal to the numbers 1...35?\n",
    "assert df[\"subj_id\"].drop_duplicates().to_list() == list(range(1, 36))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5edd4def",
   "metadata": {},
   "source": [
    "## Step 4: Find and remove multiple submissions from the same user"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "51f1cb7a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    28\n",
       "2     7\n",
       "Name: V1, dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "n_submissions = df.groupby(\"subj_id\")[\"V1\"].nunique() # Find the number of submissions per user.\n",
    "n_submissions.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f1f8a353",
   "metadata": {},
   "source": [
    "There were 35 users total: 28 submitted once, 7 submitted twice, and no one submitted more than twice.\n",
    "\n",
    "In order to remove the duplicate entries from the dataset, my strategy is:\n",
    "1. Remove ALL duplicates in the subj_id column.\n",
    "2. Remember which timestamps designate a user's first entry.\n",
    "3. Remove all rows where the timestamp does not designate a user's first entry."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e8f7735f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# List of timestamps that designate a user's first entry.\n",
    "first_entry_ts = df.drop_duplicates(subset=\"subj_id\", keep=\"first\")[\"V1\"].to_list()\n",
    "# Test: since there are 35 users, there should be 35 first entries.\n",
    "assert len(first_entry_ts) == 35 "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "21216707",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    35\n",
       "Name: V1, dtype: int64"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df = df[df[\"V1\"].isin(first_entry_ts)]\n",
    "n_submissions = df.groupby(\"subj_id\")[\"V1\"].nunique()\n",
    "n_submissions.value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a5d2dddf",
   "metadata": {},
   "source": [
    "Now, the dataset does not include any repeated submissions – all 35 users only have one each."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a6badc45",
   "metadata": {},
   "source": [
    "## Step 5: Separate acceptability judgments and responses to language background questionnaire"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f4339fd4",
   "metadata": {},
   "outputs": [],
   "source": [
    "df_ju = df[(df[\"V6\"] == \"experiment-filler\") | \n",
    "             (df[\"V6\"] == \"experiment-critical\")]\n",
    "df_bg = df[df[\"V6\"] == \"background\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "6bc79f5d",
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "# Test: in the judgment dataset, are there only two trial types in V6?\n",
    "assert df_ju[\"V6\"].nunique() == 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "id": "34323ced",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Test: in the background dataset, is there only one trial type in V6?\n",
    "assert df_bg[\"V6\"].nunique() == 1"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "88278f49",
   "metadata": {},
   "source": [
    "## Step 6: Extract relevant information from the subject info dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12f26e81",
   "metadata": {},
   "source": [
    "As I mentioned in Step 1, I'll extract the submission time now: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "id": "3cab7b30",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0   2023-02-14 17:08:33\n",
       "1   2023-02-14 17:43:31\n",
       "2   2023-02-14 18:52:07\n",
       "3   2023-02-14 18:56:30\n",
       "4   2023-02-14 21:13:03\n",
       "Name: V1, dtype: datetime64[ns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "timestamps = df_bg[\"V1\"].drop_duplicates().reset_index(drop=True)\n",
    "submission_times = timestamps.apply(lambda s: datetime.datetime.utcfromtimestamp(int(s)))\n",
    "submission_times.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "611381a4",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>V9</th>\n",
       "      <th>subj_id</th>\n",
       "      <th>input_US_age</th>\n",
       "      <th>input_age</th>\n",
       "      <th>input_birth</th>\n",
       "      <th>input_comf</th>\n",
       "      <th>input_gender</th>\n",
       "      <th>input_lang</th>\n",
       "      <th>input_lang_comf</th>\n",
       "      <th>input_parent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Others</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "V9  subj_id input_US_age input_age  input_birth input_comf input_gender  \\\n",
       "0         1          NaN        20  In the U.S.    English       Female   \n",
       "1         2          NaN        22  In the U.S.    English       Female   \n",
       "2         3          NaN        22  In the U.S.    English       Female   \n",
       "3         4          NaN        20  In the U.S.    English       Female   \n",
       "4         5          NaN        20  In the U.S.    English       Female   \n",
       "\n",
       "V9  input_lang input_lang_comf input_parent  \n",
       "0          NaN             NaN      English  \n",
       "1          NaN             NaN      English  \n",
       "2          NaN             NaN      English  \n",
       "3          NaN             NaN      English  \n",
       "4   Vietnamese             NaN       Others  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_bg = df_bg[(df_bg[\"V10\"] == \"Final\") | (df_bg[\"V10\"] == \"Choice\")]\n",
    "df_bg = df_bg[[\"subj_id\", \"V9\", \"V11\"]]\n",
    "df_bg = pd.pivot(df_bg, index=\"subj_id\", columns=\"V9\", values=\"V11\").reset_index()\n",
    "df_bg.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a751afa",
   "metadata": {},
   "source": [
    "The index column on the left titled \"V9\" will be removed when the dataset is exported.\n",
    "\n",
    "Other than that, there's a minor formatting issue where commas are represented as \"%2C\". I'll fix that real quick."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "ea5dbbd5",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>V9</th>\n",
       "      <th>subj_id</th>\n",
       "      <th>input_US_age</th>\n",
       "      <th>input_age</th>\n",
       "      <th>input_birth</th>\n",
       "      <th>input_comf</th>\n",
       "      <th>input_gender</th>\n",
       "      <th>input_lang</th>\n",
       "      <th>input_lang_comf</th>\n",
       "      <th>input_parent</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>31</th>\n",
       "      <td>32</td>\n",
       "      <td>NaN</td>\n",
       "      <td>21</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>First English, then a bit of Tagalog (Filipino)</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Others</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>32</th>\n",
       "      <td>33</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Male</td>\n",
       "      <td>Japanese, English</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Others</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>33</th>\n",
       "      <td>34</td>\n",
       "      <td>NaN</td>\n",
       "      <td>19</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>Kannada, Tamil</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Others</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "V9  subj_id input_US_age input_age  input_birth input_comf input_gender  \\\n",
       "31       32          NaN        21  In the U.S.    English       Female   \n",
       "32       33          NaN        19  In the U.S.    English         Male   \n",
       "33       34          NaN        19  In the U.S.    English       Female   \n",
       "\n",
       "V9                                       input_lang input_lang_comf  \\\n",
       "31  First English, then a bit of Tagalog (Filipino)             NaN   \n",
       "32                                Japanese, English             NaN   \n",
       "33                                   Kannada, Tamil             NaN   \n",
       "\n",
       "V9 input_parent  \n",
       "31       Others  \n",
       "32       Others  \n",
       "33       Others  "
      ]
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# If the input is a string, replace \"%2C\" with a comma.\n",
    "def comma_if_str(s):\n",
    "    if type(s) == str:\n",
    "        return s.replace(\"%2C\", \",\")\n",
    "    return np.nan\n",
    "\n",
    "df_bg[\"input_lang\"] = df_bg[\"input_lang\"].apply(comma_if_str)\n",
    "df_bg[31:34]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8d4e07cf",
   "metadata": {},
   "source": [
    "## Step 7: Add the submission date extracted in Step 1 to the subject info dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "id": "72879653",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th>V9</th>\n",
       "      <th>subj_id</th>\n",
       "      <th>input_US_age</th>\n",
       "      <th>input_age</th>\n",
       "      <th>input_birth</th>\n",
       "      <th>input_comf</th>\n",
       "      <th>input_gender</th>\n",
       "      <th>input_lang</th>\n",
       "      <th>input_lang_comf</th>\n",
       "      <th>input_parent</th>\n",
       "      <th>submission_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "      <td>2023-02-14 17:08:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "      <td>2023-02-14 17:43:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "      <td>2023-02-14 18:52:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "      <td>2023-02-14 18:56:30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Others</td>\n",
       "      <td>2023-02-14 21:13:03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "V9  subj_id input_US_age input_age  input_birth input_comf input_gender  \\\n",
       "0         1          NaN        20  In the U.S.    English       Female   \n",
       "1         2          NaN        22  In the U.S.    English       Female   \n",
       "2         3          NaN        22  In the U.S.    English       Female   \n",
       "3         4          NaN        20  In the U.S.    English       Female   \n",
       "4         5          NaN        20  In the U.S.    English       Female   \n",
       "\n",
       "V9  input_lang input_lang_comf input_parent     submission_time  \n",
       "0          NaN             NaN      English 2023-02-14 17:08:33  \n",
       "1          NaN             NaN      English 2023-02-14 17:43:31  \n",
       "2          NaN             NaN      English 2023-02-14 18:52:07  \n",
       "3          NaN             NaN      English 2023-02-14 18:56:30  \n",
       "4   Vietnamese             NaN       Others 2023-02-14 21:13:03  "
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_bg[\"submission_time\"] = submission_times\n",
    "df_bg.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fbc44aac",
   "metadata": {},
   "source": [
    "The background dataset is finished and is stored as __df_bg__."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "789eacda",
   "metadata": {},
   "source": [
    "## Step 8: Extract relevant information from the acceptability judgment data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "id": "47a800ce",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subj_id</th>\n",
       "      <th>V22</th>\n",
       "      <th>V23</th>\n",
       "      <th>V6</th>\n",
       "      <th>V24</th>\n",
       "      <th>V25</th>\n",
       "      <th>V11</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Everyone takes advantage of the few sunny days...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>Do you remember the girl who Jack commented on...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>11.0</td>\n",
       "      <td>experiment-critical</td>\n",
       "      <td>nested_island</td>\n",
       "      <td>I talked about the man that I like the truck t...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>18.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>Who edited the article that we will have acces...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    subj_id V22   V23                   V6            V24  \\\n",
       "68        1   1   9.0    experiment-filler            bad   \n",
       "73        1   1   8.0    experiment-filler           good   \n",
       "78        1   1   7.0    experiment-filler   intermediate   \n",
       "83        1   1  11.0  experiment-critical  nested_island   \n",
       "88        1   1  18.0    experiment-filler            bad   \n",
       "\n",
       "                                                  V25 V11  \n",
       "68  I had every hour many days swum 100 meters in ...   2  \n",
       "73  Everyone takes advantage of the few sunny days...   6  \n",
       "78  Do you remember the girl who Jack commented on...   5  \n",
       "83  I talked about the man that I like the truck t...   3  \n",
       "88  Who edited the article that we will have acces...   5  "
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ju = df_ju[(df_ju[\"V10\"] == \"Choice\") | (df_ju[\"V10\"] == \"Selection\")]\n",
    "df_ju = df_ju[[\"subj_id\", \"V22\", \"V23\", \"V6\", \"V24\", \"V25\", \"V11\"]]\n",
    "df_ju.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "586a0b40",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subj_id</th>\n",
       "      <th>list</th>\n",
       "      <th>item</th>\n",
       "      <th>trial_type</th>\n",
       "      <th>condition</th>\n",
       "      <th>sentence</th>\n",
       "      <th>judgment</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>2</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Everyone takes advantage of the few sunny days...</td>\n",
       "      <td>6</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>Do you remember the girl who Jack commented on...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>11.0</td>\n",
       "      <td>experiment-critical</td>\n",
       "      <td>nested_island</td>\n",
       "      <td>I talked about the man that I like the truck t...</td>\n",
       "      <td>3</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>18.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>Who edited the article that we will have acces...</td>\n",
       "      <td>5</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    subj_id list  item           trial_type      condition  \\\n",
       "68        1    1   9.0    experiment-filler            bad   \n",
       "73        1    1   8.0    experiment-filler           good   \n",
       "78        1    1   7.0    experiment-filler   intermediate   \n",
       "83        1    1  11.0  experiment-critical  nested_island   \n",
       "88        1    1  18.0    experiment-filler            bad   \n",
       "\n",
       "                                             sentence judgment  \n",
       "68  I had every hour many days swum 100 meters in ...        2  \n",
       "73  Everyone takes advantage of the few sunny days...        6  \n",
       "78  Do you remember the girl who Jack commented on...        5  \n",
       "83  I talked about the man that I like the truck t...        3  \n",
       "88  Who edited the article that we will have acces...        5  "
      ]
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ju.columns = [\"subj_id\", \"list\", \"item\", \"trial_type\", \"condition\", \"sentence\", \"judgment\"]\n",
    "df_ju.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "2ccd3f3f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Quickly gonna make the item column int instead of float.\n",
    "df_ju[\"item\"] = df_ju[\"item\"].apply(int)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7a9cdf99",
   "metadata": {},
   "source": [
    "## Step 9: Convert acceptability judgments to z-scores"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "id": "1eb71861",
   "metadata": {},
   "outputs": [],
   "source": [
    "# The ints in the judgment column are actually strings – need to typecast them first.\n",
    "df_ju[\"judgment\"] = df_ju[\"judgment\"].apply(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "73e584cf",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subj_id</th>\n",
       "      <th>list</th>\n",
       "      <th>item</th>\n",
       "      <th>trial_type</th>\n",
       "      <th>condition</th>\n",
       "      <th>sentence</th>\n",
       "      <th>judgment</th>\n",
       "      <th>zscore</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.000603</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Everyone takes advantage of the few sunny days...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.094118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>Do you remember the girl who Jack commented on...</td>\n",
       "      <td>5</td>\n",
       "      <td>0.570437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>experiment-critical</td>\n",
       "      <td>nested_island</td>\n",
       "      <td>I talked about the man that I like the truck t...</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.476923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>Who edited the article that we will have acces...</td>\n",
       "      <td>5</td>\n",
       "      <td>0.570437</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    subj_id list  item           trial_type      condition  \\\n",
       "68        1    1     9    experiment-filler            bad   \n",
       "73        1    1     8    experiment-filler           good   \n",
       "78        1    1     7    experiment-filler   intermediate   \n",
       "83        1    1    11  experiment-critical  nested_island   \n",
       "88        1    1    18    experiment-filler            bad   \n",
       "\n",
       "                                             sentence  judgment    zscore  \n",
       "68  I had every hour many days swum 100 meters in ...         2 -1.000603  \n",
       "73  Everyone takes advantage of the few sunny days...         6  1.094118  \n",
       "78  Do you remember the girl who Jack commented on...         5  0.570437  \n",
       "83  I talked about the man that I like the truck t...         3 -0.476923  \n",
       "88  Who edited the article that we will have acces...         5  0.570437  "
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Apply corresponding zscores.\n",
    "df_ju[\"zscore\"] = df_ju.groupby(\"subj_id\")[\"judgment\"].apply(lambda x: zscore(x, ddof=1))\n",
    "df_ju.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e082d194",
   "metadata": {},
   "source": [
    "Quick note – I tried to calculate z-score using mean and stdev, and found that two of the results had a __stdev of 0__ (specifically subj 17 and 22):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "0ed98024",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "subj_id\n",
       "17    0.0\n",
       "22    0.0\n",
       "Name: judgment, dtype: float64"
      ]
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "stds = df_ju.groupby(\"subj_id\")[\"judgment\"].std()\n",
    "stds[stds == 0]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "73b75fda",
   "metadata": {},
   "source": [
    "Which means that subj 17 and 22 answered the same for every trial. Maho says we should remove them (and others who might not have been paying attention) from the dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "875d645d",
   "metadata": {},
   "source": [
    "## Attention check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f2177e67",
   "metadata": {},
   "source": [
    "Procedure:\n",
    "\n",
    "1. Get a list of 10 fillers whose mean acceptability was the highest, and 10 lowest (20 in total).\n",
    "2. Check each participant's judgment of each of the 20 fillers, and see if their judgment was more than 2 standard deviations away from the mean\n",
    "3. If a participant's judgment met this criterion more than 5 out of 20 times, exclude the participant from data analysis "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4bbf13de",
   "metadata": {},
   "source": [
    "### Part 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "36c3d03a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subj_id</th>\n",
       "      <th>list</th>\n",
       "      <th>item</th>\n",
       "      <th>trial_type</th>\n",
       "      <th>condition</th>\n",
       "      <th>sentence</th>\n",
       "      <th>judgment</th>\n",
       "      <th>zscore</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.000603</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Everyone takes advantage of the few sunny days...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.094118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>Do you remember the girl who Jack commented on...</td>\n",
       "      <td>5</td>\n",
       "      <td>0.570437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>Who edited the article that we will have acces...</td>\n",
       "      <td>5</td>\n",
       "      <td>0.570437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>Which teachers are the administrator firing at...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.094118</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    subj_id list  item         trial_type     condition  \\\n",
       "68        1    1     9  experiment-filler           bad   \n",
       "73        1    1     8  experiment-filler          good   \n",
       "78        1    1     7  experiment-filler  intermediate   \n",
       "88        1    1    18  experiment-filler           bad   \n",
       "93        1    1     9  experiment-filler  intermediate   \n",
       "\n",
       "                                             sentence  judgment    zscore  \n",
       "68  I had every hour many days swum 100 meters in ...         2 -1.000603  \n",
       "73  Everyone takes advantage of the few sunny days...         6  1.094118  \n",
       "78  Do you remember the girl who Jack commented on...         5  0.570437  \n",
       "88  Who edited the article that we will have acces...         5  0.570437  \n",
       "93  Which teachers are the administrator firing at...         6  1.094118  "
      ]
     },
     "execution_count": 23,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Only include filler sentences\n",
    "df_check = df_ju[df_ju[\"trial_type\"] == \"experiment-filler\"]\n",
    "df_check.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "a3e94fc0",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subj_id</th>\n",
       "      <th>list</th>\n",
       "      <th>item</th>\n",
       "      <th>trial_type</th>\n",
       "      <th>condition</th>\n",
       "      <th>sentence</th>\n",
       "      <th>judgment</th>\n",
       "      <th>zscore</th>\n",
       "      <th>sent_mean</th>\n",
       "      <th>sent_zscore</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.000603</td>\n",
       "      <td>1.885714</td>\n",
       "      <td>0.111778</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Everyone takes advantage of the few sunny days...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.094118</td>\n",
       "      <td>5.885714</td>\n",
       "      <td>0.075676</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>Do you remember the girl who Jack commented on...</td>\n",
       "      <td>5</td>\n",
       "      <td>0.570437</td>\n",
       "      <td>4.942857</td>\n",
       "      <td>0.036380</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>Who edited the article that we will have acces...</td>\n",
       "      <td>5</td>\n",
       "      <td>0.570437</td>\n",
       "      <td>3.400000</td>\n",
       "      <td>1.162563</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>93</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>Which teachers are the administrator firing at...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.094118</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>0.550973</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    subj_id list  item         trial_type     condition  \\\n",
       "68        1    1     9  experiment-filler           bad   \n",
       "73        1    1     8  experiment-filler          good   \n",
       "78        1    1     7  experiment-filler  intermediate   \n",
       "88        1    1    18  experiment-filler           bad   \n",
       "93        1    1     9  experiment-filler  intermediate   \n",
       "\n",
       "                                             sentence  judgment    zscore  \\\n",
       "68  I had every hour many days swum 100 meters in ...         2 -1.000603   \n",
       "73  Everyone takes advantage of the few sunny days...         6  1.094118   \n",
       "78  Do you remember the girl who Jack commented on...         5  0.570437   \n",
       "88  Who edited the article that we will have acces...         5  0.570437   \n",
       "93  Which teachers are the administrator firing at...         6  1.094118   \n",
       "\n",
       "    sent_mean  sent_zscore  \n",
       "68   1.885714     0.111778  \n",
       "73   5.885714     0.075676  \n",
       "78   4.942857     0.036380  \n",
       "88   3.400000     1.162563  \n",
       "93   5.000000     0.550973  "
      ]
     },
     "execution_count": 24,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the overall mean score for each sentence\n",
    "sent_means = df_check.groupby(\"sentence\")[\"judgment\"].mean()\n",
    "df_check[\"sent_mean\"] = sent_means.loc[df_check[\"sentence\"]].values\n",
    "\n",
    "# Find the overall z-score for each judgment\n",
    "df_check[\"sent_zscore\"] = df_check.groupby(\"sentence\")[\"judgment\"].apply(lambda x: zscore(x, ddof=1))\n",
    "df_check.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "5edeaa43",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>sentence</th>\n",
       "      <th>sent_mean</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>1.885714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>303</th>\n",
       "      <td>The famous siblings together became we 17 peop...</td>\n",
       "      <td>2.000000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>283</th>\n",
       "      <td>To he likes that linguistics would like to the...</td>\n",
       "      <td>2.028571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>248</th>\n",
       "      <td>Peeled the banana in the kitchen the waitress ...</td>\n",
       "      <td>2.142857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>263</th>\n",
       "      <td>There was love one’s true self many men now fr...</td>\n",
       "      <td>2.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>228</th>\n",
       "      <td>What will you buys when you could went on a va...</td>\n",
       "      <td>2.314286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>198</th>\n",
       "      <td>The students described which counselor the dep...</td>\n",
       "      <td>2.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>233</th>\n",
       "      <td>What did Sarah eat that Bob likes a pizza more...</td>\n",
       "      <td>2.657143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>338</th>\n",
       "      <td>What did the talented chef cook ravioli with s...</td>\n",
       "      <td>2.771429</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>113</th>\n",
       "      <td>The lawyer the judge determined the defendant ...</td>\n",
       "      <td>2.800000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>268</th>\n",
       "      <td>The cookies with dried rose petals and organic...</td>\n",
       "      <td>5.600000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>158</th>\n",
       "      <td>There was an enormous painting of a Teddy bear...</td>\n",
       "      <td>5.857143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>103</th>\n",
       "      <td>The married couple who got into a fight starte...</td>\n",
       "      <td>5.857143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>Everyone takes advantage of the few sunny days...</td>\n",
       "      <td>5.885714</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>108</th>\n",
       "      <td>Everyone's favorite boy band from the 90s will...</td>\n",
       "      <td>5.942857</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>163</th>\n",
       "      <td>My personal chef made me a veggie burger when ...</td>\n",
       "      <td>6.028571</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>243</th>\n",
       "      <td>Somebody mentioned that you liked wine so I br...</td>\n",
       "      <td>6.057143</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>143</th>\n",
       "      <td>You wore festive colors to work on Independenc...</td>\n",
       "      <td>6.114286</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>183</th>\n",
       "      <td>Nobody liked the old garden because it was unk...</td>\n",
       "      <td>6.200000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>313</th>\n",
       "      <td>The city closed your favorite pizzeria due to ...</td>\n",
       "      <td>6.342857</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                              sentence  sent_mean\n",
       "68   I had every hour many days swum 100 meters in ...   1.885714\n",
       "303  The famous siblings together became we 17 peop...   2.000000\n",
       "283  To he likes that linguistics would like to the...   2.028571\n",
       "248  Peeled the banana in the kitchen the waitress ...   2.142857\n",
       "263  There was love one’s true self many men now fr...   2.200000\n",
       "228  What will you buys when you could went on a va...   2.314286\n",
       "198  The students described which counselor the dep...   2.600000\n",
       "233  What did Sarah eat that Bob likes a pizza more...   2.657143\n",
       "338  What did the talented chef cook ravioli with s...   2.771429\n",
       "113  The lawyer the judge determined the defendant ...   2.800000\n",
       "268  The cookies with dried rose petals and organic...   5.600000\n",
       "158  There was an enormous painting of a Teddy bear...   5.857143\n",
       "103  The married couple who got into a fight starte...   5.857143\n",
       "73   Everyone takes advantage of the few sunny days...   5.885714\n",
       "108  Everyone's favorite boy band from the 90s will...   5.942857\n",
       "163  My personal chef made me a veggie burger when ...   6.028571\n",
       "243  Somebody mentioned that you liked wine so I br...   6.057143\n",
       "143  You wore festive colors to work on Independenc...   6.114286\n",
       "183  Nobody liked the old garden because it was unk...   6.200000\n",
       "313  The city closed your favorite pizzeria due to ...   6.342857"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find the 10 best and 10 worst sentences (according to the sent_mean column)\n",
    "df_sort = df_check[[\"sentence\", \"sent_mean\"]].drop_duplicates(\"sentence\").sort_values(by=\"sent_mean\")\n",
    "df_sort = pd.concat((df_sort.head(10), df_sort.tail(10)))\n",
    "df_sort"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "12397051",
   "metadata": {},
   "source": [
    "### Part 2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "e15f902b",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subj_id</th>\n",
       "      <th>list</th>\n",
       "      <th>item</th>\n",
       "      <th>trial_type</th>\n",
       "      <th>condition</th>\n",
       "      <th>sentence</th>\n",
       "      <th>judgment</th>\n",
       "      <th>zscore</th>\n",
       "      <th>sent_mean</th>\n",
       "      <th>sent_zscore</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>263</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>There was love one’s true self many men now fr...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.094118</td>\n",
       "      <td>2.200000</td>\n",
       "      <td>2.694944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>313</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>10</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>The city closed your favorite pizzeria due to ...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.000603</td>\n",
       "      <td>6.342857</td>\n",
       "      <td>-3.272302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>948</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>4</td>\n",
       "      <td>0.457816</td>\n",
       "      <td>1.885714</td>\n",
       "      <td>2.067888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1213</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Nobody liked the old garden because it was unk...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.029646</td>\n",
       "      <td>6.200000</td>\n",
       "      <td>-3.023687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1223</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>17</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>What did Sarah eat that Bob likes a pizza more...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.276761</td>\n",
       "      <td>2.657143</td>\n",
       "      <td>2.055947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1243</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Somebody mentioned that you liked wine so I br...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.029646</td>\n",
       "      <td>6.057143</td>\n",
       "      <td>-2.614302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1358</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>To he likes that linguistics would like to the...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.276761</td>\n",
       "      <td>2.028571</td>\n",
       "      <td>2.351605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1378</th>\n",
       "      <td>4</td>\n",
       "      <td>1</td>\n",
       "      <td>12</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>The famous siblings together became we 17 peop...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.276761</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>2.748737</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1598</th>\n",
       "      <td>5</td>\n",
       "      <td>2</td>\n",
       "      <td>7</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>There was an enormous painting of a Teddy bear...</td>\n",
       "      <td>1</td>\n",
       "      <td>-1.096681</td>\n",
       "      <td>5.857143</td>\n",
       "      <td>-3.242640</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1889</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>15</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>The students described which counselor the dep...</td>\n",
       "      <td>7</td>\n",
       "      <td>1.588985</td>\n",
       "      <td>2.600000</td>\n",
       "      <td>2.974443</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1899</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>Peeled the banana in the kitchen the waitress ...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.043076</td>\n",
       "      <td>2.142857</td>\n",
       "      <td>2.805097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1944</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Somebody mentioned that you liked wine so I br...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.140560</td>\n",
       "      <td>6.057143</td>\n",
       "      <td>-2.614302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1959</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>My personal chef made me a veggie burger when ...</td>\n",
       "      <td>1</td>\n",
       "      <td>-1.686468</td>\n",
       "      <td>6.028571</td>\n",
       "      <td>-3.258503</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1979</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>8</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Everyone takes advantage of the few sunny days...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.140560</td>\n",
       "      <td>5.885714</td>\n",
       "      <td>-2.572998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2029</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>To he likes that linguistics would like to the...</td>\n",
       "      <td>7</td>\n",
       "      <td>1.588985</td>\n",
       "      <td>2.028571</td>\n",
       "      <td>2.943735</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2044</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>You wore festive colors to work on Independenc...</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.594651</td>\n",
       "      <td>6.114286</td>\n",
       "      <td>-2.089302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2049</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>11</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>There was love one’s true self many men now fr...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.043076</td>\n",
       "      <td>2.200000</td>\n",
       "      <td>2.694944</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2074</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>The married couple who got into a fight starte...</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.594651</td>\n",
       "      <td>5.857143</td>\n",
       "      <td>-2.077850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2094</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>12</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>The famous siblings together became we 17 peop...</td>\n",
       "      <td>7</td>\n",
       "      <td>1.588985</td>\n",
       "      <td>2.000000</td>\n",
       "      <td>3.435921</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2129</th>\n",
       "      <td>6</td>\n",
       "      <td>3</td>\n",
       "      <td>10</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>The city closed your favorite pizzeria due to ...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.140560</td>\n",
       "      <td>6.342857</td>\n",
       "      <td>-3.272302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2427</th>\n",
       "      <td>7</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>The cookies with dried rose petals and organic...</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.739069</td>\n",
       "      <td>5.600000</td>\n",
       "      <td>-2.312482</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3398</th>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>15</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>The students described which counselor the dep...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.087564</td>\n",
       "      <td>2.600000</td>\n",
       "      <td>2.298433</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3493</th>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>8</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Everyone takes advantage of the few sunny days...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.087564</td>\n",
       "      <td>5.885714</td>\n",
       "      <td>-2.572998</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3578</th>\n",
       "      <td>10</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>What did the talented chef cook ravioli with s...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.087564</td>\n",
       "      <td>2.771429</td>\n",
       "      <td>2.280073</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3748</th>\n",
       "      <td>11</td>\n",
       "      <td>4</td>\n",
       "      <td>20</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>The lawyer the judge determined the defendant ...</td>\n",
       "      <td>6</td>\n",
       "      <td>0.161257</td>\n",
       "      <td>2.800000</td>\n",
       "      <td>2.339704</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4163</th>\n",
       "      <td>12</td>\n",
       "      <td>4</td>\n",
       "      <td>4</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Everyone's favorite boy band from the 90s will...</td>\n",
       "      <td>3</td>\n",
       "      <td>0.229888</td>\n",
       "      <td>5.942857</td>\n",
       "      <td>-2.382236</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4785</th>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>17</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>What did Sarah eat that Bob likes a pizza more...</td>\n",
       "      <td>7</td>\n",
       "      <td>1.391794</td>\n",
       "      <td>2.657143</td>\n",
       "      <td>2.670974</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4960</th>\n",
       "      <td>13</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>Peeled the banana in the kitchen the waitress ...</td>\n",
       "      <td>5</td>\n",
       "      <td>0.435469</td>\n",
       "      <td>2.142857</td>\n",
       "      <td>2.077850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5538</th>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>3</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Nobody liked the old garden because it was unk...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.490599</td>\n",
       "      <td>6.200000</td>\n",
       "      <td>-3.023687</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5648</th>\n",
       "      <td>15</td>\n",
       "      <td>4</td>\n",
       "      <td>17</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>What did Sarah eat that Bob likes a pizza more...</td>\n",
       "      <td>6</td>\n",
       "      <td>0.609364</td>\n",
       "      <td>2.657143</td>\n",
       "      <td>2.055947</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6250</th>\n",
       "      <td>16</td>\n",
       "      <td>4</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>You wore festive colors to work on Independenc...</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.377964</td>\n",
       "      <td>6.114286</td>\n",
       "      <td>-2.760179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6605</th>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.885714</td>\n",
       "      <td>2.067888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9607</th>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.885714</td>\n",
       "      <td>2.067888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11121</th>\n",
       "      <td>25</td>\n",
       "      <td>2</td>\n",
       "      <td>2</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>The married couple who got into a fight starte...</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.243303</td>\n",
       "      <td>5.857143</td>\n",
       "      <td>-2.077850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11131</th>\n",
       "      <td>25</td>\n",
       "      <td>2</td>\n",
       "      <td>5</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>My personal chef made me a veggie burger when ...</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.747933</td>\n",
       "      <td>6.028571</td>\n",
       "      <td>-2.610506</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11888</th>\n",
       "      <td>26</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Everyone's favorite boy band from the 90s will...</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.640662</td>\n",
       "      <td>5.942857</td>\n",
       "      <td>-2.382236</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12373</th>\n",
       "      <td>28</td>\n",
       "      <td>3</td>\n",
       "      <td>6</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Somebody mentioned that you liked wine so I br...</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.937167</td>\n",
       "      <td>6.057143</td>\n",
       "      <td>-2.614302</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12393</th>\n",
       "      <td>28</td>\n",
       "      <td>3</td>\n",
       "      <td>1</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>To he likes that linguistics would like to the...</td>\n",
       "      <td>6</td>\n",
       "      <td>0.920580</td>\n",
       "      <td>2.028571</td>\n",
       "      <td>2.351605</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12438</th>\n",
       "      <td>28</td>\n",
       "      <td>3</td>\n",
       "      <td>20</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>The lawyer the judge determined the defendant ...</td>\n",
       "      <td>7</td>\n",
       "      <td>1.385016</td>\n",
       "      <td>2.800000</td>\n",
       "      <td>3.070861</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12473</th>\n",
       "      <td>28</td>\n",
       "      <td>3</td>\n",
       "      <td>5</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>What will you buys when you could went on a va...</td>\n",
       "      <td>7</td>\n",
       "      <td>1.385016</td>\n",
       "      <td>2.314286</td>\n",
       "      <td>3.230220</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>12498</th>\n",
       "      <td>28</td>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>The married couple who got into a fight starte...</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.937167</td>\n",
       "      <td>5.857143</td>\n",
       "      <td>-2.805097</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>13083</th>\n",
       "      <td>30</td>\n",
       "      <td>2</td>\n",
       "      <td>4</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>Peeled the banana in the kitchen the waitress ...</td>\n",
       "      <td>5</td>\n",
       "      <td>0.284768</td>\n",
       "      <td>2.142857</td>\n",
       "      <td>2.077850</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14889</th>\n",
       "      <td>35</td>\n",
       "      <td>3</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>You wore festive colors to work on Independenc...</td>\n",
       "      <td>2</td>\n",
       "      <td>-0.676631</td>\n",
       "      <td>6.114286</td>\n",
       "      <td>-2.760179</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>14919</th>\n",
       "      <td>35</td>\n",
       "      <td>3</td>\n",
       "      <td>4</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>The cookies with dried rose petals and organic...</td>\n",
       "      <td>1</td>\n",
       "      <td>-1.168725</td>\n",
       "      <td>5.600000</td>\n",
       "      <td>-2.954838</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       subj_id list  item         trial_type     condition  \\\n",
       "263          1    1    11  experiment-filler           bad   \n",
       "313          1    1    10  experiment-filler          good   \n",
       "948          3    2     9  experiment-filler           bad   \n",
       "1213         4    1     3  experiment-filler          good   \n",
       "1223         4    1    17  experiment-filler           bad   \n",
       "1243         4    1     6  experiment-filler          good   \n",
       "1358         4    1     1  experiment-filler           bad   \n",
       "1378         4    1    12  experiment-filler           bad   \n",
       "1598         5    2     7  experiment-filler          good   \n",
       "1889         6    3    15  experiment-filler           bad   \n",
       "1899         6    3     4  experiment-filler           bad   \n",
       "1944         6    3     6  experiment-filler          good   \n",
       "1959         6    3     5  experiment-filler          good   \n",
       "1979         6    3     8  experiment-filler          good   \n",
       "2029         6    3     1  experiment-filler           bad   \n",
       "2044         6    3     9  experiment-filler          good   \n",
       "2049         6    3    11  experiment-filler           bad   \n",
       "2074         6    3     2  experiment-filler          good   \n",
       "2094         6    3    12  experiment-filler           bad   \n",
       "2129         6    3    10  experiment-filler          good   \n",
       "2427         7    4     4  experiment-filler  intermediate   \n",
       "3398        10    2    15  experiment-filler           bad   \n",
       "3493        10    2     8  experiment-filler          good   \n",
       "3578        10    2     2  experiment-filler           bad   \n",
       "3748        11    4    20  experiment-filler           bad   \n",
       "4163        12    4     4  experiment-filler          good   \n",
       "4785        13    3    17  experiment-filler           bad   \n",
       "4960        13    3     4  experiment-filler           bad   \n",
       "5538        15    4     3  experiment-filler          good   \n",
       "5648        15    4    17  experiment-filler           bad   \n",
       "6250        16    4     9  experiment-filler          good   \n",
       "6605        17    1     9  experiment-filler           bad   \n",
       "9607        22    1     9  experiment-filler           bad   \n",
       "11121       25    2     2  experiment-filler          good   \n",
       "11131       25    2     5  experiment-filler          good   \n",
       "11888       26    3     4  experiment-filler          good   \n",
       "12373       28    3     6  experiment-filler          good   \n",
       "12393       28    3     1  experiment-filler           bad   \n",
       "12438       28    3    20  experiment-filler           bad   \n",
       "12473       28    3     5  experiment-filler           bad   \n",
       "12498       28    3     2  experiment-filler          good   \n",
       "13083       30    2     4  experiment-filler           bad   \n",
       "14889       35    3     9  experiment-filler          good   \n",
       "14919       35    3     4  experiment-filler  intermediate   \n",
       "\n",
       "                                                sentence  judgment    zscore  \\\n",
       "263    There was love one’s true self many men now fr...         6  1.094118   \n",
       "313    The city closed your favorite pizzeria due to ...         2 -1.000603   \n",
       "948    I had every hour many days swum 100 meters in ...         4  0.457816   \n",
       "1213   Nobody liked the old garden because it was unk...         2 -1.029646   \n",
       "1223   What did Sarah eat that Bob likes a pizza more...         6  1.276761   \n",
       "1243   Somebody mentioned that you liked wine so I br...         2 -1.029646   \n",
       "1358   To he likes that linguistics would like to the...         6  1.276761   \n",
       "1378   The famous siblings together became we 17 peop...         6  1.276761   \n",
       "1598   There was an enormous painting of a Teddy bear...         1 -1.096681   \n",
       "1889   The students described which counselor the dep...         7  1.588985   \n",
       "1899   Peeled the banana in the kitchen the waitress ...         6  1.043076   \n",
       "1944   Somebody mentioned that you liked wine so I br...         2 -1.140560   \n",
       "1959   My personal chef made me a veggie burger when ...         1 -1.686468   \n",
       "1979   Everyone takes advantage of the few sunny days...         2 -1.140560   \n",
       "2029   To he likes that linguistics would like to the...         7  1.588985   \n",
       "2044   You wore festive colors to work on Independenc...         3 -0.594651   \n",
       "2049   There was love one’s true self many men now fr...         6  1.043076   \n",
       "2074   The married couple who got into a fight starte...         3 -0.594651   \n",
       "2094   The famous siblings together became we 17 peop...         7  1.588985   \n",
       "2129   The city closed your favorite pizzeria due to ...         2 -1.140560   \n",
       "2427   The cookies with dried rose petals and organic...         2 -0.739069   \n",
       "3398   The students described which counselor the dep...         6  1.087564   \n",
       "3493   Everyone takes advantage of the few sunny days...         2 -1.087564   \n",
       "3578   What did the talented chef cook ravioli with s...         6  1.087564   \n",
       "3748   The lawyer the judge determined the defendant ...         6  0.161257   \n",
       "4163   Everyone's favorite boy band from the 90s will...         3  0.229888   \n",
       "4785   What did Sarah eat that Bob likes a pizza more...         7  1.391794   \n",
       "4960   Peeled the banana in the kitchen the waitress ...         5  0.435469   \n",
       "5538   Nobody liked the old garden because it was unk...         2 -1.490599   \n",
       "5648   What did Sarah eat that Bob likes a pizza more...         6  0.609364   \n",
       "6250   You wore festive colors to work on Independenc...         2 -0.377964   \n",
       "6605   I had every hour many days swum 100 meters in ...         4       NaN   \n",
       "9607   I had every hour many days swum 100 meters in ...         4       NaN   \n",
       "11121  The married couple who got into a fight starte...         3 -0.243303   \n",
       "11131  My personal chef made me a veggie burger when ...         2 -0.747933   \n",
       "11888  Everyone's favorite boy band from the 90s will...         3 -0.640662   \n",
       "12373  Somebody mentioned that you liked wine so I br...         2 -0.937167   \n",
       "12393  To he likes that linguistics would like to the...         6  0.920580   \n",
       "12438  The lawyer the judge determined the defendant ...         7  1.385016   \n",
       "12473  What will you buys when you could went on a va...         7  1.385016   \n",
       "12498  The married couple who got into a fight starte...         2 -0.937167   \n",
       "13083  Peeled the banana in the kitchen the waitress ...         5  0.284768   \n",
       "14889  You wore festive colors to work on Independenc...         2 -0.676631   \n",
       "14919  The cookies with dried rose petals and organic...         1 -1.168725   \n",
       "\n",
       "       sent_mean  sent_zscore  \n",
       "263     2.200000     2.694944  \n",
       "313     6.342857    -3.272302  \n",
       "948     1.885714     2.067888  \n",
       "1213    6.200000    -3.023687  \n",
       "1223    2.657143     2.055947  \n",
       "1243    6.057143    -2.614302  \n",
       "1358    2.028571     2.351605  \n",
       "1378    2.000000     2.748737  \n",
       "1598    5.857143    -3.242640  \n",
       "1889    2.600000     2.974443  \n",
       "1899    2.142857     2.805097  \n",
       "1944    6.057143    -2.614302  \n",
       "1959    6.028571    -3.258503  \n",
       "1979    5.885714    -2.572998  \n",
       "2029    2.028571     2.943735  \n",
       "2044    6.114286    -2.089302  \n",
       "2049    2.200000     2.694944  \n",
       "2074    5.857143    -2.077850  \n",
       "2094    2.000000     3.435921  \n",
       "2129    6.342857    -3.272302  \n",
       "2427    5.600000    -2.312482  \n",
       "3398    2.600000     2.298433  \n",
       "3493    5.885714    -2.572998  \n",
       "3578    2.771429     2.280073  \n",
       "3748    2.800000     2.339704  \n",
       "4163    5.942857    -2.382236  \n",
       "4785    2.657143     2.670974  \n",
       "4960    2.142857     2.077850  \n",
       "5538    6.200000    -3.023687  \n",
       "5648    2.657143     2.055947  \n",
       "6250    6.114286    -2.760179  \n",
       "6605    1.885714     2.067888  \n",
       "9607    1.885714     2.067888  \n",
       "11121   5.857143    -2.077850  \n",
       "11131   6.028571    -2.610506  \n",
       "11888   5.942857    -2.382236  \n",
       "12373   6.057143    -2.614302  \n",
       "12393   2.028571     2.351605  \n",
       "12438   2.800000     3.070861  \n",
       "12473   2.314286     3.230220  \n",
       "12498   5.857143    -2.805097  \n",
       "13083   2.142857     2.077850  \n",
       "14889   6.114286    -2.760179  \n",
       "14919   5.600000    -2.954838  "
      ]
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Only include sentences that are in the above dataset ^\n",
    "df_check = df_check[df_check[\"sentence\"].isin(df_sort[\"sentence\"])]\n",
    "\n",
    "# Only include judgments that were significantly different from the mean\n",
    "df_check = df_check[abs(df_check[\"sent_zscore\"]) > 2]\n",
    "df_check"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "29cb1bb0",
   "metadata": {},
   "source": [
    "### Part 3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "a0876e0a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[6, 4, 28]"
      ]
     },
     "execution_count": 27,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Find which subject(s) to remove\n",
    "should_remove = (df_check[\"subj_id\"].value_counts() >= 5)\n",
    "remove_subj = should_remove.index[should_remove]\n",
    "remove_subj.to_list()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "dd8ff2fa",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Remove these subjects\n",
    "df_ju = df_ju[~df_ju[\"subj_id\"].isin(remove_subj)]\n",
    "df_bg = df_bg[~df_bg[\"subj_id\"].isin(remove_subj)]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "68681973",
   "metadata": {},
   "source": [
    "### Notes\n",
    "\n",
    "Looking at the judgments from subject 4, 6, and 28, it seems like this attention check did a pretty good job of finding subjects who might not have been paying attention. However, it misses subject 17 and 22 (the two subjects who answered \"4\" every time). This is because there is only one filler sentence for which a judgment of 4 is significantly different from the mean:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "id": "d89d8d8a",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>subj_id</th>\n",
       "      <th>list</th>\n",
       "      <th>item</th>\n",
       "      <th>trial_type</th>\n",
       "      <th>condition</th>\n",
       "      <th>sentence</th>\n",
       "      <th>judgment</th>\n",
       "      <th>zscore</th>\n",
       "      <th>sent_mean</th>\n",
       "      <th>sent_zscore</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>948</th>\n",
       "      <td>3</td>\n",
       "      <td>2</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>4</td>\n",
       "      <td>0.457816</td>\n",
       "      <td>1.885714</td>\n",
       "      <td>2.067888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6605</th>\n",
       "      <td>17</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.885714</td>\n",
       "      <td>2.067888</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9607</th>\n",
       "      <td>22</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>1.885714</td>\n",
       "      <td>2.067888</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "      subj_id list  item         trial_type condition  \\\n",
       "948         3    2     9  experiment-filler       bad   \n",
       "6605       17    1     9  experiment-filler       bad   \n",
       "9607       22    1     9  experiment-filler       bad   \n",
       "\n",
       "                                               sentence  judgment    zscore  \\\n",
       "948   I had every hour many days swum 100 meters in ...         4  0.457816   \n",
       "6605  I had every hour many days swum 100 meters in ...         4       NaN   \n",
       "9607  I had every hour many days swum 100 meters in ...         4       NaN   \n",
       "\n",
       "      sent_mean  sent_zscore  \n",
       "948    1.885714     2.067888  \n",
       "6605   1.885714     2.067888  \n",
       "9607   1.885714     2.067888  "
      ]
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_check[df_check[\"judgment\"] == 4]"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e60e1c36",
   "metadata": {},
   "source": [
    "Is it worth adding another kind of attention check in order to remove subjects like 17 and 22 from the dataset? Obviously, I'll definitely want to make sure my code is correct first, so let me know if you see any errors or get different results (or if you have a more efficient/organized process)."
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2d02270a",
   "metadata": {},
   "source": [
    "## Step 10: Export the dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "id": "2d32b228",
   "metadata": {},
   "outputs": [],
   "source": [
    "# index=False: the ugly index column on the left will go away.\n",
    "df_bg.to_csv(\"background_data_clean.csv\", index=False)\n",
    "df_ju.to_csv(\"judgment_data_clean.csv\", index=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8f1cf7c2",
   "metadata": {},
   "source": [
    "## Bonus: Find a better way to clean the dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d62d83e9",
   "metadata": {},
   "source": [
    "Maho pointed out that the index column at the start of each entry could be used to find the corresponding columns. That way, we wouldn't need to manually figure out which column is which, nor would we need to rename them.\n",
    "\n",
    "My strategy is to collect these index columns, clean the dataset the same way as before up to Step 6, and add the index columns back as the new headers before extracting relevant columns. Then continue cleaning, but without renaming the headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "id": "494641c2",
   "metadata": {},
   "outputs": [],
   "source": [
    "df = pd.read_csv(\"results_0221.csv\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "id": "deedebbc",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['# 1. Results reception time.',\n",
       " \"# 2. MD5 hash of participant's IP address.\",\n",
       " '# 3. Controller name.',\n",
       " '# 4. Order number of item.',\n",
       " '# 5. Inner element number.',\n",
       " '# 6. Label.',\n",
       " '# 7. Latin Square Group.',\n",
       " '# 8. PennElementType.',\n",
       " '# 9. PennElementName.',\n",
       " '# 10. Parameter.',\n",
       " '# 11. Value.',\n",
       " '# 12. EventTime.',\n",
       " '# 13. id.',\n",
       " '# 14. lang_comf.',\n",
       " '# 15. comf.',\n",
       " '# 16. lang.',\n",
       " '# 17. parents.',\n",
       " '# 18. age_US.',\n",
       " '# 19. birth.',\n",
       " '# 20. gender.',\n",
       " '# 21. age.',\n",
       " '# 22. Comments.',\n",
       " '# 22. LIST.',\n",
       " '# 23. ITEM.',\n",
       " '# 24. CONDITION.',\n",
       " '# 25. SENTENCE.',\n",
       " '# 26. Comments.']"
      ]
     },
     "execution_count": 32,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# The index columns all start with \"#\" and a number.\n",
    "# Return true if a string fits the above description.\n",
    "def is_index_col(s):\n",
    "    if type(s) == str:\n",
    "        if len(s) > 2:\n",
    "            if \"#\" in s and s[2].isnumeric():\n",
    "                return True\n",
    "    return False\n",
    "\n",
    "index_cols = list(df[df[\"V1\"].apply(is_index_col)][\"V1\"].unique())\n",
    "index_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "id": "0ddd4f56",
   "metadata": {},
   "outputs": [],
   "source": [
    "# For some reason, \"Comments\" and \"LIST\" are both labelled as #22. I'll remove \"comments\" manually.\n",
    "index_cols.remove(\"# 22. Comments.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f9b1c14f",
   "metadata": {},
   "source": [
    "There are 26 columns to match with the 26 \"V\" columns. Now continue cleaning the same way as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "ddfd8149",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 2\n",
    "df = df[(df[\"V6\"] == \"experiment-filler\") | \n",
    "        (df[\"V6\"] == \"experiment-critical\") | \n",
    "        (df[\"V6\"] == \"background\")]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "0b5dcdf1",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 3\n",
    "df[\"V2\"] = df.groupby(\"V2\", sort=False).ngroup() + 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "13508ada",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 4\n",
    "first_entry_ts = df.drop_duplicates(subset=\"V2\", keep=\"first\")[\"V1\"].to_list()\n",
    "df = df[df[\"V1\"].isin(first_entry_ts)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "034764a8",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 5\n",
    "df_ju = df[(df[\"V6\"] == \"experiment-filler\") | \n",
    "             (df[\"V6\"] == \"experiment-critical\")]\n",
    "df_bg = df[df[\"V6\"] == \"background\"]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "5dfd1790",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Replace columns\n",
    "df_ju.columns = index_cols\n",
    "df_bg.columns = index_cols"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 39,
   "id": "8b310e23",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th># 1. Results reception time.</th>\n",
       "      <th># 2. MD5 hash of participant's IP address.</th>\n",
       "      <th># 3. Controller name.</th>\n",
       "      <th># 4. Order number of item.</th>\n",
       "      <th># 5. Inner element number.</th>\n",
       "      <th># 6. Label.</th>\n",
       "      <th># 7. Latin Square Group.</th>\n",
       "      <th># 8. PennElementType.</th>\n",
       "      <th># 9. PennElementName.</th>\n",
       "      <th># 10. Parameter.</th>\n",
       "      <th>...</th>\n",
       "      <th># 17. parents.</th>\n",
       "      <th># 18. age_US.</th>\n",
       "      <th># 19. birth.</th>\n",
       "      <th># 20. gender.</th>\n",
       "      <th># 21. age.</th>\n",
       "      <th># 22. LIST.</th>\n",
       "      <th># 23. ITEM.</th>\n",
       "      <th># 24. CONDITION.</th>\n",
       "      <th># 25. SENTENCE.</th>\n",
       "      <th># 26. Comments.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>65</th>\n",
       "      <td>1676394513</td>\n",
       "      <td>1</td>\n",
       "      <td>PennController</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PennController</td>\n",
       "      <td>33</td>\n",
       "      <td>_Trial_</td>\n",
       "      <td>...</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>66</th>\n",
       "      <td>1676394513</td>\n",
       "      <td>1</td>\n",
       "      <td>PennController</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PennController</td>\n",
       "      <td>33</td>\n",
       "      <td>_Header_</td>\n",
       "      <td>...</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>67</th>\n",
       "      <td>1676394513</td>\n",
       "      <td>1</td>\n",
       "      <td>PennController</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PennController</td>\n",
       "      <td>33</td>\n",
       "      <td>_Header_</td>\n",
       "      <td>...</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>1676394513</td>\n",
       "      <td>1</td>\n",
       "      <td>PennController</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Scale</td>\n",
       "      <td>judgment</td>\n",
       "      <td>Choice</td>\n",
       "      <td>...</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>69</th>\n",
       "      <td>1676394513</td>\n",
       "      <td>1</td>\n",
       "      <td>PennController</td>\n",
       "      <td>32.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PennController</td>\n",
       "      <td>33</td>\n",
       "      <td>_Trial_</td>\n",
       "      <td>...</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>undefined</td>\n",
       "      <td>1</td>\n",
       "      <td>9.0</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   # 1. Results reception time.  # 2. MD5 hash of participant's IP address.  \\\n",
       "65                   1676394513                                           1   \n",
       "66                   1676394513                                           1   \n",
       "67                   1676394513                                           1   \n",
       "68                   1676394513                                           1   \n",
       "69                   1676394513                                           1   \n",
       "\n",
       "   # 3. Controller name.  # 4. Order number of item.  \\\n",
       "65        PennController                        32.0   \n",
       "66        PennController                        32.0   \n",
       "67        PennController                        32.0   \n",
       "68        PennController                        32.0   \n",
       "69        PennController                        32.0   \n",
       "\n",
       "    # 5. Inner element number.        # 6. Label.  # 7. Latin Square Group.  \\\n",
       "65                         0.0  experiment-filler                       NaN   \n",
       "66                         0.0  experiment-filler                       NaN   \n",
       "67                         0.0  experiment-filler                       NaN   \n",
       "68                         0.0  experiment-filler                       NaN   \n",
       "69                         0.0  experiment-filler                       NaN   \n",
       "\n",
       "   # 8. PennElementType. # 9. PennElementName. # 10. Parameter.  ...  \\\n",
       "65        PennController                    33          _Trial_  ...   \n",
       "66        PennController                    33         _Header_  ...   \n",
       "67        PennController                    33         _Header_  ...   \n",
       "68                 Scale              judgment           Choice  ...   \n",
       "69        PennController                    33          _Trial_  ...   \n",
       "\n",
       "   # 17. parents.  # 18. age_US. # 19. birth. # 20. gender. # 21. age.  \\\n",
       "65      undefined      undefined    undefined     undefined  undefined   \n",
       "66      undefined      undefined    undefined     undefined  undefined   \n",
       "67      undefined      undefined    undefined     undefined  undefined   \n",
       "68      undefined      undefined    undefined     undefined  undefined   \n",
       "69      undefined      undefined    undefined     undefined  undefined   \n",
       "\n",
       "   # 22. LIST. # 23. ITEM. # 24. CONDITION.  \\\n",
       "65           1         9.0              bad   \n",
       "66           1         9.0              bad   \n",
       "67           1         9.0              bad   \n",
       "68           1         9.0              bad   \n",
       "69           1         9.0              bad   \n",
       "\n",
       "                                      # 25. SENTENCE. # 26. Comments.  \n",
       "65  I had every hour many days swum 100 meters in ...             NaN  \n",
       "66  I had every hour many days swum 100 meters in ...             NaN  \n",
       "67  I had every hour many days swum 100 meters in ...             NaN  \n",
       "68  I had every hour many days swum 100 meters in ...             NaN  \n",
       "69  I had every hour many days swum 100 meters in ...             NaN  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 39,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_ju.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "440cd8c7",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th># 1. Results reception time.</th>\n",
       "      <th># 2. MD5 hash of participant's IP address.</th>\n",
       "      <th># 3. Controller name.</th>\n",
       "      <th># 4. Order number of item.</th>\n",
       "      <th># 5. Inner element number.</th>\n",
       "      <th># 6. Label.</th>\n",
       "      <th># 7. Latin Square Group.</th>\n",
       "      <th># 8. PennElementType.</th>\n",
       "      <th># 9. PennElementName.</th>\n",
       "      <th># 10. Parameter.</th>\n",
       "      <th>...</th>\n",
       "      <th># 17. parents.</th>\n",
       "      <th># 18. age_US.</th>\n",
       "      <th># 19. birth.</th>\n",
       "      <th># 20. gender.</th>\n",
       "      <th># 21. age.</th>\n",
       "      <th># 22. LIST.</th>\n",
       "      <th># 23. ITEM.</th>\n",
       "      <th># 24. CONDITION.</th>\n",
       "      <th># 25. SENTENCE.</th>\n",
       "      <th># 26. Comments.</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>346</th>\n",
       "      <td>1676394513</td>\n",
       "      <td>1</td>\n",
       "      <td>PennController</td>\n",
       "      <td>64.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>background</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PennController</td>\n",
       "      <td>2</td>\n",
       "      <td>_Trial_</td>\n",
       "      <td>...</td>\n",
       "      <td>English</td>\n",
       "      <td>NaN</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>Female</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>347</th>\n",
       "      <td>1676394513</td>\n",
       "      <td>1</td>\n",
       "      <td>PennController</td>\n",
       "      <td>64.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>background</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PennController</td>\n",
       "      <td>2</td>\n",
       "      <td>_Header_</td>\n",
       "      <td>...</td>\n",
       "      <td>English</td>\n",
       "      <td>NaN</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>Female</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>348</th>\n",
       "      <td>1676394513</td>\n",
       "      <td>1</td>\n",
       "      <td>PennController</td>\n",
       "      <td>64.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>background</td>\n",
       "      <td>NaN</td>\n",
       "      <td>PennController</td>\n",
       "      <td>2</td>\n",
       "      <td>_Header_</td>\n",
       "      <td>...</td>\n",
       "      <td>English</td>\n",
       "      <td>NaN</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>Female</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>349</th>\n",
       "      <td>1676394513</td>\n",
       "      <td>1</td>\n",
       "      <td>PennController</td>\n",
       "      <td>64.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>background</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TextInput</td>\n",
       "      <td>input_age</td>\n",
       "      <td>Final</td>\n",
       "      <td>...</td>\n",
       "      <td>English</td>\n",
       "      <td>NaN</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>Female</td>\n",
       "      <td>20</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>350</th>\n",
       "      <td>1676394513</td>\n",
       "      <td>1</td>\n",
       "      <td>PennController</td>\n",
       "      <td>64.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>background</td>\n",
       "      <td>NaN</td>\n",
       "      <td>TextInput</td>\n",
       "      <td>input_age</td>\n",
       "      <td>First</td>\n",
       "      <td>...</td>\n",
       "      <td>English</td>\n",
       "      <td>NaN</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>Female</td>\n",
       "      <td>20</td>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 26 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "    # 1. Results reception time.  # 2. MD5 hash of participant's IP address.  \\\n",
       "346                   1676394513                                           1   \n",
       "347                   1676394513                                           1   \n",
       "348                   1676394513                                           1   \n",
       "349                   1676394513                                           1   \n",
       "350                   1676394513                                           1   \n",
       "\n",
       "    # 3. Controller name.  # 4. Order number of item.  \\\n",
       "346        PennController                        64.0   \n",
       "347        PennController                        64.0   \n",
       "348        PennController                        64.0   \n",
       "349        PennController                        64.0   \n",
       "350        PennController                        64.0   \n",
       "\n",
       "     # 5. Inner element number. # 6. Label.  # 7. Latin Square Group.  \\\n",
       "346                         0.0  background                       NaN   \n",
       "347                         0.0  background                       NaN   \n",
       "348                         0.0  background                       NaN   \n",
       "349                         0.0  background                       NaN   \n",
       "350                         0.0  background                       NaN   \n",
       "\n",
       "    # 8. PennElementType. # 9. PennElementName. # 10. Parameter.  ...  \\\n",
       "346        PennController                     2          _Trial_  ...   \n",
       "347        PennController                     2         _Header_  ...   \n",
       "348        PennController                     2         _Header_  ...   \n",
       "349             TextInput             input_age            Final  ...   \n",
       "350             TextInput             input_age            First  ...   \n",
       "\n",
       "    # 17. parents.  # 18. age_US. # 19. birth. # 20. gender. # 21. age.  \\\n",
       "346        English            NaN  In the U.S.        Female         20   \n",
       "347        English            NaN  In the U.S.        Female         20   \n",
       "348        English            NaN  In the U.S.        Female         20   \n",
       "349        English            NaN  In the U.S.        Female         20   \n",
       "350        English            NaN  In the U.S.        Female         20   \n",
       "\n",
       "    # 22. LIST. # 23. ITEM. # 24. CONDITION. # 25. SENTENCE. # 26. Comments.  \n",
       "346         NaN         NaN              NaN             NaN             NaN  \n",
       "347         NaN         NaN              NaN             NaN             NaN  \n",
       "348         NaN         NaN              NaN             NaN             NaN  \n",
       "349         NaN         NaN              NaN             NaN             NaN  \n",
       "350           2         NaN              NaN             NaN             NaN  \n",
       "\n",
       "[5 rows x 26 columns]"
      ]
     },
     "execution_count": 40,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df_bg.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "94fd8978",
   "metadata": {},
   "source": [
    "Looks like it worked! All the columns seem to line up. It might have been better to rename V2 as \"subj_id,\" but other than that, this seems to be helpful. Now continue cleaning as before:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 41,
   "id": "2aa47589",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Continuing Step 6\n",
    "c_time = \"# 1. Results reception time.\"\n",
    "timestamps = df_bg[c_time].drop_duplicates().reset_index(drop=True)\n",
    "submission_times = timestamps.apply(lambda s: datetime.datetime.utcfromtimestamp(int(s)))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "207e53d3",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th># 9. PennElementName.</th>\n",
       "      <th># 2. MD5 hash of participant's IP address.</th>\n",
       "      <th>input_US_age</th>\n",
       "      <th>input_age</th>\n",
       "      <th>input_birth</th>\n",
       "      <th>input_comf</th>\n",
       "      <th>input_gender</th>\n",
       "      <th>input_lang</th>\n",
       "      <th>input_lang_comf</th>\n",
       "      <th>input_parent</th>\n",
       "      <th>submission_time</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "      <td>2023-02-14 17:08:33</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "      <td>2023-02-14 17:43:31</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>3</td>\n",
       "      <td>NaN</td>\n",
       "      <td>22</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "      <td>2023-02-14 18:52:07</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>4</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>English</td>\n",
       "      <td>2023-02-14 18:56:30</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>5</td>\n",
       "      <td>NaN</td>\n",
       "      <td>20</td>\n",
       "      <td>In the U.S.</td>\n",
       "      <td>English</td>\n",
       "      <td>Female</td>\n",
       "      <td>Vietnamese</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Others</td>\n",
       "      <td>2023-02-14 21:13:03</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "# 9. PennElementName.  # 2. MD5 hash of participant's IP address.  \\\n",
       "0                                                               1   \n",
       "1                                                               2   \n",
       "2                                                               3   \n",
       "3                                                               4   \n",
       "4                                                               5   \n",
       "\n",
       "# 9. PennElementName. input_US_age input_age  input_birth input_comf  \\\n",
       "0                              NaN        20  In the U.S.    English   \n",
       "1                              NaN        22  In the U.S.    English   \n",
       "2                              NaN        22  In the U.S.    English   \n",
       "3                              NaN        20  In the U.S.    English   \n",
       "4                              NaN        20  In the U.S.    English   \n",
       "\n",
       "# 9. PennElementName. input_gender  input_lang input_lang_comf input_parent  \\\n",
       "0                           Female         NaN             NaN      English   \n",
       "1                           Female         NaN             NaN      English   \n",
       "2                           Female         NaN             NaN      English   \n",
       "3                           Female         NaN             NaN      English   \n",
       "4                           Female  Vietnamese             NaN       Others   \n",
       "\n",
       "# 9. PennElementName.     submission_time  \n",
       "0                     2023-02-14 17:08:33  \n",
       "1                     2023-02-14 17:43:31  \n",
       "2                     2023-02-14 18:52:07  \n",
       "3                     2023-02-14 18:56:30  \n",
       "4                     2023-02-14 21:13:03  "
      ]
     },
     "execution_count": 42,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 7\n",
    "c_param = \"# 10. Parameter.\"\n",
    "c_subj = \"# 2. MD5 hash of participant's IP address.\"\n",
    "c_quest = \"# 9. PennElementName.\"\n",
    "c_val = \"# 11. Value.\"\n",
    "\n",
    "df_bg = df_bg[(df_bg[c_param] == \"Final\") | \n",
    "              (df_bg[c_param] == \"Choice\")]\n",
    "df_bg = df_bg[[c_subj, c_quest, c_val]]\n",
    "\n",
    "df_bg = pd.pivot(df_bg, index=c_subj, columns=c_quest, values=c_val).reset_index()\n",
    "df_bg[\"input_lang\"] = df_bg[\"input_lang\"].apply(comma_if_str)\n",
    "df_bg[\"submission_time\"] = submission_times\n",
    "df_bg.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 43,
   "id": "f6779c5d",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 8\n",
    "c_label = \"# 6. Label.\"\n",
    "c_list = \"# 22. LIST.\"\n",
    "c_item = \"# 23. ITEM.\"\n",
    "c_cond = \"# 24. CONDITION.\"\n",
    "c_sent = \"# 25. SENTENCE.\"\n",
    "\n",
    "df_ju = df_ju[(df_ju[c_param] == \"Choice\") | (df_ju[c_param] == \"Selection\")]\n",
    "df_ju = df_ju[[c_subj, c_list, c_item, c_label, c_cond, c_sent, c_val]]\n",
    "df_ju[c_item] = df_ju[c_item].apply(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "id": "c16e0116",
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th># 2. MD5 hash of participant's IP address.</th>\n",
       "      <th># 22. LIST.</th>\n",
       "      <th># 23. ITEM.</th>\n",
       "      <th># 6. Label.</th>\n",
       "      <th># 24. CONDITION.</th>\n",
       "      <th># 25. SENTENCE.</th>\n",
       "      <th># 11. Value.</th>\n",
       "      <th>zscore</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>68</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>9</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>I had every hour many days swum 100 meters in ...</td>\n",
       "      <td>2</td>\n",
       "      <td>-1.000603</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>73</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>8</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>good</td>\n",
       "      <td>Everyone takes advantage of the few sunny days...</td>\n",
       "      <td>6</td>\n",
       "      <td>1.094118</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>78</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>7</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>intermediate</td>\n",
       "      <td>Do you remember the girl who Jack commented on...</td>\n",
       "      <td>5</td>\n",
       "      <td>0.570437</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>83</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>11</td>\n",
       "      <td>experiment-critical</td>\n",
       "      <td>nested_island</td>\n",
       "      <td>I talked about the man that I like the truck t...</td>\n",
       "      <td>3</td>\n",
       "      <td>-0.476923</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>88</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>18</td>\n",
       "      <td>experiment-filler</td>\n",
       "      <td>bad</td>\n",
       "      <td>Who edited the article that we will have acces...</td>\n",
       "      <td>5</td>\n",
       "      <td>0.570437</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "    # 2. MD5 hash of participant's IP address. # 22. LIST.  # 23. ITEM.  \\\n",
       "68                                           1           1            9   \n",
       "73                                           1           1            8   \n",
       "78                                           1           1            7   \n",
       "83                                           1           1           11   \n",
       "88                                           1           1           18   \n",
       "\n",
       "            # 6. Label. # 24. CONDITION.  \\\n",
       "68    experiment-filler              bad   \n",
       "73    experiment-filler             good   \n",
       "78    experiment-filler     intermediate   \n",
       "83  experiment-critical    nested_island   \n",
       "88    experiment-filler              bad   \n",
       "\n",
       "                                      # 25. SENTENCE.  # 11. Value.    zscore  \n",
       "68  I had every hour many days swum 100 meters in ...             2 -1.000603  \n",
       "73  Everyone takes advantage of the few sunny days...             6  1.094118  \n",
       "78  Do you remember the girl who Jack commented on...             5  0.570437  \n",
       "83  I talked about the man that I like the truck t...             3 -0.476923  \n",
       "88  Who edited the article that we will have acces...             5  0.570437  "
      ]
     },
     "execution_count": 44,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# Step 9\n",
    "df_ju[c_val] = df_ju[c_val].apply(int)\n",
    "df_ju[\"zscore\"] = df_ju.groupby(c_subj)[c_val].apply(lambda x: zscore(x, ddof=1))\n",
    "df_ju.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "80dc848b",
   "metadata": {},
   "outputs": [],
   "source": [
    "# Step 10\n",
    "#df_bg.to_csv(\"background_data_clean_bonus.csv\", index=False)\n",
    "#df_ju.to_csv(\"judgment_data_clean_bonus.csv\", index=False)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2b414802",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}