1_Data_preprocessing.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# TQIP Data Preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This notebook illustrates how we  preprocessed the data for the paper: \"Assesing the Utility of Deep Neural Networks in Dynamic Risk Prediction After Trauma\"\n",
    "\n",
    "The TQIP database can be requested through the American College of Surgeons website at\n",
    "https://facs.org"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Notebook set-up"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import pre-installed packages: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import sys\n",
    "import fastai\n",
    "import pandas as pd\n",
    "from pathlib import Path\n",
    "from fastai.tabular.all import *\n",
    "from platform import python_version "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Package versions: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Python version: 3.8.3\n",
      "Pandas version: 1.1.4\n",
      "Pytorch version: 1.6.0\n",
      "Fastai version: 2.0.11\n"
     ]
    }
   ],
   "source": [
    "print(\"Python version: \" + python_version())\n",
    "print(\"Pandas version: \" + pd.__version__)\n",
    "print(\"Pytorch version: \" + torch.__version__)\n",
    "print(\"Fastai version: \" + fastai.__version__)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Set seed for reproducable results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "seed = 42"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Disable warnings for chained assignments:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pd.options.mode.chained_assignment=None"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a path to the TQIP data folder:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = Path('E:\\Data\\TQIP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Input variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the 2017 TQIP database into a dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP = pd.read_csv(data/'2017/PUF_TRAUMA.csv', low_memory=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Ordinal columns"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Preprocess columns that contain strings with a natural order:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal_colums = ['EMSGCSEYE', 'EMSGCSVERBAL', 'GCSEYE', 'GCSVERBAL', 'GCSMOTOR', 'VERIFICATIONLEVEL', \n",
    "                  'PEDIATRICVERIFICATIONLEVEL', 'STATEDESIGNATION', 'STATEPEDIATRICDESIGNATION', 'BEDSIZE']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Function that changes colums to categories:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "def ordinal(column, sizes):\n",
    "    TQIP[column] = TQIP[column].astype('category')\n",
    "    return TQIP[column].cat.set_categories(sizes, ordered=True, inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('EMSGCSEYE', ('Opens eyes spontaneously','Opens eyes in response to verbal stimulation',\n",
    "                      'Opens eyes in response to painful stimulation', 'No eye movement when assessed'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('EMSGCSVERBAL', ('Smiles, oriented to sounds, follows objects, interacts (P) | Oriented (A)',\n",
    "                      'Cries but is consolable, inappropriate interactions (P) | Confused (A)',\n",
    "                      'Inconsistently consolable, moaning (P) | Inappropriate words (A)',\n",
    "                      'Inconsolable, agitated (P) | Incomprehensible sounds (A)', \n",
    "                      'No vocal response (P) | No verbal response (A)'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('EMSGCSMOTOR', ('Appropriate response to stimulation (P) | Obeys commands (A)', 'Localizing pain', \n",
    "                        'Withdrawal from pain', 'Flexion to pain', 'Extension to pain', 'No motor response'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('GCSEYE', ('Opens eyes spontaneously','Opens eyes in response to verbal stimulation',\n",
    "                   'Opens eyes in response to painful stimulation', 'No eye movement when assessed'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('GCSVERBAL', ('Smiles, oriented to sounds, follows objects, interacts (P) | Oriented (A)',\n",
    "                      'Cries but is consolable, inappropriate interactions (P) | Confused (A)',\n",
    "                      'Inconsistently consolable, moaning (P) | Inappropriate words (A)',\n",
    "                      'Inconsolable, agitated (P) | Incomprehensible sounds (A)', \n",
    "                      'No vocal response (P) | No verbal response (A)'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('GCSMOTOR', ('Appropriate response to stimulation (P) | Obeys commands (A)', 'Localizing pain', 'Withdrawal from pain',\n",
    "                      'Flexion to pain', 'Extension to pain', 'No motor response'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('VERIFICATIONLEVEL', ('I - Level I Trauma Center', 'II - Level II Trauma Center', 'III - Level III Trauma Center'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('PEDIATRICVERIFICATIONLEVEL', ('I - Level I Pediatric Trauma Center', 'II - Level II Pediatric Trauma Center'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('STATEDESIGNATION', ('I', 'II', 'III', 'IV', 'Other', 'Not applicable'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('STATEPEDIATRICDESIGNATION', ('I', 'II', 'III', 'IV', 'Other', 'Not Applicable'))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ordinal('BEDSIZE', ('<= 200', '201-400','401-600', '> 600'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Check that all columns have been coded correctly: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "EMSGCSEYE string value counts == category value counts: True\n",
      "EMSGCSVERBAL string value counts == category value counts: True\n",
      "GCSEYE string value counts == category value counts: True\n",
      "GCSVERBAL string value counts == category value counts: True\n",
      "GCSMOTOR string value counts == category value counts: True\n",
      "VERIFICATIONLEVEL string value counts == category value counts: True\n",
      "PEDIATRICVERIFICATIONLEVEL string value counts == category value counts: True\n",
      "STATEDESIGNATION string value counts == category value counts: True\n",
      "STATEPEDIATRICDESIGNATION string value counts == category value counts: True\n",
      "BEDSIZE string value counts == category value counts: True\n"
     ]
    }
   ],
   "source": [
    "for i in ordinal_colums: \n",
    "    globals()[i + '_str_vc'] = TQIP[i].value_counts(dropna=False).to_list()\n",
    "    globals()[i + '_cat_vc'] = TQIP[i].cat.codes.value_counts(dropna=False).tolist()\n",
    "    print(i, 'string value counts == category value counts:', globals()[i + '_str_vc'] == globals()[i + '_cat_vc'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Filter input variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recode input variables that were **not** collected within the first day of hospitalization:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.loc[TQIP.VTEProphylaxisDays != 1, \"VTEPROPHYLAXISTYPE\"] = 'None'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.loc[TQIP.HMRRHGCTRLSURGDays != 1, \"HMRRHGCTRLSURGTYPE\"] = 'None'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.loc[TQIP.EDDAYS != 1, \"EDDISCHARGEDISPOSITION\"] = 'None'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the dataframe in feather format: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.to_feather(data/'feather/2017_TQIP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### The severity of each injured body region"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this section, we will get the severity of each injured body region from the first- and the last numbers of the AIS code."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import AIS diagnoses:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ais_df = pd.read_csv(data/'2017/PUF_AISDIAGNOSIS.csv', low_memory=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a list with AIS codes from 1-9 (body_reg_num) and a list with the corresponding body regions (body_reg):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "body_reg_num =  [str(x) for x in range(1,10)]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "body_reg = ['HEAD','FACE','NECK','THORAX','ABDOMEN','SPINE','UPPER_EXT','LOWER_EXT','UNSPEC']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Change AIS codes to strings:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ais_df['AISSeverity'] = ais_df['AISSeverity'].astype('Int32').astype(str)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ais_df['AISPREDOT'] = ais_df['AISPREDOT'].astype('Int32').astype(str)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a column for each injured body region, and fill in the severity of that that injury:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for x, y in zip(body_reg_num, body_reg):    \n",
    "    ais_df.loc[ais_df.loc[(ais_df['AISPREDOT'].str.startswith(x))].index,y] = ais_df.loc[ais_df.loc[\n",
    "        (ais_df['AISPREDOT'].str.startswith(x))].index,'AISSeverity']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Drop redundant columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ais_df=ais_df.drop(['AISPREDOT', 'AISPREDOT_BIU', 'AISSeverity', 'AISSeverity_BIU', 'AISVersion'], axis=1)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In the resulting dataframe, each row represents one injury, and each patient may have multiple injuries"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(3434829, 10)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "ais_df.shape"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert the 'long' dataframe to multiple 'wide' dataframes - \n",
    "one for each body region:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in body_reg:\n",
    "    globals()[i + '_Inj'] = pd.crosstab(ais_df['inc_key'], ais_df[i]).add_prefix(i + '_')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We now have one dataframe for each body region. The columns represent the severity of the injury, and the values represent the number of injuries:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "data_frames = HEAD_Inj, FACE_Inj, NECK_Inj, THORAX_Inj, ABDOMEN_Inj, SPINE_Inj, UPPER_EXT_Inj, LOWER_EXT_Inj, UNSPEC_Inj"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merge the 'wide' dataframes for each body region:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "code_folding": []
   },
   "outputs": [],
   "source": [
    "ais_df = reduce(lambda  left,right: pd.merge(left,right,on=['inc_key'], how='outer'), data_frames)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fill missing values with 0"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ais_df = ais_df.fillna(0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reset the index and save the preprocessed injury characteristics in feather format"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ais_df.reset_index().to_feather(data/'feather/2017_ais')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Merge injury characteristics with the other input variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP = pd.read_feather(data/'feather/2017_TQIP')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "ais_df = pd.read_feather(data/'feather/2017_ais')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Merge dataframes on the patient identifier column (inc_key):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP = TQIP.merge(ais_df, on = 'inc_key', how = 'left')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the merged dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.to_feather(data/'feather/2017_TQIP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Output variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the 2017 TQIP dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP = pd.read_feather(data/'feather/2017_TQIP') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Delete cases were length of stay was not recorded:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "997970"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l1 = len(TQIP)\n",
    "l1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP = TQIP[~TQIP.LOSDays.isna()]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Deleted cases: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "18962"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "l1 - len(TQIP)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Statistics for length of stay:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25    2.0\n",
       "0.50    3.0\n",
       "0.75    6.0\n",
       "Name: LOSDays, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP['LOSDays'].quantile(q=(0.25, 0.5, 0.75))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Early trauma deaths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new column for early mortality:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP['DECEASED_early'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Identify all patients who were coded as \"Deceased/expired\" on **ED discharge**, with a length of stay (ED or total) of 1 day or less:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.loc[(TQIP.EDDISCHARGEDISPOSITION == \"Deceased/expired\") & ((TQIP.EDDAYS <= 1.0)|(TQIP.LOSDays <= 1.0)), \"DECEASED_early\"] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Identify all patients who were coded as \"Deceased/expired\" on **hospital discharge**, with a total length of stay of 1 day or less:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.loc[(TQIP.HOSPDISCHARGEDISPOSITION == \"Deceased/Expired\") & (TQIP.LOSDays <= 1.0), \"DECEASED_early\"] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Late trauma deaths"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a new column for late mortality:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP['DECEASED_late'] = 0"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Identify all patients who were coded as \"Deceased/expired\" on ED discharge, with a length of stay (ED or total) of more than 1:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.loc[(TQIP.EDDISCHARGEDISPOSITION == \"Deceased/expired\") & ((TQIP.EDDAYS > 1.0)&(TQIP.LOSDays > 1.0)), \"DECEASED_late\"] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Identify all patients who were coded as \"Deceased/expired\" on hospital discharge, with a total length of stay of more than 1 day:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.loc[(TQIP.HOSPDISCHARGEDISPOSITION == \"Deceased/Expired\") & (TQIP.LOSDays > 1.0), \"DECEASED_late\"] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Post-trauma complications "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a list of post-trauma complications (PTCs):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "PTC_list = ['HC_CLABSI', 'HC_DEEPSSI', 'HC_DVTHROMBOSIS', 'HC_CARDARREST', 'HC_CAUTI', 'HC_EMBOLISM', \n",
    "           'HC_EXTREMITYCS', 'HC_INTUBATION', 'HC_KIDNEY', 'HC_MI', 'HC_ORGANSPACESSI', \n",
    "           'HC_RESPIRATORY', 'HC_SEPSIS', 'HC_STROKECVA', 'HC_SUPERFICIALINCISIONSSI', 'HC_PRESSUREULCER', 'HC_VAPNEUMONIA']"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Convert yes/no answers to 1/0 values:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in PTC_list:\n",
    "    TQIP[str(i)] = TQIP[str(i)].map({'No':0, 'Yes':1})"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reset the index and save the dataframe in feather format:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.reset_index(drop = True).to_feather(data/'feather/2017_TQIP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load and preprocess data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load the 2017 TQIP dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP = pd.read_feather(data/'feather/2017_TQIP') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Select columns used for this project:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_names_short = ['SEX', 'ASIAN', 'PACIFICISLANDER', 'RACEOTHER', 'AMERICANINDIAN', 'BLACK', 'WHITE', 'ETHNICITY', \n",
    "                   'TRANSPORTMODE', 'PREHOSPITALCARDIACARREST', 'TCCGCSLE13', 'TCCSBPLT30', 'TCC10RR29', 'TCCPEN', 'TCCCHEST', \n",
    "                   'TCCLONGBONE', 'TCCCRUSHED', 'TCCAMPUTATION', 'TCCPELVIC', 'TCCSKULLFRACTURE', 'TCCPARALYSIS', \n",
    "                   'TEACHINGSTATUS', 'BEDSIZE', 'HOSPITALTYPE', 'VERIFICATIONLEVEL', 'PEDIATRICVERIFICATIONLEVEL', \n",
    "                   'STATEDESIGNATION', 'STATEPEDIATRICDESIGNATION', 'EMSGCSEYE', 'EMSGCSVERBAL', 'EMSGCSMOTOR', \n",
    "                   'PRIMARYECODEICD10', 'AGEYEARS', 'WEIGHT', 'HEIGHT', 'EMSSBP', 'EMSPULSERATE', 'EMSRESPIRATORYRATE', \n",
    "                   'EMSPULSEOXIMETRY', 'EMSTOTALGCS', 'GCSEYE', 'GCSVERBAL', 'GCSMOTOR', 'GCSQ_SEDATEDPARALYZED', \n",
    "                   'GCSQ_EYEOBSTRUCTION', 'GCSQ_INTUBATED', 'GCSQ_VALID', 'CC_ADHD', 'CC_ALCOHOLISM', 'CC_ANGINAPECTORIS', \n",
    "                   'CC_ANTICOAGULANT', 'CC_BLEEDING', 'CC_CHEMO', 'CC_CIRRHOSIS', 'CC_COPD', 'CC_CVA', 'CC_DEMENTIA', \n",
    "                   'CC_DIABETES', 'CC_DISCANCER', 'CC_FUNCTIONAL', 'CC_CHF', 'CC_HYPERTENSION', 'CC_MI', 'CC_PAD', 'CC_OTHER', \n",
    "                   'CC_MENTALPERSONALITY', 'CC_RENAL', 'CC_SMOKING', 'CC_STEROID', 'CC_SUBSTANCEABUSE', 'HEAD_1', 'HEAD_2', \n",
    "                   'HEAD_3', 'HEAD_4', 'HEAD_5', 'HEAD_6', 'HEAD_9', 'FACE_1', 'FACE_2', 'FACE_3', 'FACE_4', 'FACE_5', 'FACE_9',\n",
    "                   'NECK_1', 'NECK_2', 'NECK_3', 'NECK_4', 'NECK_5', 'NECK_6', 'NECK_9', 'THORAX_1', 'THORAX_2', 'THORAX_3', \n",
    "                   'THORAX_4', 'THORAX_5', 'THORAX_6', 'THORAX_9', 'ABDOMEN_1', 'ABDOMEN_2', 'ABDOMEN_3', 'ABDOMEN_4', \n",
    "                   'ABDOMEN_5', 'ABDOMEN_6', 'ABDOMEN_9', 'SPINE_1', 'SPINE_2', 'SPINE_3', 'SPINE_4', 'SPINE_5', 'SPINE_6', \n",
    "                   'SPINE_9', 'UPPER_EXT_1', 'UPPER_EXT_2', 'UPPER_EXT_3', 'UPPER_EXT_4', 'UPPER_EXT_5', 'UPPER_EXT_6', \n",
    "                   'UPPER_EXT_9', 'LOWER_EXT_1', 'LOWER_EXT_2', 'LOWER_EXT_3', 'LOWER_EXT_4', 'LOWER_EXT_5', 'LOWER_EXT_6', \n",
    "                   'LOWER_EXT_9', 'UNSPEC_1', 'UNSPEC_2', 'UNSPEC_3', 'UNSPEC_4', 'UNSPEC_5', 'UNSPEC_6', 'UNSPEC_9', \n",
    "                   'RESPIRATORYASSISTANCE', 'SUPPLEMENTALOXYGEN', 'EMSRESPONSEMINS', 'EMSSCENEMINS', 'EMSMINS', 'SBP', \n",
    "                   'PULSERATE', 'TEMPERATURE', 'RESPIRATORYRATE', 'PULSEOXIMETRY', 'ISS_05', 'VTEPROPHYLAXISTYPE', \n",
    "                   'HMRRHGCTRLSURGTYPE', 'EDDISCHARGEDISPOSITION', 'HC_CLABSI', 'HC_CAUTI', 'HC_SUPERFICIALINCISIONSSI', \n",
    "                   'HC_DEEPSSI', 'HC_ORGANSPACESSI', 'HC_VAPNEUMONIA', 'HC_SEPSIS', 'HC_DVTHROMBOSIS',  'HC_EMBOLISM', \n",
    "                   'HC_EXTREMITYCS', 'HC_PRESSUREULCER', 'HC_KIDNEY', 'HC_MI', 'HC_CARDARREST', 'HC_STROKECVA', 'HC_INTUBATION',\n",
    "                   'HC_RESPIRATORY', 'DECEASED_early', 'DECEASED_late'] "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "Rename columns:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "col_names_long = ['Gender', 'Race Category: Asian', 'Race Category: Pacific Islander', 'Race Category: Other', \n",
    "                  'Race Category: American Indian', 'Race Category: Black', 'Race Category: White', 'Ethnicity', \n",
    "                  'Transport Mode', 'Pre-Hospital Cardiac Arrest', 'TCC: Glasgow Coma Scale of 13 or less', \n",
    "                  'TCC: Systolic Blood Pressure under 90', 'TCC: Respiratory rate less than 10 or more than 29', \n",
    "                  'TCC: Penetrating Injuries', 'TCC: Chest wall instability or deformity', \n",
    "                  'TCC: Two or more proximal long-bone fractures', 'TCC: Crushed, degloved, mangled, or pulseless extremity', \n",
    "                  'TCC: Amputation proximal to wrist or ankle', 'TCC: Pelvic fracture', 'TCC: Open or depressed skull fracture',\n",
    "                  'TCC: Paralysis',  'Hospital Teaching Status', 'Bed Size', 'Hospital Type', 'ACS Verification Level',\n",
    "                  'Pediatric Verification Level', 'State Designation', 'State Pediatric Designation', 'EMS GCS - Eye', \n",
    "                  'EMS GCS - Verbal', 'EMS GCS - Motor', 'Primary External Cause', 'Age (years)', 'Weight', 'Height', \n",
    "                  'Initial EMS Systolic Blood Pressure', 'Initial EMS Pulse Rate', 'Initial EMS Respiratory Rate', \n",
    "                  'Initial EMS Oxygen Saturation', 'Initial EMS Total GCS', 'GCS - Eye', 'GCS - Verbal', 'GCS -Motor', \n",
    "                  'GCS Assessment Qualifiers: Patient Chemically Sedated or Paralyzed', \n",
    "                  'GCS Assessment Qualifiers: Obstruction to the Patients Eye', 'GCS Assessment Qualifiers: Patient Intubated',\n",
    "                  'GCS Assessment Qualifiers: Valid GCS', \n",
    "                  'Comorbid Condition: Attention Deficit Disorder/Attention Deficit Hyperactivity Disorder (ADD/ADHD)',\n",
    "                  'Comorbid Condition: Alcohol Use Disorder', 'Comorbid Condition: Angina Pectoris', \n",
    "                  'Comorbid Condition: Anticoagulant Therapy', 'Comorbid Condition: Bleeding Disorder', \n",
    "                  'Comorbid Condition: Currently Receiving Chemotherapy for Cancer', \n",
    "                  'Comorbid Condition: Cirrhosis', 'Comorbid Condition: Chronic Obstructive Pulmonary Disease (COPD)', \n",
    "                  'Comorbid Condition: Cerebrovascular Accident (CVA)', 'Comorbid Condition: Dementia', \n",
    "                  'Comorbid Condition: Diabetes Mellitus', 'Comorbid Condition: Disseminated Cancer', \n",
    "                  'Comorbid Condition: Functionally Dependent Health Status', 'Comorbid Condition: Congestive Heart Failure', \n",
    "                  'Comorbid Condition: Hypertension', 'Comorbid Condition: Myocardial Infarction (MI)', \n",
    "                  'Comorbid Condition: Peripheral Arterial Disease (PAD)', 'Comorbid Condition: Other', \n",
    "                  'Comorbid Condition: Mental/Personality Disorder', 'Comorbid Condition: Chronic Renal Failure', \n",
    "                  'Comorbid Condition: Current Smoker', 'Comorbid Condition: Steroid Use', \n",
    "                  'Comorbid Condition: Substance Abuse Disorder', 'Minor Head Injury', 'Moderate Head Injury', \n",
    "                  'Serious Head Injury', 'Severe Head Injury', 'Critical Head Injury', 'Maximum Head Injury', \n",
    "                  'Head Injury (NFS)', 'Minor Face Injury', 'Moderate Face Injury', 'Serious Face Injury', 'Severe Face Injury',\n",
    "                  'Critical Face Injury', 'Face Injury (NFS)', 'Minor Neck Injury', 'Moderate Neck Injury', \n",
    "                  'Serious Neck Injury', 'Severe Neck Injury', 'Critical Neck Injury', 'Maximum Neck Injury', \n",
    "                  'Neck Injury (NFS)' , 'Minor Thoracic Injury', 'Moderate Thoracic Injury', 'Serious Thoracic Injury', \n",
    "                  'Severe Thoracic Injury', 'Critical Thoracic Injury', 'Maximum Thoracic Injury', 'Thoracic Injury (NFS)', \n",
    "                  'Minor Abdominal Injury', 'Moderate Abdominal Injury', 'Serious Abdominal Injury', 'Severe Abdominal Injury', \n",
    "                  'Critical Abdominal Injury', 'Maximum Abdominal Injury', 'Abdominal Injury (NFS)', 'Minor Spine Injury', \n",
    "                  'Moderate Spine Injury', 'Serious Spine Injury', 'Severe Spine Injury', 'Critical Spine Injury', \n",
    "                  'Maximum Spine Injury', 'Spine Injury (NFS)', 'Minor Upper Extremity Injury', \n",
    "                  'Moderate Upper Extremity Injury', 'Serious Upper Extremity Injury', 'Severe Upper Extremity Injury', \n",
    "                  'Critical Upper Extremity Injury', 'Maximum Upper Extremity Injury', 'Upper Extremity Injury (NFS)', \n",
    "                  'Minor Lower Extremity Injury', 'Moderate Lower Extremity Injury', 'Serious Lower Extremity Injury', \n",
    "                  'Severe Lower Extremity Injury', 'Critical Lower Extremity Injury', 'Maximum Lower Extremity Injury', \n",
    "                  'Lower Extremity Injury (NFS)', 'Minor Unspecified Injury', 'Moderate Unspecified Injury', \n",
    "                  'Serious Unspecified Injury', 'Severe Unspecified Injury', 'Critical Unspecified Injury', \n",
    "                  'Maximum Unspecified Injury', 'Unspecified Injury (NFS)', 'Respiratory Assistance', 'Supplemental Oxygen', \n",
    "                  'Time to EMS Response (mins)', 'Time EMS spent at scene (mins)', \n",
    "                  'Time from dispatch to ED/hospital arrival (mins)', 'Initial ED Systolic Blood Pressure', \n",
    "                  'Initial ED Pulse Rate', 'Initial ED Temperature', 'Initial ED Respiratory Rate', \n",
    "                  'Initial ED Oxygen Saturation', 'AIS derived ISS', 'VTE Prophylaxis Type', \n",
    "                  'Type of Surgery for Hemorrhage Control', 'ED Discharge Disposition', \n",
    "                  'Central Line-Associated Bloodstream Infection', 'Catheter-Associated Urinary Tract Infection', \n",
    "                  'Superficial Incisional Surgical Site Infection', 'Deep Surgical Site Infection', \n",
    "                  'Organspace Surgical Site Infection', 'Ventilator-Associated Pneumonia', 'Severe Sepsis', \n",
    "                  'Deep Vein Thrombosis', 'Pulmonary Embolism', 'Extremity Compartment Syndrome', 'Pressure Ulcer', \n",
    "                  'Acute Kidney Injury', 'Myocardial Infarction', 'Cardiac Arrest', 'Stroke', 'Unplanned Intubation',\n",
    "                  'Acute Respiratory Distress Syndrome', 'Early Mortality', 'Late Mortality']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.rename(columns=dict(zip(col_names_short, col_names_long)), inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP.to_feather(data/'feather/2017_TQIP')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "deletable": false,
    "editable": false,
    "run_control": {
     "frozen": true
    }
   },
   "source": [
    "### Test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP = pd.read_feather(data/'feather/2017_TQIP') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a test dataset by identifying all patients that were treated at medium-sized (bed size = 201-400) university hospitals."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_test = TQIP[(TQIP['Bed Size'] == '201-400') & (TQIP['Hospital Teaching Status'] == 'university')]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reset index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_test.reset_index(inplace=True, drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save test dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_test.to_feather(data/'feather/2017_TQIP_test')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Train and validation data"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create a dataframe for training and validatoin: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_train_val = TQIP[~((TQIP['Bed Size'] == '201-400') & (TQIP['Hospital Teaching Status'] == 'university'))]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reset index"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_train_val.reset_index(inplace=True, drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Split the train_val data randomly into a training dataset (80%) and a validation dataset (20%):"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "splits = RandomSplitter(seed=seed)(range_of(TQIP_train_val))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define preprocessingm steps:\n",
    "* *FillMissing = Replace missing values with the median of the group while simultaneously creating a new binary column indicating whether a variable was missing or not*<br>\n",
    "* *Categorify = Turn categorical variables into categories*<br\n",
    "* *Normalize = Normalize continous data by subtraction of the mean and division by the standard deviation*"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "procs = [FillMissing, Categorify, Normalize]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the training dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_train = TQIP_train_val.iloc[splits[0], :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reset index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_train.reset_index(inplace=True, drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save test dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_train.to_feather(data/'feather/2017_TQIP_train')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the validation dataset"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_valid= TQIP_train_val.iloc[splits[1], :]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reset index:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_valid.reset_index(inplace=True, drop=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save test dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP_valid.to_feather(data/'feather/2017_TQIP_valid')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess data for the Pre-Hospital Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define categorical input variables for the Pre-Hospital Model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_names_prehosp = col_names_long[0:32]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define continous input variables for the Pre-Hospital Model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cont_names_prehosp = col_names_long[32:40]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define output variables for the Pre-Hospital Model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_names_prehosp = col_names_long[146:166]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use TabularPandas from Fastai to write a tabular processor for the train_val dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_prehosp = TabularPandas(TQIP_train_val, procs=procs, cat_names = cat_names_prehosp, cont_names = cont_names_prehosp, \n",
    "                           y_names=y_names_prehosp, y_block=MultiCategoryBlock(encoded=True, vocab=y_names_prehosp), \n",
    "                           inplace=True, splits=splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the processed dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(to_prehosp, open('to_prehosp.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use TabularPandas from Fastai to write a tabular processor for the test dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_test_prehosp = TabularPandas(TQIP_test, procs=procs, cat_names = cat_names_prehosp, cont_names = cont_names_prehosp, \n",
    "                                y_names=y_names_prehosp, y_block=MultiCategoryBlock(encoded=True, vocab=y_names_prehosp),\n",
    "                                inplace=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the processed test dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(to_test_prehosp, open('to_prehosp_test.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess data for the ED Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define categorical input features for the ED model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_names_ed = col_names_long[0:32] + col_names_long[40:134]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define continous input features for the ED model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cont_names_ed = col_names_long[32:40] + col_names_long[134:143]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define output variables for the ED Model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_names_ed = col_names_long[146:165]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use TabularPandas from Fastai to write a tabular processor for the train_val dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_ed = TabularPandas(TQIP_train_val, procs=procs, cat_names = cat_names_ed, cont_names = cont_names_ed, \n",
    "                   y_names=y_names_ed, y_block=MultiCategoryBlock(encoded=True, vocab=y_names_ed), splits=splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the processed dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(to_ed, open('to_ED.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use TabularPandas from Fastai to write a tabular processor for the test dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_test_ed = TabularPandas(TQIP_test, procs=procs, cat_names = cat_names_ed, cont_names = cont_names_ed, \n",
    "                        y_names=y_names_ed, y_block=MultiCategoryBlock(encoded=True, vocab=y_names_ed))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the processed test dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(to_test_ed, open('to_ED_test.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocess data without detailed injury characteristics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define categorical input features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_names_ed = col_names_long[0:32] + col_names_long[40:70] + col_names_long[132:134]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define additional continous input features for the ED model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cont_names_ed = col_names_long[32:40] + col_names_long[134:143]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define output variables for the ED Model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_names_ed = col_names_long[146:165]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use TabularPandas from Fastai to write a tabular processor for the train_val dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_ed = TabularPandas(TQIP_train_val, procs=procs, cat_names = cat_names_ed, cont_names = cont_names_ed, \n",
    "                   y_names=y_names_ed, y_block=MultiCategoryBlock(encoded=True, vocab=y_names_ed), splits=splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the processed dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(to_ed, open('to_ED_noinj.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Preprocess data for the In-Hospital Model"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define additional categorical input features for the In-Hospital model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_names_inhosp = col_names_long[0:32] + col_names_long[40:134] + col_names_long[143:146] + col_names_long[163:164]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The In-Hospital Model has the same continous input variables as the ED Model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cont_names_inhosp = col_names_long[32:40] + col_names_long[134:143]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define output variables for the In-Hospital Model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_names_inhosp = col_names_long[146:163] + col_names_long[164:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use TabularPandas from Fastai to write a tabular processor for the train_val dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_inhosp = TabularPandas(TQIP_train_val, procs=procs, cat_names = cat_names_inhosp, cont_names = cont_names_inhosp, \n",
    "                   y_names=y_names_inhosp, y_block=MultiCategoryBlock(encoded=True, vocab=y_names_inhosp), splits=splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the processinhosp dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(to_inhosp, open('to_inhosp.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use TabularPandas from Fastai to write a tabular processor for the test dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_test_inhosp = TabularPandas(TQIP_test, procs=procs, cat_names = cat_names_inhosp, cont_names = cont_names_inhosp, \n",
    "                        y_names=y_names_inhosp, y_block=MultiCategoryBlock(encoded=True, vocab=y_names_inhosp))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the processinhosp test dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(to_test_inhosp, open('to_inhosp_test.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Preprocess data without early mortality and ED discharge disposition"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define categorical input features:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_names_inhosp = col_names_long[0:32] + col_names_long[40:134] + col_names_long[143:145]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define additional continous input features for the ED model:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "cont_names_inhosp = col_names_long[32:40] + col_names_long[134:143]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define output variables for the ED Model: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_names_inhosp = col_names_long[146:163] + col_names_long[164:]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Use TabularPandas from Fastai to write a tabular processor for the train_val dataframe:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "to_inhosp = TabularPandas(TQIP_train_val, procs=procs, cat_names = cat_names_inhosp, cont_names = cont_names_inhosp, \n",
    "                   y_names=y_names_inhosp, y_block=MultiCategoryBlock(encoded=True, vocab=y_names_inhosp), splits=splits)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Save the processed dataframes:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "pickle.dump(to_ed, open('to_inhosp_noearlymort.pkl', 'wb'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistics across the entire dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Import the preprocessed in- and output data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP = pd.read_feather(data/'feather/2017_TQIP') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Gender and race statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(Male      587840\n",
       " Female    391028\n",
       " NaN          140\n",
       " Name: Gender, dtype: int64,\n",
       " Male      0.600445\n",
       " Female    0.399412\n",
       " NaN       0.000143\n",
       " Name: Gender, dtype: float64)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP['Gender'].value_counts(dropna = False), TQIP['Gender'].value_counts(normalize=True, dropna=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Race statistics:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "race_cats = ['Race Category: Asian', 'Race Category: Pacific Islander', 'Race Category: Other', \n",
    "             'Race Category: American Indian', 'Race Category: Black', 'Race Category: White']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Race Category: Asian - Count = 18379 - Percentage = 1.9%\n",
      "Race Category: Pacific Islander - Count = 2580 - Percentage = 0.3%\n",
      "Race Category: Other - Count = 78819 - Percentage = 8.1%\n",
      "Race Category: American Indian - Count = 8837 - Percentage = 0.9%\n",
      "Race Category: Black - Count = 139747 - Percentage = 14.3%\n",
      "Race Category: White - Count = 714967 - Percentage = 73.0%\n"
     ]
    }
   ],
   "source": [
    "for i in race_cats: \n",
    "    counts = len(TQIP.loc[TQIP[i]=='Yes'])\n",
    "    perc = counts/len(TQIP)\n",
    "    print(i, '- Count =', counts, '- Percentage =', \"{:.1%}\".format(perc))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The TQIP data from 2017 is predominately white (71.5%) and male (60.0%). "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Additional statistics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "W01.0XXA    133371\n",
       "V43.52XA     41582\n",
       "W19.XXXA     37256\n",
       "Name: Primary External Cause, dtype: int64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP['Primary External Cause'].value_counts()[:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Age:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25    26.0\n",
       "0.50    49.0\n",
       "0.75    69.0\n",
       "Name: Age (years), dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP['Age (years)'].quantile(q=(0.25, 0.5, 0.75))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0.25     4.0\n",
       "0.50     8.0\n",
       "0.75    10.0\n",
       "Name: AIS derived ISS, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP['AIS derived ISS'].quantile(q=(0.25, 0.5, 0.75))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Statistics for early- and late mortality: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    966357\n",
       "1     12651\n",
       "Name: Early Mortality, dtype: int64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP['Early Mortality'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.980006\n",
       "1    0.019994\n",
       "Name: Late Mortality, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP['Late Mortality'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    959434\n",
       "1     19574\n",
       "Name: Late Mortality, dtype: int64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP['Late Mortality'].value_counts()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.987078\n",
       "1    0.012922\n",
       "Name: Early Mortality, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP['Early Mortality'].value_counts(normalize=True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Statistics for overall morbidity:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "morbidity = col_names_long[146:163]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "TQIP['Morbidity'] = 0\n",
    "for i in morbidity:\n",
    "    TQIP.loc[TQIP[i] == 1, 'Morbidity'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    0.967399\n",
       "1    0.032601\n",
       "Name: Morbidity, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP['Morbidity'].value_counts(normalize = True)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Statistics for the primary external cause of injury:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TQIP\n",
      "\t Unique values: 2,042\n",
      "\t Missing: 1,422 (0.1 %)\n"
     ]
    }
   ],
   "source": [
    "counts = TQIP['Primary External Cause'].nunique()\n",
    "count_na = TQIP['Primary External Cause'].isna().value_counts()[1]\n",
    "per_na = TQIP['Primary External Cause'].isna().value_counts(normalize=True)[1]\n",
    "print('TQIP')\n",
    "print('\\t' + ' Unique values:', f\"{counts:,}\")\n",
    "print('\\t' + ' Missing:', f\"{count_na:,}\", '('+\"{:.1%}\".format(per_na).replace('%', ' %')+')')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Statistics across the train, validate and test datasets"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Load datasets: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "datasets = ['train', 'valid', 'test']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for i in datasets:\n",
    "    globals()['TQIP_' + i] = pd.read_feather(data/('feather/2017_TQIP_' + i))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The length of the three dataframes: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(718375, 179593, 81040)"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "len(TQIP_train), len(TQIP_valid), len(TQIP_test)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Categorical input features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define categorical input variables: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_names = col_names_long[0:32] + col_names_long[40:134] + col_names_long[143:146]  + col_names_long[163:164]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove primary external cause of injury:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_names.remove('Primary External Cause')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Remove the injuries: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "regions = ['Head', 'Face', 'Neck', 'Thoracic', 'Abdominal', 'Spine', 'Upper Extremity', 'Lower Extremity', 'Unspecified Injury']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols = []\n",
    "for i in regions: \n",
    "    cols = cols + ([col for col in TQIP.columns if i in col])\n",
    "cat_names_no_inj = [x for x in cat_names if x not in cols]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an empty dataframe for the statistics of the categorical input variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_df = pd.DataFrame(columns=['Category', 'Train','Valid', 'Test'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fill in the dataframe: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in cat_names_no_inj:\n",
    "    Train_counts = TQIP_train[c].value_counts(dropna=False)\n",
    "    Train_per = Train_counts / Train_counts.sum()\n",
    "    Valid_counts = TQIP_valid[c].value_counts(dropna=False)\n",
    "    Valid_per = Valid_counts / Valid_counts.sum()\n",
    "    Test_counts = TQIP_test[c].value_counts(dropna=False)\n",
    "    Test_per = Test_counts / Test_counts.sum()\n",
    "    fmt_pct = '{:.1%}'.format\n",
    "    fmt_ths = '{:,}'.format\n",
    "    cat_df = cat_df.append(pd.DataFrame({'Category': c, \n",
    "                                         'Train': Train_counts.map(fmt_ths) + ' (' + Train_per.map(fmt_pct) + ')', \n",
    "                                         'Valid': Valid_counts.map(fmt_ths) + ' (' + Valid_per.map(fmt_pct) + ')', \n",
    "                                         'Test': Test_counts.map(fmt_ths) + ' (' + Test_per.map(fmt_pct) + ')'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reset index and save the dataframe as a csv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_df.index = cat_df.index.set_names('variable')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_df.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cat_df.to_csv('cat_df.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Statistics for the primary external cause of injury"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TQIP_train\n",
      "\t Unique values: 1,885\n",
      "\t Missing: 1,044 (0.1 %)\n",
      "TQIP_valid\n",
      "\t Unique values: 1,318\n",
      "\t Missing: 276 (0.2 %)\n",
      "TQIP_test\n",
      "\t Unique values: 1,061\n",
      "\t Missing: 102 (0.1 %)\n"
     ]
    }
   ],
   "source": [
    "for i in datasets:\n",
    "    df = globals()['TQIP_' +i]\n",
    "    counts = df['Primary External Cause'].nunique()\n",
    "    count_na = df['Primary External Cause'].isna().value_counts()[1]\n",
    "    per_na = df['Primary External Cause'].isna().value_counts(normalize=True)[1]\n",
    "    print('TQIP_' + i)\n",
    "    print('\\t' + ' Unique values:', f\"{counts:,}\")\n",
    "    print('\\t' + ' Missing:', f\"{count_na:,}\", '('+\"{:.1%}\".format(per_na).replace('%', ' %')+')')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Statistics for the injury characteristics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for x in datasets: \n",
    "    for i in regions:\n",
    "        var = globals()['TQIP_' + x]\n",
    "        cols = [col for col in var.columns if i in col]\n",
    "        var[i + '_inj'] = 0\n",
    "        for y in cols:\n",
    "            var.loc[(var[y] > 0), i + '_inj'] = 1"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TQIP_train: Statistics for injuries:\n",
      "\t Head_inj:464,723 (64.7 %)\n",
      "\t Head_inj:253,652  (35.3 %)\n",
      "\t Face_inj:533,559 (74.3 %)\n",
      "\t Face_inj:184,816  (25.7 %)\n",
      "\t Neck_inj:698,239 (97.2 %)\n",
      "\t Neck_inj:20,136  (2.8 %)\n",
      "\t Thoracic_inj:544,243 (75.8 %)\n",
      "\t Thoracic_inj:174,132  (24.2 %)\n",
      "\t Abdominal_inj:628,579 (87.5 %)\n",
      "\t Abdominal_inj:89,796  (12.5 %)\n",
      "\t Spine_inj:596,058 (83.0 %)\n",
      "\t Spine_inj:122,317  (17.0 %)\n",
      "\t Upper Extremity_inj:477,082 (66.4 %)\n",
      "\t Upper Extremity_inj:241,293  (33.6 %)\n",
      "\t Lower Extremity_inj:409,575 (57.0 %)\n",
      "\t Lower Extremity_inj:308,800  (43.0 %)\n",
      "\t Unspecified Injury_inj:673,385 (93.7 %)\n",
      "\t Unspecified Injury_inj:44,990  (6.3 %)\n",
      "TQIP_valid: Statistics for injuries:\n",
      "\t Head_inj:116,268 (64.7 %)\n",
      "\t Head_inj:63,325  (35.3 %)\n",
      "\t Face_inj:133,540 (74.4 %)\n",
      "\t Face_inj:46,053  (25.6 %)\n",
      "\t Neck_inj:174,594 (97.2 %)\n",
      "\t Neck_inj:4,999  (2.8 %)\n",
      "\t Thoracic_inj:135,965 (75.7 %)\n",
      "\t Thoracic_inj:43,628  (24.3 %)\n",
      "\t Abdominal_inj:157,417 (87.7 %)\n",
      "\t Abdominal_inj:22,176  (12.3 %)\n",
      "\t Spine_inj:148,882 (82.9 %)\n",
      "\t Spine_inj:30,711  (17.1 %)\n",
      "\t Upper Extremity_inj:119,290 (66.4 %)\n",
      "\t Upper Extremity_inj:60,303  (33.6 %)\n",
      "\t Lower Extremity_inj:102,419 (57.0 %)\n",
      "\t Lower Extremity_inj:77,174  (43.0 %)\n",
      "\t Unspecified Injury_inj:168,316 (93.7 %)\n",
      "\t Unspecified Injury_inj:11,277  (6.3 %)\n",
      "TQIP_test: Statistics for injuries:\n",
      "\t Head_inj:51,919 (64.1 %)\n",
      "\t Head_inj:29,121  (35.9 %)\n",
      "\t Face_inj:59,741 (73.7 %)\n",
      "\t Face_inj:21,299  (26.3 %)\n",
      "\t Neck_inj:78,662 (97.1 %)\n",
      "\t Neck_inj:2,378  (2.9 %)\n",
      "\t Thoracic_inj:64,821 (80.0 %)\n",
      "\t Thoracic_inj:16,219  (20.0 %)\n",
      "\t Abdominal_inj:71,004 (87.6 %)\n",
      "\t Abdominal_inj:10,036  (12.4 %)\n",
      "\t Spine_inj:70,142 (86.6 %)\n",
      "\t Spine_inj:10,898  (13.4 %)\n",
      "\t Upper Extremity_inj:53,378 (65.9 %)\n",
      "\t Upper Extremity_inj:27,662  (34.1 %)\n",
      "\t Lower Extremity_inj:50,066 (61.8 %)\n",
      "\t Lower Extremity_inj:30,974  (38.2 %)\n",
      "\t Unspecified Injury_inj:74,658 (92.1 %)\n",
      "\t Unspecified Injury_inj:6,382  (7.9 %)\n"
     ]
    }
   ],
   "source": [
    "for x in datasets:\n",
    "    print('TQIP_' + x + ': Statistics for injuries:')\n",
    "    for i in regions:\n",
    "        var = globals()['TQIP_' + x]\n",
    "        counts = var[i + '_inj'].value_counts()\n",
    "        per = var[i + '_inj'].value_counts(normalize = True)\n",
    "        print(\"\\t\", i + '_inj:' + f\"{counts[0]:,}\" ' ('+\"{:.1%}\".format(per[0]).replace('%', ' %')+')')\n",
    "        print(\"\\t\", i + '_inj:' + f\"{counts[1]:,}\", ' ('+\"{:.1%}\".format(per[1]).replace('%', ' %')+')')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Continous input features"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define continous input variables: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "cont_names_ = col_names_long[32:40] + col_names_long[134:143]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Print statistics for the train, validation and test datasets: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "TQIP_train: Statistics for continous variables:\n",
      "\tAge (years): 50.0 [27.0, 70.0] - 43,990 (6.1%)\n",
      "\tWeight: 75.0 [61.2, 90.0] - 49,008 (6.8%)\n",
      "\tHeight: 170.0 [160.02, 177.8] - 102,799 (14.3%)\n",
      "\tInitial EMS Systolic Blood Pressure: 138.0 [120.0, 156.0] - 324,404 (45.2%)\n",
      "\tInitial EMS Pulse Rate: 88.0 [76.0, 102.0] - 317,477 (44.2%)\n",
      "\tInitial EMS Respiratory Rate: 18.0 [16.0, 20.0] - 329,167 (45.8%)\n",
      "\tInitial EMS Oxygen Saturation: 97.0 [95.0, 99.0] - 386,044 (53.7%)\n",
      "\tInitial EMS Total GCS: 15.0 [14.0, 15.0] - 322,875 (44.9%)\n",
      "\tTime to EMS Response (mins): 8.0 [5.0, 14.0] - 255,163 (35.5%)\n",
      "\tTime EMS spent at scene (mins): 16.0 [11.0, 23.0] - 250,686 (34.9%)\n",
      "\tTime from dispatch to ED/hospital arrival (mins): 49.0 [35.0, 71.0] - 249,122 (34.7%)\n",
      "\tInitial ED Systolic Blood Pressure: 137.0 [121.0, 154.0] - 23,814 (3.3%)\n",
      "\tInitial ED Pulse Rate: 86.0 [74.0, 100.0] - 16,610 (2.3%)\n",
      "\tInitial ED Temperature: 36.7 [36.4, 36.9] - 73,549 (10.2%)\n",
      "\tInitial ED Respiratory Rate: 18.0 [16.0, 20.0] - 25,312 (3.5%)\n",
      "\tInitial ED Oxygen Saturation: 98.0 [96.0, 99.0] - 29,271 (4.1%)\n",
      "\tAIS derived ISS: 9.0 [4.0, 10.0] - 2,396 (0.3%)\n",
      "TQIP_valid: Statistics for continous variables:\n",
      "\tAge (years): 50.0 [27.0, 69.0] - 10,915 (6.1%)\n",
      "\tWeight: 75.0 [61.2, 90.0] - 12,161 (6.8%)\n",
      "\tHeight: 170.0 [160.02, 177.8] - 25,715 (14.3%)\n",
      "\tInitial EMS Systolic Blood Pressure: 138.0 [121.0, 156.0] - 81,583 (45.4%)\n",
      "\tInitial EMS Pulse Rate: 88.0 [76.0, 102.0] - 79,853 (44.5%)\n",
      "\tInitial EMS Respiratory Rate: 18.0 [16.0, 20.0] - 82,790 (46.1%)\n",
      "\tInitial EMS Oxygen Saturation: 97.0 [95.0, 99.0] - 96,725 (53.9%)\n",
      "\tInitial EMS Total GCS: 15.0 [15.0, 15.0] - 81,044 (45.1%)\n",
      "\tTime to EMS Response (mins): 8.0 [5.0, 14.0] - 63,810 (35.5%)\n",
      "\tTime EMS spent at scene (mins): 16.0 [11.0, 23.0] - 62,722 (34.9%)\n",
      "\tTime from dispatch to ED/hospital arrival (mins): 49.0 [35.0, 71.0] - 62,320 (34.7%)\n",
      "\tInitial ED Systolic Blood Pressure: 137.0 [121.0, 154.0] - 5,966 (3.3%)\n",
      "\tInitial ED Pulse Rate: 87.0 [74.0, 100.0] - 4,113 (2.3%)\n",
      "\tInitial ED Temperature: 36.7 [36.4, 36.9] - 18,449 (10.3%)\n",
      "\tInitial ED Respiratory Rate: 18.0 [16.0, 20.0] - 6,371 (3.5%)\n",
      "\tInitial ED Oxygen Saturation: 98.0 [96.0, 99.0] - 7,328 (4.1%)\n",
      "\tAIS derived ISS: 9.0 [4.0, 10.0] - 583 (0.3%)\n",
      "TQIP_test: Statistics for continous variables:\n",
      "\tAge (years): 34.0 [15.0, 60.0] - 4,979 (6.1%)\n",
      "\tWeight: 68.04 [45.5, 84.1] - 9,297 (11.5%)\n",
      "\tHeight: 167.64 [155.0, 177.8] - 23,416 (28.9%)\n",
      "\tInitial EMS Systolic Blood Pressure: 136.0 [119.0, 153.0] - 39,844 (49.2%)\n",
      "\tInitial EMS Pulse Rate: 90.0 [78.0, 106.0] - 38,913 (48.0%)\n",
      "\tInitial EMS Respiratory Rate: 18.0 [16.0, 20.0] - 40,404 (49.9%)\n",
      "\tInitial EMS Oxygen Saturation: 98.0 [96.0, 99.0] - 45,792 (56.5%)\n",
      "\tInitial EMS Total GCS: 15.0 [15.0, 15.0] - 38,700 (47.8%)\n",
      "\tTime to EMS Response (mins): 8.0 [5.0, 14.0] - 29,766 (36.7%)\n",
      "\tTime EMS spent at scene (mins): 16.0 [11.0, 23.0] - 30,138 (37.2%)\n",
      "\tTime from dispatch to ED/hospital arrival (mins): 49.0 [35.0, 74.0] - 29,333 (36.2%)\n",
      "\tInitial ED Systolic Blood Pressure: 131.0 [117.0, 149.0] - 3,376 (4.2%)\n",
      "\tInitial ED Pulse Rate: 90.0 [77.0, 106.0] - 1,570 (1.9%)\n",
      "\tInitial ED Temperature: 36.7 [36.5, 37.0] - 7,463 (9.2%)\n",
      "\tInitial ED Respiratory Rate: 18.0 [16.0, 22.0] - 2,563 (3.2%)\n",
      "\tInitial ED Oxygen Saturation: 98.0 [96.0, 100.0] - 6,809 (8.4%)\n",
      "\tAIS derived ISS: 5.0 [4.0, 10.0] - 225 (0.3%)\n"
     ]
    }
   ],
   "source": [
    "for x in datasets:\n",
    "    print('TQIP_' + x + ': Statistics for continous variables:')\n",
    "    for i in cont_names_:\n",
    "        var = globals()['TQIP_'+x]\n",
    "        l,m,h = var[i].quantile(q=(0.25, 0.5, 0.75))\n",
    "        missing = var[i].isna().sum()\n",
    "        perc = missing/(len(var))\n",
    "        print(\"\\t\" + i + ':', m, '['+str(l)+', ' + str(h) +'] - ' + f\"{missing:,}\", '(' + \"{:.1%}\".format(perc) + ')')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Output variables"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Define output variables: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_names = col_names_long[146:165]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Create an empty dataframe for the statistics of the output variables:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_names_df = pd.DataFrame(columns=['Category', 'Train','Valid', 'Test'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Fill in the dataframe: "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "for c in y_names:\n",
    "    Train_counts = TQIP_train[c].value_counts(dropna=False)\n",
    "    Train_per = Train_counts / Train_counts.sum()\n",
    "    Valid_counts = TQIP_valid[c].value_counts(dropna=False)\n",
    "    Valid_per = Valid_counts / Valid_counts.sum()\n",
    "    Test_counts = TQIP_test[c].value_counts(dropna=False)\n",
    "    Test_per = Test_counts / Test_counts.sum()\n",
    "    fmt_pct = '{:.1%}'.format\n",
    "    fmt_ths = '{:,}'.format\n",
    "    y_names_df = y_names_df.append(pd.DataFrame({'Category': c, \n",
    "                                         'Train': Train_counts.map(fmt_ths) + ' (' + Train_per.map(fmt_pct) + ')', \n",
    "                                         'Valid': Valid_counts.map(fmt_ths) + ' (' + Valid_per.map(fmt_pct) + ')', \n",
    "                                         'Test': Test_counts.map(fmt_ths) + ' (' + Test_per.map(fmt_pct) + ')'}))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Reset index and save the dataframe as a csv file:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_names_df.index = y_names_df.index.set_names('variable')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_names_df.reset_index(inplace=True)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "y_names_df.to_csv('y_names_df_1.csv')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Data Leakage:"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Early mortality and prehospital cardiac arrest:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "Yes    0.520438\n",
       "No     0.479562\n",
       "Name: Pre-Hospital Cardiac Arrest, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP.loc[TQIP['Early Mortality'] == 1]['Pre-Hospital Cardiac Arrest'].value_counts(Normalize) "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "1    0.553715\n",
       "0    0.446285\n",
       "Name: Early Mortality, dtype: float64"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP.loc[TQIP['Pre-Hospital Cardiac Arrest'] == 'Yes']['Early Mortality'].value_counts(Normalize) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Examining the data, we found that 52.0% of the patients who died within 24 hours had a prehospital cardiac arrest, and that 55.4% of the patients with a prehospital cardiac arrest died within 24 hours. "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "#### Early and late mortality:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>inc_key</th>\n",
       "      <th>Gender</th>\n",
       "      <th>SEX_BIU</th>\n",
       "      <th>Age (years)</th>\n",
       "      <th>Race Category: Asian</th>\n",
       "      <th>Race Category: Pacific Islander</th>\n",
       "      <th>Race Category: Other</th>\n",
       "      <th>Race Category: American Indian</th>\n",
       "      <th>Race Category: Black</th>\n",
       "      <th>Race Category: White</th>\n",
       "      <th>...</th>\n",
       "      <th>Minor Unspecified Injury</th>\n",
       "      <th>Moderate Unspecified Injury</th>\n",
       "      <th>Serious Unspecified Injury</th>\n",
       "      <th>Severe Unspecified Injury</th>\n",
       "      <th>Critical Unspecified Injury</th>\n",
       "      <th>Maximum Unspecified Injury</th>\n",
       "      <th>Unspecified Injury (NFS)</th>\n",
       "      <th>UNSPEC_&lt;NA&gt;</th>\n",
       "      <th>Early Mortality</th>\n",
       "      <th>Late Mortality</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>0 rows × 402 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "Empty DataFrame\n",
       "Columns: [inc_key, Gender, SEX_BIU, Age (years), Race Category: Asian, Race Category: Pacific Islander, Race Category: Other, Race Category: American Indian, Race Category: Black, Race Category: White, RACE_NA, RACE_UK, Ethnicity, ETHNICITY_BIU, WORKRELATED, WORKRELATED_BIU, PTOCCUPATIONALINDUSTRY, PTOCCUPATIONALINDUSTRY_BIU, PATIENTSOCCUPATION, PATIENTSOCCUPATION_BIU, Primary External Cause, PRIMARYECODEICD10_BIU, PLACEOFINJURYCODE, PLACEOFINJURYCODE_BIU, ADDITIONALECODEICD10, ADDITIONALECODEICD10_BIU, PROTDEV_NONE, PROTDEV_LAP_BELT, PROTDEV_PER_FLOAT, PROTDEV_PROTECT_GEAR, PROTDEV_EYE_PROTECT, PROTDEV_CHILD_RESTRAINT, PROTDEV_HELMET, PROTDEV_AIRBAG_PRESENT, PROTDEV_PROTECT_CLOTH, PROTDEV_SHOULDER_BELT, PROTDEV_OTHER, PROTDEV_NA, PROTDEV_UK, CHILDSPECIFICRESTRAINT, AIRBAG_NOTDEPLOYED, AIRBAG_DEPLOYED_FRNT, AIRBAG_DEPLOYED_SIDE, AIRBAG_DEPLOYED_OTHER, AIRBAG_DEPLOYED_NA, AIRBAG_DEPLOYED_UK, ABUSEREPORT, ABUSEREPORT_BIU, ABUSEINVESTIGATION, ABUSEINVESTIGATION_BIU, CAREGIVERATDISCHARGE, CAREGIVERATDISCHARGE_BIU, Transport Mode, TRANSPORTMODE_BIU, TM_GROUNDAMBULANCE, TM_HELICOPTERAMBULANCE, TM_FIXEDWINGAMBULANCE, TM_PRIVPUBVEHWALKIN, TM_POLICE, TM_OTHER, TM_NA, TM_UK, Initial EMS Systolic Blood Pressure, EMSSBP_BIU, Initial EMS Pulse Rate, EMSPULSERATE_BIU, Initial EMS Respiratory Rate, EMSRESPIRATORYRATE_BIU, Initial EMS Oxygen Saturation, EMSPULSEOXIMETRY_BIU, EMS GCS - Eye, EMSGCSEYE_BIU, EMS GCS - Verbal, EMSGCSVERBAL_BIU, EMS GCS - Motor, EMSGCSMOTOR_BIU, Initial EMS Total GCS, EMSTOTALGCS_BIU, Time to EMS Response (mins), EMSRESPONSEDAYS, Time EMS spent at scene (mins), EMSSCENEDAYS, Time from dispatch to ED/hospital arrival (mins), EMSDAYS, INTERFACILITYTRANSFER, INTERFACILITYTRANSFER_BIU, Pre-Hospital Cardiac Arrest, PREHOSPITALCARDIACARREST_BIU, TCC: Glasgow Coma Scale of 13 or less, TCC: Systolic Blood Pressure under 90, TCC: Respiratory rate less than 10 or more than 29, TCC: Penetrating Injuries, TCC: Chest wall instability or deformity, TCC: Two or more proximal long-bone fractures, TCC: Crushed, degloved, mangled, or pulseless extremity, TCC: Amputation proximal to wrist or ankle, TCC: Pelvic fracture, TCC: Open or depressed skull fracture, TCC: Paralysis, TCC_NA, ...]\n",
       "Index: []\n",
       "\n",
       "[0 rows x 402 columns]"
      ]
     },
     "execution_count": null,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "TQIP.loc[(TQIP['Early Mortality'] == 1) & (TQIP['Late Mortality'] == 1)]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "No patients were coded as having experienced both early- and late death."
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.3"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}