data_wrangling.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Data wrangling\n",
    "Using hourly [FLUXNET](https://fluxnet.org/) data, compute monthly means and add IGBP information from the Biological, Ancillary, Disturbance, and Metadata (BADM) file as a column to the table. Output a csv file for use in other notebooks.<br><br>\n",
    "This notebook uses the python package [pandas](https://pandas.pydata.org/docs/) to load, visualize, and resample data. \n",
    "\n",
    "\n",
    "**Input**: \n",
    " - [FLUXNET 2015 SUBSET hourly data](https://fluxnet.org/data/fluxnet2015-dataset/subset-data-product/) for all tower sites\n",
    " - BADM file from FLUXNET 2015\n",
    "\n",
    "\n",
    "**Output**: \n",
    " - csv file for use in other notebooks"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import notebook dependencies"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd \n",
    "import numpy as np \n",
    "from glob import glob\n",
    "import sys\n",
    "import os"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load FLUXNET 2015 files into notebook\n",
    "Csv files from all tower sites in the FLUXNET 2015 dataset are stored in a directory in the repository. Here, we load in the files as DataFrame tables using pandas, and save the sitename (part of the filename for each file) as a column in the table.<br><br>A progress bar function is used in code development to show time remaining to run the code cell-- the cell loops through lots of data, and can take a while to run. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "#helper function used in code development\n",
    "def progressBar(i, tot): \n",
    "    \"\"\"Display a progress bar inside a for loop \n",
    "    Args: \n",
    "        i (int): iteration number\n",
    "        tot (int): total number of iterations\n",
    "    \"\"\"\n",
    "    j = (i + 1) / tot\n",
    "    sys.stdout.write('\\r [%-20s] %d%% complete' % ('='*int(20*j), 100*j))\n",
    "    sys.stdout.flush()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " [====================] 100% complete"
     ]
    }
   ],
   "source": [
    "# ----------------- load data ---------------------\n",
    "\n",
    "#list of SUBSET filenames \n",
    "files = glob('data/FLUXNET2015/' + '*FLUXNET2015_SUBSET_HH*.csv')\n",
    "\n",
    "#list for storing pandas dataframe objects\n",
    "SUBSET_list = []\n",
    "\n",
    "#loop through files in directory and add SUBSET hourly data to list\n",
    "for i in range(len(files)):\n",
    "    df = pd.read_csv(files[i])\n",
    "    \n",
    "    #add sitename to df \n",
    "    sitename = files[i].split('FLX_')[1].split('_FLUXNET2015')[0]\n",
    "    df['SITE_ID'] = sitename\n",
    "    \n",
    "    #append to list\n",
    "    SUBSET_list.append(df)\n",
    "    \n",
    "    #display progress bar\n",
    "    progressBar(i, len(files))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Compute monthly averages\n",
    "Resample the time series data, remove unneccessary columns, and add standard deviation of relative humidity to the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "SUBSET_monthly = []\n",
    "for i in range(len(SUBSET_list)): \n",
    "    df = SUBSET_list[i] \n",
    "    sitename = df[\"SITE_ID\"][0]\n",
    "    \n",
    "    #remove unneccessary columns\n",
    "    column_names = [\"TIMESTAMP_START\",\"SITE_ID\",\"LE_F_MDS\",\"VPD_F\",\"TA_F\",\"NETRAD\",\"G_F_MDS\", \"PA_F\", \"RH\"]\n",
    "    if set(column_names).issubset(df.columns):\n",
    "        #rename columns to include units\n",
    "        df = df[column_names].rename(columns= {\"LE_F_MDS\":\"LE (W/m^2)\",\"VPD_F\":\"VPD (hPa)\",\"TA_F\":\"TA (deg C)\",\"NETRAD\":\"NETRAD (W/m^2)\",\n",
    "                                             \"G_F_MDS\":\"G (W/m^2)\",\"PA_F\":\"PA (kPa)\"})\n",
    "        #reformat time\n",
    "        df[\"TIMESTAMP_START\"] = pd.to_datetime(df[\"TIMESTAMP_START\"], format='%Y%m%d%H%M')\n",
    "        \n",
    "        #convert VPD from hPa to kPa \n",
    "        df.insert(loc = 3, column = \"VPD (kPa)\", value = df[\"VPD (hPa)\"]*0.1)\n",
    "        df = df.drop(columns = \"VPD (hPa)\")\n",
    "        \n",
    "        #drop columns with missing data\n",
    "        df = df.dropna() #drop columns with NaN\n",
    "        missing_data = -9999.0 #missing data flag"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>TIMESTAMP_START</th>\n",
       "      <th>SITE_ID</th>\n",
       "      <th>LE (W/m^2)</th>\n",
       "      <th>VPD (kPa)</th>\n",
       "      <th>TA (deg C)</th>\n",
       "      <th>NETRAD (W/m^2)</th>\n",
       "      <th>G (W/m^2)</th>\n",
       "      <th>PA (kPa)</th>\n",
       "      <th>RH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>6000</th>\n",
       "      <td>2002-05-06 00:00:00</td>\n",
       "      <td>US-Wi3</td>\n",
       "      <td>27.93980</td>\n",
       "      <td>0.1346</td>\n",
       "      <td>2.439</td>\n",
       "      <td>-9999.000</td>\n",
       "      <td>-5.41635</td>\n",
       "      <td>96.846</td>\n",
       "      <td>-9999.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6001</th>\n",
       "      <td>2002-05-06 00:30:00</td>\n",
       "      <td>US-Wi3</td>\n",
       "      <td>27.93980</td>\n",
       "      <td>0.1346</td>\n",
       "      <td>2.439</td>\n",
       "      <td>-9999.000</td>\n",
       "      <td>-5.41635</td>\n",
       "      <td>96.844</td>\n",
       "      <td>-9999.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6002</th>\n",
       "      <td>2002-05-06 01:00:00</td>\n",
       "      <td>US-Wi3</td>\n",
       "      <td>27.93980</td>\n",
       "      <td>0.1346</td>\n",
       "      <td>2.439</td>\n",
       "      <td>-9999.000</td>\n",
       "      <td>-5.41635</td>\n",
       "      <td>96.841</td>\n",
       "      <td>-9999.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6003</th>\n",
       "      <td>2002-05-06 01:30:00</td>\n",
       "      <td>US-Wi3</td>\n",
       "      <td>27.93980</td>\n",
       "      <td>0.1346</td>\n",
       "      <td>2.439</td>\n",
       "      <td>-9999.000</td>\n",
       "      <td>-5.41635</td>\n",
       "      <td>96.839</td>\n",
       "      <td>-9999.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6004</th>\n",
       "      <td>2002-05-06 02:00:00</td>\n",
       "      <td>US-Wi3</td>\n",
       "      <td>27.93980</td>\n",
       "      <td>0.1346</td>\n",
       "      <td>2.439</td>\n",
       "      <td>-9999.000</td>\n",
       "      <td>-5.41635</td>\n",
       "      <td>96.836</td>\n",
       "      <td>-9999.000</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52603</th>\n",
       "      <td>2004-12-31 21:30:00</td>\n",
       "      <td>US-Wi3</td>\n",
       "      <td>4.78240</td>\n",
       "      <td>0.0360</td>\n",
       "      <td>-10.535</td>\n",
       "      <td>-12.574</td>\n",
       "      <td>-9999.00000</td>\n",
       "      <td>98.122</td>\n",
       "      <td>87.006</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52604</th>\n",
       "      <td>2004-12-31 22:00:00</td>\n",
       "      <td>US-Wi3</td>\n",
       "      <td>4.15636</td>\n",
       "      <td>0.0280</td>\n",
       "      <td>-10.653</td>\n",
       "      <td>-9.567</td>\n",
       "      <td>-9999.00000</td>\n",
       "      <td>98.178</td>\n",
       "      <td>90.032</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52605</th>\n",
       "      <td>2004-12-31 22:30:00</td>\n",
       "      <td>US-Wi3</td>\n",
       "      <td>3.88138</td>\n",
       "      <td>0.0300</td>\n",
       "      <td>-11.158</td>\n",
       "      <td>-14.133</td>\n",
       "      <td>-9999.00000</td>\n",
       "      <td>98.252</td>\n",
       "      <td>88.835</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52606</th>\n",
       "      <td>2004-12-31 23:00:00</td>\n",
       "      <td>US-Wi3</td>\n",
       "      <td>3.89083</td>\n",
       "      <td>0.0270</td>\n",
       "      <td>-11.438</td>\n",
       "      <td>-31.407</td>\n",
       "      <td>-9999.00000</td>\n",
       "      <td>98.312</td>\n",
       "      <td>89.645</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>52607</th>\n",
       "      <td>2004-12-31 23:30:00</td>\n",
       "      <td>US-Wi3</td>\n",
       "      <td>3.89083</td>\n",
       "      <td>0.0270</td>\n",
       "      <td>-11.438</td>\n",
       "      <td>-31.407</td>\n",
       "      <td>-9999.00000</td>\n",
       "      <td>98.312</td>\n",
       "      <td>89.645</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>46608 rows × 9 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "          TIMESTAMP_START SITE_ID  LE (W/m^2)  VPD (kPa)  TA (deg C)  \\\n",
       "6000  2002-05-06 00:00:00  US-Wi3    27.93980     0.1346       2.439   \n",
       "6001  2002-05-06 00:30:00  US-Wi3    27.93980     0.1346       2.439   \n",
       "6002  2002-05-06 01:00:00  US-Wi3    27.93980     0.1346       2.439   \n",
       "6003  2002-05-06 01:30:00  US-Wi3    27.93980     0.1346       2.439   \n",
       "6004  2002-05-06 02:00:00  US-Wi3    27.93980     0.1346       2.439   \n",
       "...                   ...     ...         ...        ...         ...   \n",
       "52603 2004-12-31 21:30:00  US-Wi3     4.78240     0.0360     -10.535   \n",
       "52604 2004-12-31 22:00:00  US-Wi3     4.15636     0.0280     -10.653   \n",
       "52605 2004-12-31 22:30:00  US-Wi3     3.88138     0.0300     -11.158   \n",
       "52606 2004-12-31 23:00:00  US-Wi3     3.89083     0.0270     -11.438   \n",
       "52607 2004-12-31 23:30:00  US-Wi3     3.89083     0.0270     -11.438   \n",
       "\n",
       "       NETRAD (W/m^2)   G (W/m^2)  PA (kPa)        RH  \n",
       "6000        -9999.000    -5.41635    96.846 -9999.000  \n",
       "6001        -9999.000    -5.41635    96.844 -9999.000  \n",
       "6002        -9999.000    -5.41635    96.841 -9999.000  \n",
       "6003        -9999.000    -5.41635    96.839 -9999.000  \n",
       "6004        -9999.000    -5.41635    96.836 -9999.000  \n",
       "...               ...         ...       ...       ...  \n",
       "52603         -12.574 -9999.00000    98.122    87.006  \n",
       "52604          -9.567 -9999.00000    98.178    90.032  \n",
       "52605         -14.133 -9999.00000    98.252    88.835  \n",
       "52606         -31.407 -9999.00000    98.312    89.645  \n",
       "52607         -31.407 -9999.00000    98.312    89.645  \n",
       "\n",
       "[46608 rows x 9 columns]"
      ]
     },
     "execution_count": 20,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "df[6000:]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0        False\n",
       "1        False\n",
       "2        False\n",
       "3        False\n",
       "4        False\n",
       "         ...  \n",
       "52603    False\n",
       "52604    False\n",
       "52605    False\n",
       "52606    False\n",
       "52607    False\n",
       "Length: 52608, dtype: bool"
      ]
     },
     "execution_count": 21,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "~df.eq(missing_data).any(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "tags": [
     "remove-output"
    ]
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " [====================] 100% complete"
     ]
    }
   ],
   "source": [
    "SUBSET_monthly = []\n",
    "for i in range(len(SUBSET_list)): \n",
    "    df = SUBSET_list[i] \n",
    "    sitename = df[\"SITE_ID\"][0]\n",
    "    \n",
    "    #remove unneccessary columns\n",
    "    column_names = [\"TIMESTAMP_START\",\"SITE_ID\",\"LE_F_MDS\",\"VPD_F\",\"TA_F\",\"NETRAD\",\"G_F_MDS\", \"PA_F\", \"RH\"]\n",
    "    if set(column_names).issubset(df.columns):\n",
    "        #rename columns to include units\n",
    "        df = df[column_names].rename(columns= {\"LE_F_MDS\":\"LE (W/m^2)\",\"VPD_F\":\"VPD (hPa)\",\"TA_F\":\"TA (deg C)\",\"NETRAD\":\"NETRAD (W/m^2)\",\n",
    "                                             \"G_F_MDS\":\"G (W/m^2)\",\"PA_F\":\"PA (kPa)\"})\n",
    "        #reformat time\n",
    "        df[\"TIMESTAMP_START\"] = pd.to_datetime(df[\"TIMESTAMP_START\"], format='%Y%m%d%H%M')\n",
    "        \n",
    "        #convert VPD from hPa to kPa \n",
    "        df.insert(loc = 3, column = \"VPD (kPa)\", value = df[\"VPD (hPa)\"]*0.1)\n",
    "        df = df.drop(columns = \"VPD (hPa)\")\n",
    "        \n",
    "        #drop columns with missing data\n",
    "        df = df.dropna() #drop columns with NaN\n",
    "        missing_data = -9999.0 #missing data flag\n",
    "        df = df[~df.eq(missing_data).any(1)].reset_index(drop = True) \n",
    "\n",
    "        #convert RH from percent to fraction\n",
    "        df[\"RH\"] = df[\"RH\"]/100 \n",
    "        \n",
    "        #resample dataset to get monthly means\n",
    "        df = df.set_index([\"TIMESTAMP_START\"])\n",
    "        std_rh = df[\"RH\"].resample('MS').std()\n",
    "        df = df.resample('MS').mean()\n",
    "             \n",
    "        #append to list if dataframe contains data\n",
    "        if len(df) > 0: \n",
    "            df[\"Std Dev RH\"] = std_rh\n",
    "            df.insert(loc = 0, column = \"TIMESTAMP_START\", value = df.index)\n",
    "            df.insert(loc = 1, column = \"SITE_ID\", value = sitename)\n",
    "            df = df.reset_index(drop = True)\n",
    "            SUBSET_monthly.append(df)\n",
    "        \n",
    "    #display progress bar\n",
    "    progressBar(i, len(SUBSET_list))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Load BADM file into notebook\n",
    "This file contains data about each FLUXNET tower. We will add the IGBP ecotype corresponding to each tower to the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "BADM = pd.read_csv('data/FLUXNET2015/FLX_AA-Flx_BIF_WW_20200501.csv', usecols = ['SITE_ID','VARIABLE','DATAVALUE'])\n",
    "ecotype = BADM.loc[BADM['VARIABLE'] == 'IGBP'].reset_index(drop = True).drop(columns = 'VARIABLE')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Merge BADM IGBP data with SUBSET HH\n",
    "Add the BADM IGBP ecotype data as a column to the SUBSET hourly DataFrame. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>DATE</th>\n",
       "      <th>FLUXNET-ID</th>\n",
       "      <th>IGBP</th>\n",
       "      <th>LE (W/m^2)</th>\n",
       "      <th>VPD (kPa)</th>\n",
       "      <th>TA (deg C)</th>\n",
       "      <th>NETRAD (W/m^2)</th>\n",
       "      <th>G (W/m^2)</th>\n",
       "      <th>PA (kPa)</th>\n",
       "      <th>RH</th>\n",
       "      <th>Std Dev RH</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>2011-01-01</td>\n",
       "      <td>US-CRT</td>\n",
       "      <td>CRO</td>\n",
       "      <td>5.317504</td>\n",
       "      <td>0.101241</td>\n",
       "      <td>-5.489418</td>\n",
       "      <td>9.618696</td>\n",
       "      <td>-4.961629</td>\n",
       "      <td>99.831905</td>\n",
       "      <td>0.757487</td>\n",
       "      <td>0.102424</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2011-02-01</td>\n",
       "      <td>US-CRT</td>\n",
       "      <td>CRO</td>\n",
       "      <td>11.828405</td>\n",
       "      <td>0.145528</td>\n",
       "      <td>-2.971717</td>\n",
       "      <td>20.088240</td>\n",
       "      <td>-0.771094</td>\n",
       "      <td>99.859839</td>\n",
       "      <td>0.737630</td>\n",
       "      <td>0.134346</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2011-03-01</td>\n",
       "      <td>US-CRT</td>\n",
       "      <td>CRO</td>\n",
       "      <td>34.717653</td>\n",
       "      <td>0.197390</td>\n",
       "      <td>2.153819</td>\n",
       "      <td>92.061066</td>\n",
       "      <td>11.126964</td>\n",
       "      <td>100.084974</td>\n",
       "      <td>0.747895</td>\n",
       "      <td>0.144803</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2011-04-01</td>\n",
       "      <td>US-CRT</td>\n",
       "      <td>CRO</td>\n",
       "      <td>45.258565</td>\n",
       "      <td>0.307019</td>\n",
       "      <td>7.368077</td>\n",
       "      <td>96.613316</td>\n",
       "      <td>11.045620</td>\n",
       "      <td>99.294018</td>\n",
       "      <td>0.737826</td>\n",
       "      <td>0.150827</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2011-05-01</td>\n",
       "      <td>US-CRT</td>\n",
       "      <td>CRO</td>\n",
       "      <td>82.249253</td>\n",
       "      <td>0.444950</td>\n",
       "      <td>15.870246</td>\n",
       "      <td>143.585133</td>\n",
       "      <td>13.552872</td>\n",
       "      <td>99.448677</td>\n",
       "      <td>0.794998</td>\n",
       "      <td>0.157474</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2011-06-01</td>\n",
       "      <td>US-CRT</td>\n",
       "      <td>CRO</td>\n",
       "      <td>53.869295</td>\n",
       "      <td>0.992942</td>\n",
       "      <td>22.068460</td>\n",
       "      <td>172.166276</td>\n",
       "      <td>8.117042</td>\n",
       "      <td>99.467544</td>\n",
       "      <td>0.676487</td>\n",
       "      <td>0.183932</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>2011-07-01</td>\n",
       "      <td>US-CRT</td>\n",
       "      <td>CRO</td>\n",
       "      <td>125.305980</td>\n",
       "      <td>1.061575</td>\n",
       "      <td>25.869129</td>\n",
       "      <td>183.696062</td>\n",
       "      <td>6.692552</td>\n",
       "      <td>99.519419</td>\n",
       "      <td>0.717682</td>\n",
       "      <td>0.184548</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>2011-08-01</td>\n",
       "      <td>US-CRT</td>\n",
       "      <td>CRO</td>\n",
       "      <td>137.802859</td>\n",
       "      <td>0.581209</td>\n",
       "      <td>21.853141</td>\n",
       "      <td>161.626838</td>\n",
       "      <td>-0.528036</td>\n",
       "      <td>99.361281</td>\n",
       "      <td>0.806610</td>\n",
       "      <td>0.152929</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>2011-09-01</td>\n",
       "      <td>US-CRT</td>\n",
       "      <td>CRO</td>\n",
       "      <td>73.312711</td>\n",
       "      <td>0.347930</td>\n",
       "      <td>17.358452</td>\n",
       "      <td>100.017480</td>\n",
       "      <td>-3.180038</td>\n",
       "      <td>99.601412</td>\n",
       "      <td>0.853520</td>\n",
       "      <td>0.136794</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>2011-10-01</td>\n",
       "      <td>US-CRT</td>\n",
       "      <td>CRO</td>\n",
       "      <td>36.766119</td>\n",
       "      <td>0.394087</td>\n",
       "      <td>11.548182</td>\n",
       "      <td>73.515675</td>\n",
       "      <td>-5.649803</td>\n",
       "      <td>99.628152</td>\n",
       "      <td>0.763188</td>\n",
       "      <td>0.181707</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "        DATE FLUXNET-ID IGBP  LE (W/m^2)  VPD (kPa)  TA (deg C)  \\\n",
       "0 2011-01-01     US-CRT  CRO    5.317504   0.101241   -5.489418   \n",
       "1 2011-02-01     US-CRT  CRO   11.828405   0.145528   -2.971717   \n",
       "2 2011-03-01     US-CRT  CRO   34.717653   0.197390    2.153819   \n",
       "3 2011-04-01     US-CRT  CRO   45.258565   0.307019    7.368077   \n",
       "4 2011-05-01     US-CRT  CRO   82.249253   0.444950   15.870246   \n",
       "5 2011-06-01     US-CRT  CRO   53.869295   0.992942   22.068460   \n",
       "6 2011-07-01     US-CRT  CRO  125.305980   1.061575   25.869129   \n",
       "7 2011-08-01     US-CRT  CRO  137.802859   0.581209   21.853141   \n",
       "8 2011-09-01     US-CRT  CRO   73.312711   0.347930   17.358452   \n",
       "9 2011-10-01     US-CRT  CRO   36.766119   0.394087   11.548182   \n",
       "\n",
       "   NETRAD (W/m^2)  G (W/m^2)    PA (kPa)        RH  Std Dev RH  \n",
       "0        9.618696  -4.961629   99.831905  0.757487    0.102424  \n",
       "1       20.088240  -0.771094   99.859839  0.737630    0.134346  \n",
       "2       92.061066  11.126964  100.084974  0.747895    0.144803  \n",
       "3       96.613316  11.045620   99.294018  0.737826    0.150827  \n",
       "4      143.585133  13.552872   99.448677  0.794998    0.157474  \n",
       "5      172.166276   8.117042   99.467544  0.676487    0.183932  \n",
       "6      183.696062   6.692552   99.519419  0.717682    0.184548  \n",
       "7      161.626838  -0.528036   99.361281  0.806610    0.152929  \n",
       "8      100.017480  -3.180038   99.601412  0.853520    0.136794  \n",
       "9       73.515675  -5.649803   99.628152  0.763188    0.181707  "
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "#combine dataframes\n",
    "combined = pd.concat(SUBSET_monthly).merge(ecotype, on = 'SITE_ID', how = 'inner')\n",
    "\n",
    "#rearrange order of IGBP column\n",
    "combined.insert(loc = 2, column = \"IGBP\", value = combined[\"DATAVALUE\"].values)\n",
    "combined = combined.drop(columns = \"DATAVALUE\")\n",
    "\n",
    "#rename columns\n",
    "combined = combined.rename(columns= {\"TIMESTAMP_START\":\"DATE\", \"SITE_ID\":\"FLUXNET-ID\"})\n",
    "\n",
    "#display part of dataframe \n",
    "display(combined[:10])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Save DataFrame as csv to local drive \n",
    "This csv file will be used in other notebooks for data analysis, and is available as \"FLUXNET2015_monthly.csv\" in the GitHub repository for this project. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "combined.to_csv('data/FLUXNET2015_monthly.csv', index = False)"
   ]
  }
 ],
 "metadata": {
  "celltoolbar": "Tags",
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}