Surface Volume CSV Processor.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Surface Volume and CSV Processor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This code generates Surface Volume .csv files that are used in the Flood Estimate code, taking DEM .TIFF files divided by division and DEM files that have been grouped with the two .TIFF files to the south and north of them. The resulting .csv files contain the volume of water above the surface of the DEM at different heights in increments of 0.25m betwwen 0-3m and in increments of 0.5m from 3.5-6.5m. This processor also creates a folder called \"Intermediate Data\" which can be deleted after the code has completely run.\n",
    "\n",
    "See https://pro.arcgis.com/en/pro-app/tool-reference/3d-analyst/surface-volume.htm for more information on how the arcpy Surface Volume function works."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Imports and Initializations"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.848956Z",
     "start_time": "2020-10-21T03:38:50.522830Z"
    }
   },
   "outputs": [],
   "source": [
    "#built-in packages within python that you have to input to run certain functions\n",
    "import arcpy as ap\n",
    "import scipy as sp\n",
    "import numpy as np\n",
    "import pandas as pd\n",
    "import os\n",
    "import csv\n",
    "import glob\n",
    "import re"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Your DEM should be divided into \"divisions\" to conveniently model the propagation of water throughout your area of interest."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.852945Z",
     "start_time": "2020-10-21T03:38:50.529Z"
    }
   },
   "outputs": [],
   "source": [
    "num_divs = 18 #however many divisions the DEM is divided into"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Ungrouped Divs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inputs to these cells are the DEMs that have been cut into divisions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Surface Volume Processor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Process for 0-5m at 0.25 m intervals for each div."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.854939Z",
     "start_time": "2020-10-21T03:38:50.534Z"
    }
   },
   "outputs": [],
   "source": [
    "ele_0_txt = \"\\\\ele_0.txt\"; ele_1_txt = \"\\\\ele_1.txt\"; ele_2_txt = \"\\\\ele_2.txt\" \n",
    "ele_3_txt = \"\\\\ele_3.txt\"; ele_4_txt = \"\\\\ele_4.txt\"; ele_5_txt = \"\\\\ele_5.txt\"\n",
    "ele_0_5_txt = \"\\\\ele_0_5.txt\"; ele_1_5_txt = \"\\\\ele_1_5.txt\"; ele_2_5_txt = \"\\\\ele_2_5.txt\" \n",
    "ele_3_5_txt = \"\\\\ele_3_5.txt\"; ele_4_5_txt = \"\\\\ele_4_5.txt\"\n",
    "ele_0_25_txt = \"\\\\ele_0_25.txt\"; ele_1_25_txt = \"\\\\ele_1_25.txt\"; ele_2_25_txt = \"\\\\ele_2_25.txt\" \n",
    "ele_3_25_txt = \"\\\\ele_3_25.txt\"; ele_4_25_txt = \"\\\\ele_4_25.txt\"\n",
    "ele_0_75_txt = \"\\\\ele_0_75.txt\"; ele_1_75_txt = \"\\\\ele_1_75.txt\"; ele_2_75_txt = \"\\\\ele_2_75.txt\" \n",
    "ele_3_75_txt = \"\\\\ele_3_75.txt\"; ele_4_75_txt = \"\\\\ele_4_75.txt\"\n",
    "\n",
    "for i in range(num_divs):\n",
    "    \n",
    "    #this creates the destination folder for the txt files to go into if it hasn't been created yet\n",
    "    #you can comment this out if the folders have been created since it takes a while to run\n",
    "    divfolder = r\"IntermediateData\\newtext\\div (%d)\"%i\n",
    "    if not os.path.exists(divfolder):\n",
    "        os.makedirs(divfolder)\n",
    "    \n",
    "    #this says what folder we want it to go into\n",
    "    ap.env.workspace = divfolder\n",
    "    \n",
    "    #this says what DEM we want to use in the Surface Volume function. \n",
    "    #####These are your input DEM files################################\n",
    "    #####Change these to the ones relevant to your area of interest####\n",
    "    dem_lm = r\"Data\\LM_div18\\dem_lm_z35_%d.TIF\"%i\n",
    "\n",
    "    \n",
    "    # Process: Surface Volume\n",
    "    # this performs Surface Volume on the DEM at the specified points.\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_txt, \"BELOW\", \"0\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_txt, \"BELOW\", \"1\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_txt, \"BELOW\", \"2\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_txt, \"BELOW\", \"3\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_txt, \"BELOW\", \"4\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_5_txt, \"BELOW\", \"5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_5_txt, \"BELOW\", \"0.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_5_txt, \"BELOW\", \"1.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_5_txt, \"BELOW\", \"2.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_5_txt, \"BELOW\", \"3.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_5_txt, \"BELOW\", \"4.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_25_txt, \"BELOW\", \"0.25\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_25_txt, \"BELOW\", \"1.25\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_25_txt, \"BELOW\", \"2.25\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_25_txt, \"BELOW\", \"3.25\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_25_txt, \"BELOW\", \"4.25\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_75_txt, \"BELOW\", \"0.75\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_75_txt, \"BELOW\", \"1.75\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_75_txt, \"BELOW\", \"2.75\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_75_txt, \"BELOW\", \"3.75\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_75_txt, \"BELOW\", \"4.75\", \"1\", \"0\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Process for 0-10m at 0.5m intervals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.855936Z",
     "start_time": "2020-10-21T03:38:50.538Z"
    }
   },
   "outputs": [],
   "source": [
    "ele_0_txt = \"\\\\ele_0.txt\"; ele_2_txt = \"\\\\ele_2.txt\"; ele_4_txt = \"\\\\ele_4.txt\" \n",
    "ele_6_txt = \"\\\\ele_6.txt\"; ele_8_txt = \"\\\\ele_8.txt\"; ele_10_txt = \"\\\\ele_10.txt\"\n",
    "ele_1_txt = \"\\\\ele_1.txt\"; ele_3_txt = \"\\\\ele_3.txt\"; ele_5_txt = \"\\\\ele_5.txt\" \n",
    "ele_7_txt = \"\\\\ele_7.txt\"; ele_9_txt = \"\\\\ele_9.txt\"\n",
    "ele_0_5_txt = \"\\\\ele_0_5.txt\"; ele_2_5_txt = \"\\\\ele_2_5.txt\"; ele_4_5_txt = \"\\\\ele_4_5.txt\" \n",
    "ele_6_5_txt = \"\\\\ele_6_5.txt\"; ele_8_5_txt = \"\\\\ele_8_5.txt\"\n",
    "ele_1_5_txt = \"\\\\ele_1_5.txt\"; ele_3_5_txt = \"\\\\ele_3_5.txt\"; ele_5_5_txt = \"\\\\ele_5_5.txt\" \n",
    "ele_7_5_txt = \"\\\\ele_7_5.txt\"; ele_9_5_txt = \"\\\\ele_9_5.txt\"\n",
    "    \n",
    "for i in range(num_divs):\n",
    "    \n",
    "    #this creates the destination folder for the txt files to go into if it hasn't been created yet\n",
    "    #you can comment this out if the folders have been created since it takes a while to run\n",
    "    divfolder = r\"IntermediateData\\newtextlarge\\div (%d)\"%i\n",
    "    if not os.path.exists(divfolder):\n",
    "        os.makedirs(divfolder)\n",
    "    \n",
    "    #this says what folder we want the txt files to go into\n",
    "    ap.env.workspace = divfolder\n",
    "    \n",
    "    #this says what DEM we want to use in the Surface Volume function\n",
    "    #####These are your input DEM files################################\n",
    "    #####Change these to the ones relevant to your area of interest####\n",
    "    dem_lm = r\"Data\\LM_div18\\dem_lm_z35_%d.TIF\"%i\n",
    "    \n",
    "    \n",
    "    # Process: Surface Volume\n",
    "    # this performs Surface Volume on the DEM at the specified points.\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_txt, \"BELOW\", \"0\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_txt, \"BELOW\", \"2\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_txt, \"BELOW\", \"4\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_6_txt, \"BELOW\", \"6\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_8_txt, \"BELOW\", \"8\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_10_txt, \"BELOW\", \"10\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_txt, \"BELOW\", \"1\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_txt, \"BELOW\", \"3\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_5_txt, \"BELOW\", \"5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_7_txt, \"BELOW\", \"7\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_9_txt, \"BELOW\", \"9\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_5_txt, \"BELOW\", \"0.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_5_txt, \"BELOW\", \"2.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_5_txt, \"BELOW\", \"4.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_6_5_txt, \"BELOW\", \"6.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_8_5_txt, \"BELOW\", \"8.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_5_txt, \"BELOW\", \"1.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_5_txt, \"BELOW\", \"3.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_5_5_txt, \"BELOW\", \"5.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_7_5_txt, \"BELOW\", \"7.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_9_5_txt, \"BELOW\", \"9.5\", \"1\", \"0\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TXT to CSV Maker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creates csv files for each div for 0-5m at 0.25m intervals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.857931Z",
     "start_time": "2020-10-21T03:38:50.544Z"
    }
   },
   "outputs": [],
   "source": [
    "surfacevolumefolder = r\"IntermediateData\\NewUngroupedSmallSurfaceVolume\"\n",
    "if not os.path.exists(surfacevolumefolder):\n",
    "    os.makedirs(surfacevolumefolder)\n",
    "    \n",
    "for div in range(num_divs):\n",
    "    l=[]\n",
    "    \n",
    "    files = glob.glob(r'IntermediateData\\newtext\\div (%d)\\*.txt'%div)      #glob shoves all the text files into one container\n",
    "\n",
    "    for file in files:\n",
    "        f = open(file,'r')\n",
    "        for i,line in enumerate(f):     #read through our list of text files as a series of strings\n",
    "            s = line.split(',')         #split up the strings\n",
    "            result = re.findall(r\"[-+]?\\d*\\.\\d+|\\d+\", s[-1])        #grabs floating point numbers out of strings, which is always the last item in the string\n",
    "            l.append(result)\n",
    "        f.close()\n",
    "\n",
    "    x = [] #heights\n",
    "    y = [] #volumes\n",
    "    for i in range(len(l)):             #the dummy list, l, has the values we want stored as every other item in the list\n",
    "        if(i%2==1):                     #so we need this mod 2 to get those indices\n",
    "            y.append(float(l[i][0]))\n",
    "            x.append(i/8 - .125)           #this fixes the incrementation of the heights\n",
    "\n",
    "    y = sorted(y)\n",
    "    z = zip(x,y) #with the volumes sorted we can now tie all the values together with their respective reference heights\n",
    "\n",
    "    small_ungrouped_csv = r'IntermediateData\\NewUngroupedSmallSurfaceVolume\\LMN_div18_new_ungrouped_small_div{:02d}.csv'.format(div)\n",
    "    with open(small_ungrouped_csv,'w') as out: #naming the CSV you're trying to create\n",
    "        write = csv.writer(out)\n",
    "        write.writerow(['height','volume'])\n",
    "        for i in list(z):\n",
    "            write.writerow(i)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creates csv files for each div from 0-10 m at 0.5m intervals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.859926Z",
     "start_time": "2020-10-21T03:38:50.547Z"
    }
   },
   "outputs": [],
   "source": [
    "surfacevolumefolder = r\"IntermediateData\\NewUngroupedLargeSurfaceVolume\"\n",
    "if not os.path.exists(surfacevolumefolder):\n",
    "    os.makedirs(surfacevolumefolder)\n",
    "\n",
    "for div in range(num_divs):\n",
    "   \n",
    "    l=[]\n",
    "    \n",
    "    files = glob.glob(r'IntermediateData\\newtextlarge\\div (%d)\\*.txt'%div)      #glob shoves all the text files into one container\n",
    "\n",
    "    for file in files:\n",
    "        f = open(file,'r')\n",
    "        for i,line in enumerate(f):     #read through our list of text files as a series of strings\n",
    "            s = line.split(',')         #split up the strings\n",
    "            result = re.findall(r\"[-+]?\\d*\\.\\d+|\\d+\", s[-1])        #grabs floating point numbers out of strings, which is always the last item in the string\n",
    "            l.append(result)\n",
    "        f.close()\n",
    "\n",
    "    x = [] #heights\n",
    "    y = [] #volumes\n",
    "    \n",
    "    \n",
    "    for i in range(len(l)):             #the dummy list, l, has the values we want stored as every other item in the list\n",
    "        if(i%2==1):                     #so we need this mod 2 to get those indices\n",
    "\n",
    "            y.append(float(l[i][0]))\n",
    "            x.append(i/4 - .25)           #this fixes the incrementation of the heights\n",
    "\n",
    "    y = sorted(y)\n",
    "    z = zip(x,y) #with the volumes sorted we can now tie all the values together with their respective reference heights\n",
    "\n",
    "    large_ungrouped_csv = r'IntermediateData\\NewUngroupedLargeSurfaceVolume\\LMN_div18_new_ungrouped_large_div_{:02d}.csv'.format(div)\n",
    "    with open(large_ungrouped_csv,'w') as out: #naming the CSV you're trying to create\n",
    "        write = csv.writer(out)\n",
    "        write.writerow(['height','volume'])\n",
    "        for i in list(z):\n",
    "            write.writerow(i)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CSV Combiner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.860923Z",
     "start_time": "2020-10-21T03:38:50.550Z"
    }
   },
   "outputs": [],
   "source": [
    "totalsurfacevolumefolder = r\"Data\\NewSurfaceVolumeCombined\"\n",
    "if not os.path.exists(totalsurfacevolumefolder):\n",
    "    os.makedirs(totalsurfacevolumefolder)\n",
    "\n",
    "for div in range(num_divs): #does it for each div\n",
    "\n",
    "    combined_csv = []\n",
    "    \n",
    "    #taking 0 to 3 in increments of 0.25\n",
    "    small_ungrouped_csv = r'IntermediateData\\NewUngroupedSmallSurfaceVolume\\LMN_div18_new_ungrouped_small_div{:02d}.csv'.format(div)\n",
    "    \n",
    "    with open(small_ungrouped_csv, newline='') as File:  \n",
    "        small_reader = csv.reader(File)\n",
    "        small_countrow = 0\n",
    "        for small_row in small_reader:\n",
    "            if small_countrow < 27 and small_row != []:\n",
    "                combined_csv.append(small_row)\n",
    "            small_countrow+=1\n",
    "\n",
    "    #taking 3.5 to 6.5 in increments of 0.5\n",
    "    large_ungrouped_csv = r'IntermediateData\\NewUngroupedLargeSurfaceVolume\\LMN_div18_new_ungrouped_large_div_{:02d}.csv'.format(div)\n",
    "    \n",
    "    with open(large_ungrouped_csv, newline='') as File:  \n",
    "        large_reader = csv.reader(File)\n",
    "        large_countrow = 0\n",
    "        for large_row in large_reader:\n",
    "            if large_countrow > 14 and large_countrow < 32  and large_row != []:\n",
    "                combined_csv.append(large_row)\n",
    "            large_countrow+=1\n",
    "\n",
    "#combined csv is now a list with header and 20 values of height\n",
    "\n",
    "#now we create a new csv\n",
    "###################################### Change name of file here * #################################\n",
    "    combined_ungrouped_csv = r'Data\\NewSurfaceVolumeCombined\\LMN_div18_new_{:02d}.csv'.format(div)\n",
    "    with open(combined_ungrouped_csv,'w') as out:\n",
    "        write = csv.writer(out)\n",
    "        for i in list(combined_csv):\n",
    "            write.writerow(i)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This produces 1 csv file for each of the divs in your area of interest. Now we can put NewSurfaceVolumeCombined into the Flood Estimate code!"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Grouped Divs"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Inputs to these cells are DEMs that have been grouped using ArcGIS's Mosaic to Raster tool with the 2 divisions north of them and the 2 divisions south of them. The name of the DEM is the centermost division."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Surface Volume Processor"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Process for 0-5m at 0.25 m intervals for each div"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.862917Z",
     "start_time": "2020-10-21T03:38:50.553Z"
    }
   },
   "outputs": [],
   "source": [
    "ele_0_txt = \"\\\\ele_0.txt\"; ele_1_txt = \"\\\\ele_1.txt\"; ele_2_txt = \"\\\\ele_2.txt\" \n",
    "ele_3_txt = \"\\\\ele_3.txt\"; ele_4_txt = \"\\\\ele_4.txt\"; ele_5_txt = \"\\\\ele_5.txt\"\n",
    "ele_0_5_txt = \"\\\\ele_0_5.txt\"; ele_1_5_txt = \"\\\\ele_1_5.txt\"; ele_2_5_txt = \"\\\\ele_2_5.txt\" \n",
    "ele_3_5_txt = \"\\\\ele_3_5.txt\"; ele_4_5_txt = \"\\\\ele_4_5.txt\"\n",
    "ele_0_25_txt = \"\\\\ele_0_25.txt\"; ele_1_25_txt = \"\\\\ele_1_25.txt\"; ele_2_25_txt = \"\\\\ele_2_25.txt\" \n",
    "ele_3_25_txt = \"\\\\ele_3_25.txt\"; ele_4_25_txt = \"\\\\ele_4_25.txt\"\n",
    "ele_0_75_txt = \"\\\\ele_0_75.txt\"; ele_1_75_txt = \"\\\\ele_1_75.txt\"; ele_2_75_txt = \"\\\\ele_2_75.txt\" \n",
    "ele_3_75_txt = \"\\\\ele_3_75.txt\"; ele_4_75_txt = \"\\\\ele_4_75.txt\"\n",
    "\n",
    "for i in range(num_divs):\n",
    "    \n",
    "    #this creates the destination folder for the txt files to go into if it hasn't been created yet\n",
    "    #you can comment this out if the folders have been created since it takes a while to run\n",
    "    divfolder = r\"IntermediateData\\newtextgrouped\\div (%d)\"%i\n",
    "    if not os.path.exists(divfolder):\n",
    "        os.makedirs(divfolder)\n",
    "    \n",
    "    #this says what folder we want it to go into\n",
    "    ap.env.workspace = divfolder\n",
    "    \n",
    "    #this says what DEM we want to use in the Surface Volume function\n",
    "    #####These are your input DEM files################################\n",
    "    #####Change these to the ones relevant to your area of interest####\n",
    "    dem_lm = r\"Data\\LM_div18_grouped\\groupRaster_{:02d}.TIF\".format(i)\n",
    "\n",
    "    \n",
    "    # Process: Surface Volume\n",
    "    # this performs Surface Volume on the DEM at the specified points.\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_txt, \"BELOW\", \"0\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_txt, \"BELOW\", \"1\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_txt, \"BELOW\", \"2\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_txt, \"BELOW\", \"3\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_txt, \"BELOW\", \"4\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_5_txt, \"BELOW\", \"5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_5_txt, \"BELOW\", \"0.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_5_txt, \"BELOW\", \"1.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_5_txt, \"BELOW\", \"2.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_5_txt, \"BELOW\", \"3.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_5_txt, \"BELOW\", \"4.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_25_txt, \"BELOW\", \"0.25\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_25_txt, \"BELOW\", \"1.25\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_25_txt, \"BELOW\", \"2.25\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_25_txt, \"BELOW\", \"3.25\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_25_txt, \"BELOW\", \"4.25\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_75_txt, \"BELOW\", \"0.75\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_75_txt, \"BELOW\", \"1.75\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_75_txt, \"BELOW\", \"2.75\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_75_txt, \"BELOW\", \"3.75\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_75_txt, \"BELOW\", \"4.75\", \"1\", \"0\")\n",
    "    "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Process for 0-10m at 0.5m intervals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.864912Z",
     "start_time": "2020-10-21T03:38:50.556Z"
    }
   },
   "outputs": [],
   "source": [
    "ele_0_txt = \"\\\\ele_0.txt\"; ele_2_txt = \"\\\\ele_2.txt\"; ele_4_txt = \"\\\\ele_4.txt\" \n",
    "ele_6_txt = \"\\\\ele_6.txt\"; ele_8_txt = \"\\\\ele_8.txt\"; ele_10_txt = \"\\\\ele_10.txt\"\n",
    "ele_1_txt = \"\\\\ele_1.txt\"; ele_3_txt = \"\\\\ele_3.txt\"; ele_5_txt = \"\\\\ele_5.txt\" \n",
    "ele_7_txt = \"\\\\ele_7.txt\"; ele_9_txt = \"\\\\ele_9.txt\"\n",
    "ele_0_5_txt = \"\\\\ele_0_5.txt\"; ele_2_5_txt = \"\\\\ele_2_5.txt\"; ele_4_5_txt = \"\\\\ele_4_5.txt\" \n",
    "ele_6_5_txt = \"\\\\ele_6_5.txt\"; ele_8_5_txt = \"\\\\ele_8_5.txt\"\n",
    "ele_1_5_txt = \"\\\\ele_1_5.txt\"; ele_3_5_txt = \"\\\\ele_3_5.txt\"; ele_5_5_txt = \"\\\\ele_5_5.txt\" \n",
    "ele_7_5_txt = \"\\\\ele_7_5.txt\"; ele_9_5_txt = \"\\\\ele_9_5.txt\"\n",
    "    \n",
    "for i in range(num_divs):\n",
    "    \n",
    "    #this creates the destination folder for the txt files to go into if it hasn't been created yet\n",
    "    #you can comment this out if the folders have been created since it takes a while to run\n",
    "    divfolder = r\"IntermediateData\\newtextgroupedlarge\\div (%d)\"%i\n",
    "    if not os.path.exists(divfolder):\n",
    "        os.makedirs(divfolder)\n",
    "    \n",
    "    #this says what folder we want the txt files to go into\n",
    "    ap.env.workspace = divfolder\n",
    "    \n",
    "    #this says what DEM we want to use in the Surface Volume function\n",
    "    #####These are your input DEM files################################\n",
    "    #####Change these to the ones relevant to your area of interest####\n",
    "    dem_lm = r\"Data\\LM_div18_grouped\\groupRaster_{:02d}.TIF\".format(i)\n",
    "    \n",
    "    \n",
    "    # Process: Surface Volume\n",
    "    # this performs Surface Volume on the DEM at the specified points.\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_txt, \"BELOW\", \"0\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_txt, \"BELOW\", \"2\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_txt, \"BELOW\", \"4\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_6_txt, \"BELOW\", \"6\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_8_txt, \"BELOW\", \"8\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_10_txt, \"BELOW\", \"10\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_txt, \"BELOW\", \"1\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_txt, \"BELOW\", \"3\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_5_txt, \"BELOW\", \"5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_7_txt, \"BELOW\", \"7\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_9_txt, \"BELOW\", \"9\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_0_5_txt, \"BELOW\", \"0.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_2_5_txt, \"BELOW\", \"2.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_4_5_txt, \"BELOW\", \"4.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_6_5_txt, \"BELOW\", \"6.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_8_5_txt, \"BELOW\", \"8.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_1_5_txt, \"BELOW\", \"1.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_3_5_txt, \"BELOW\", \"3.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_5_5_txt, \"BELOW\", \"5.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_7_5_txt, \"BELOW\", \"7.5\", \"1\", \"0\")\n",
    "    ap.SurfaceVolume_3d(dem_lm, ele_9_5_txt, \"BELOW\", \"9.5\", \"1\", \"0\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## TXT to CSV Maker"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creates csv files for each div for 0-5m at 0.25m intervals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.865909Z",
     "start_time": "2020-10-21T03:38:50.559Z"
    }
   },
   "outputs": [],
   "source": [
    "surfacevolumefolder = r\"IntermediateData\\NewGroupedSmallSurfaceVolume\"\n",
    "if not os.path.exists(surfacevolumefolder):\n",
    "    os.makedirs(surfacevolumefolder)\n",
    "    \n",
    "for div in range(num_divs):\n",
    "    l=[]\n",
    "    \n",
    "    files = glob.glob(r'IntermediateData\\newtextgrouped\\div (%d)\\*.txt'%div)      #glob shoves all the text files into one container\n",
    "\n",
    "    for file in files:\n",
    "        f = open(file,'r')\n",
    "        for i,line in enumerate(f):     #read through our list of text files as a series of strings\n",
    "            s = line.split(',')         #split up the strings\n",
    "            result = re.findall(r\"[-+]?\\d*\\.\\d+|\\d+\", s[-1])        #grabs floating point numbers out of strings, which is always the last item in the string\n",
    "            l.append(result)\n",
    "        f.close()\n",
    "\n",
    "    x = [] #heights\n",
    "    y = [] #volumes\n",
    "    for i in range(len(l)):             #the dummy list, l, has the values we want stored as every other item in the list\n",
    "        if(i%2==1):                     #so we need this mod 2 to get those indices\n",
    "            y.append(float(l[i][0]))\n",
    "            x.append(i/8 - .125)           #this fixes the incrementation of the heights\n",
    "\n",
    "    y = sorted(y)\n",
    "    z = zip(x,y) #with the volumes sorted we can now tie all the values together with their respective reference heights\n",
    "\n",
    "    small_grouped_csv = r'IntermediateData\\NewGroupedSmallSurfaceVolume\\LMN_div18_new_grouped_small_div{:02d}.csv'.format(div)\n",
    "    with open(small_grouped_csv,'w') as out: #naming the CSV you're trying to create\n",
    "        write = csv.writer(out)\n",
    "        write.writerow(['height','volume'])\n",
    "        for i in list(z):\n",
    "            write.writerow(i)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Creates csv files for each div from 0-10 m at 0.5m intervals"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.867904Z",
     "start_time": "2020-10-21T03:38:50.562Z"
    }
   },
   "outputs": [],
   "source": [
    "surfacevolumefolder = r\"IntermediateData\\NewGroupedLargeSurfaceVolume\"\n",
    "if not os.path.exists(surfacevolumefolder):\n",
    "    os.makedirs(surfacevolumefolder)\n",
    "\n",
    "for div in range(num_divs):\n",
    "   \n",
    "    l=[]\n",
    "    \n",
    "    files = glob.glob(r'IntermediateData\\newtextgroupedlarge\\div (%d)\\*.txt'%div)      #glob shoves all the text files into one container\n",
    "\n",
    "    for file in files:\n",
    "        f = open(file,'r')\n",
    "        for i,line in enumerate(f):     #read through our list of text files as a series of strings\n",
    "            s = line.split(',')         #split up the strings\n",
    "            result = re.findall(r\"[-+]?\\d*\\.\\d+|\\d+\", s[-1])        #grabs floating point numbers out of strings, which is always the last item in the string\n",
    "            l.append(result)\n",
    "        f.close()\n",
    "\n",
    "    x = [] #heights\n",
    "    y = [] #volumes\n",
    "    \n",
    "    \n",
    "    for i in range(len(l)):             #the dummy list, l, has the values we want stored as every other item in the list\n",
    "        if(i%2==1):                     #so we need this mod 2 to get those indices\n",
    "\n",
    "            y.append(float(l[i][0]))\n",
    "            x.append(i/4 - .25)           #this fixes the incrementation of the heights\n",
    "\n",
    "    y = sorted(y)\n",
    "    z = zip(x,y) #with the volumes sorted we can now tie all the values together with their respective reference heights\n",
    "\n",
    "    large_grouped_csv = r'IntermediateData\\NewGroupedLargeSurfaceVolume\\LMN_div18_new_grouped_large_div_{:02d}.csv'.format(div)\n",
    "    with open(large_grouped_csv,'w') as out: #naming the CSV you're trying to create\n",
    "        write = csv.writer(out)\n",
    "        write.writerow(['height','volume'])\n",
    "        for i in list(z):\n",
    "            write.writerow(i)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## CSV Combiner"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "ExecuteTime": {
     "end_time": "2020-10-21T03:38:50.868901Z",
     "start_time": "2020-10-21T03:38:50.565Z"
    }
   },
   "outputs": [],
   "source": [
    "totalsurfacevolumefolder = r\"Data\\NewSurfaceVolumeGrouped\"\n",
    "if not os.path.exists(totalsurfacevolumefolder):\n",
    "    os.makedirs(totalsurfacevolumefolder)\n",
    "\n",
    "for div in range(num_divs): #does it for each div\n",
    "\n",
    "    combined_csv = []\n",
    "    \n",
    "    #taking 0 to 3 in increments of 0.25\n",
    "    small_grouped_csv = r'IntermediateData\\NewGroupedSmallSurfaceVolume\\LMN_div18_new_grouped_small_div{:02d}.csv'.format(div)\n",
    "    \n",
    "    with open(small_grouped_csv, newline='') as File:  \n",
    "        small_reader = csv.reader(File)\n",
    "        small_countrow = 0\n",
    "        for small_row in small_reader:\n",
    "            if small_countrow < 27 and small_row != []:\n",
    "                combined_csv.append(small_row)\n",
    "            small_countrow+=1\n",
    "\n",
    "    #taking 3.5 to 6.5 in increments of 0.5\n",
    "    large_grouped_csv = r'IntermediateData\\NewGroupedLargeSurfaceVolume\\LMN_div18_new_grouped_large_div_{:02d}.csv'.format(div)\n",
    "    \n",
    "    with open(large_grouped_csv, newline='') as File:  \n",
    "        large_reader = csv.reader(File)\n",
    "        large_countrow = 0\n",
    "        for large_row in large_reader:\n",
    "            if large_countrow > 14 and large_countrow < 32  and large_row != []:\n",
    "                combined_csv.append(large_row)\n",
    "            large_countrow+=1\n",
    "\n",
    "#combined csv is now a list with header and 20 values of height\n",
    "\n",
    "#now we create a new csv\n",
    "############################just change the name of this guy here * ######################################\n",
    "    combined_grouped_csv = r'Data\\NewSurfaceVolumeGrouped\\LMN_div18_new_grouped_{:02d}.csv'.format(div)\n",
    "    with open(combined_grouped_csv,'w') as out:\n",
    "        write = csv.writer(out)\n",
    "        for i in list(combined_csv):\n",
    "            write.writerow(i)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This produces 1 \"grouped\" csv file for each of the divs in your area of interest. Now we can put NewSurfaceVolumeGrouped into the Flood Estimate code!"
   ]
  }
 ],
 "metadata": {
  "hide_input": false,
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {},
   "toc_section_display": true,
   "toc_window_display": false
  },
  "varInspector": {
   "cols": {
    "lenName": 16,
    "lenType": 16,
    "lenVar": 40
   },
   "kernels_config": {
    "python": {
     "delete_cmd_postfix": "",
     "delete_cmd_prefix": "del ",
     "library": "var_list.py",
     "varRefreshCmd": "print(var_dic_list())"
    },
    "r": {
     "delete_cmd_postfix": ") ",
     "delete_cmd_prefix": "rm(",
     "library": "var_list.r",
     "varRefreshCmd": "cat(var_dic_list()) "
    }
   },
   "types_to_exclude": [
    "module",
    "function",
    "builtin_function_or_method",
    "instance",
    "_Feature"
   ],
   "window_display": false
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}