How/where will we store our data?

[jsadler2]

I'm wondering what the best place is to store our data.

I think the options are

1. locally
2. Caldera
3. CHS S3

[jsadler2]

My vote currently is currently Caldera. I don't like "local" b/c things can get out of sync. I'm open to S3, but I'm not sure that there are many advantages over Caldera. I'd like to hear your thoughts.

[lekoenig]

@galengorski and @lekoenig - are you both comfortable working on Tallgrass and storing our data in Caldera?

@jsadler2 Yes, this plan sounds fine to me. I haven't used Tallgrass/Caldera yet, so I don't have firm preferences on data storage options for HPC. I'm also open to this plan even if the salinity project uses S3 since gaining experience with these different HPC configurations would help me become more familiar with the pros/cons of each.

[galengorski]

@galengorski and @lekoenig - are you both comfortable working on Tallgrass and storing our data in Caldera?

@jsadler2 This sounds good to me, I just requested an account on tallgrass and denali, so that should be coming soon.

[jzwart]

@jds485 pipeline integration is the main benefit I was thinking of. If using dynamic branching targets (e.g. model runs, sites, etc..), it'd be quite simple to push / pull the correct files rather than doing that manually. When putting files on HPC manually in the past, I know I've occasionally forgotten which files are needed

[jsadler2]

Okay @galengorski and @lekoenig - let's DO it! ... as in let's go with Caldera/Tallgrass. We can revisit if for some reason it's not working for us.

[galengorski]

sorry to jump in late with this, but yes tallgrass/caldera sound great. I requested an account over a week ago now, so hopefully it should be coming through pretty soon here.

[jsadler2]

Also, I'm wondering which format to store the data in. I'm partial to netcdf, personally.

[jsadler2]

Coding independently but sharing data products

@galengorski and @lekoenig - I've been thinking about how to

• avoid stepping on each others toes in coding, while also
• staying in sync with each others data products

Maybe there is an established way of doing this already. If there is, maybe @aappling-usgs you know about it??

Here's my proposed solution:

1. for code we all fork this repo and work on it in our own "homes" 🏡 (/home/jsadler/ for me) and make PRs when ready to this repo
2. for data we set up the output path for our pipelines (targets and maybe snakemake) to our pump folder* (/caldera/projects/usgs/water/impd/pump/drb-do/drb-do-ml/)

* The data folder is not a git repo and it does not contain any code. It's the products of our pipeline (the code that will be committed to this repo). It will have the same folder structure, as the git repo but will only have the out folders. So we will commit all of the drb-do-ml/*/src/* files to the repo, but all of the outputs (drb-do-ml/*/out/*) will go to the /caldera/projects/usgs/water/impd/pump/drb-do/drb-do-ml/ folder.

I think this way we can all work on code independently but work from the same data files. One other thing that we could do is add a jsadler or scratch data folder (eg, /caldera/projects/usgs/water/impd/pump/drb-do/drb-do-ml/jsadler) so that we can have a bit of independence on the data side too if we are doing some experimental/prototyping work and not ready for others to use the data.

note - this approach assumes that we would never want to commit any of our outputs because they will be in a different folder. If we did end up wanting to commit outputs to the repo we'd have to produce them also in a out (or similar) folder in the repo as well as in the shared, caldera data folder.