Finding grid files on NCAR or NERSC systems

[pgoddard08]

Hello,
I need to find the grid for E3SMv2 output. I see in "Plotting E3SMv2 Output with UXarray", this path is used:
"grid_file = (
"/glade/campaign/cisl/vast/uxarray/data/e3sm_keeling/E3SM_grid/ne30pg2_grd.nc"
)

I don't have permissions for this path. How can I get the ne30pg2_grd.nc file? I have access both to NCAR and NERSC systems.
Thanks!
Paul

[rajeeja]

Thanks for the question. Where do you see it?

[xylar]

https://uxarray.readthedocs.io/en/latest/examples/visualization/e3sm-load-viz.html

[xylar]

Also:
https://uxarray.readthedocs.io/en/stable/examples/e3sm-calc-workflow.html

[xylar]

The files are presumably at NCAR somewhere and if one has an account, one can run the notebooks. But I agree with @pgoddard08 that it would be nice to host the files somewhere public so someone without an NCAR account can also run the notebooks.

[philipc2]

Hi @pgoddard08

I can coordinate with @rytam2 to make these files readable on the the NCAR systems.

Storing these publicly in the cloud somewhere as @xylar mentioned is something we've been wanted to do for a while now. If you have any suggestions, we'd be more than happy to look into it.

[pgoddard08]

Great, thanks @philipc2 and @xylar . I really appreciate your help. Please let me know when these files are either readable on NCAR or housed somewhere for the general public. I've found such grids used for this example and used for general regridding difficult to find, so a public domain from the E3SM website would be ideal. Many thanks!

[philipc2]

@xylar

I've stored the grid file & a selection of data variables in the following google drive folder.

I've included the data variables both in zarr and netcdf file formats. Feel free to use whichever format is better suited for the data server.

Thanks again!

[xylar]

@philipc2, I'm happy to host both but wondering about the zarr file, which is actually a directory on GoogleDrive. It seems completely unwieldy for folks to download an entire directory over https. I don't have much experience with zarr format but is there a way to bundle it up more conveniently?

[xylar]

In any case, I have the data here:
https://web.lcrc.anl.gov/public/e3sm/diagnostics/uxarray_data/ENSO_ctl_1std/
If you want me to tar up the zarr data or something like that, let me know. Feel free to point to that location in your documentation, etc.

[philipc2]

Hi @xylar

Thank you so much for uploading the data!

It seems completely unwieldy for folks to download an entire directory over https. I don't have much experience with zarr format but is there a way to bundle it up more conveniently?

Zarr files are stored as a directory instead of as a single dataset. Since it is already compressed, I believe zipping it should also be an appropriate way to store it.

Pinging @erogluorhan if he has anything to add.

[erogluorhan]

Hi @xylar , thanks a lot for your help with storing these on E3SM public data server!

Zarr should already be chunked (I don't know the details of how this zarr file was created though) and compressed. While it is optimized for object storage systems, it should also be suitable for being efficiently accessed via https without downloading the entire directory AFAIK (e.g. xarray's open_dataset() using Zarr as the engine).

That said, if everything works as expected, hosting such a relatively large dataset in the zarr format than netcdf would be beneficial for future analysis workflows (I am not against keeping the netcdf file that you already put there though).

Please note @philipc2 : Some Zarr-based workflows mentioned metadata being stored in additional Zarr formats such as .zgroup etc. I can see the separate directories for several variables in this Zarr directory you created; does each of them have the metadata embedded into itself? Otherwise, should we have metadata explicitly created?

[xylar]

Okay, I'm in the process of adding a E3SM_ENSO_ctl_keeling_eam_h0_0006-12.zarr.zip file.

[xylar]

I'm also going to delete the zarr directory because I don't think it will be super useful and it's a bit of redundant storage.