FES2022 released!

[robbibt]

Very exciting update today: FES2022 has finally been released! 🎉

Product page: https://www.aviso.altimetry.fr/en/data/products/auxiliary-products/global-tide-fes.html

User guide: https://www.aviso.altimetry.fr/fileadmin/documents/data/tools/hdbk_FES2022.pdf

Seems like some pretty nice improvements over coastal areas particularly compared to FES2014!

[robbibt]

Woah, just saw that there is already support for this - amazing! 🤯
https://github.com/tsutterley/pyTMD/releases/tag/2.1.2

[tsutterley]

Yep! Super excited to support FES2022!

[robbibt]

Hey @tsutterley, trying this out for the first time, I seem to be having some issues calculating tides from FES2022. This works fine for FES2014:

from pyTMD.compute import tide_elevations
import pandas as pd

tide_elevations(
    x=[122.21],
    y=[-18.20],
    delta_time=pd.date_range(start="2022-09-01", end="2022-09-30", freq="1h"),
    DIRECTORY="/gdata1/data/tide_models/",
    MODEL="FES2014",
    EPSG=4326,
    TIME="datetime",
)

However if I try the same with FES2022, I get TypeError: ufunc 'bitwise_and' not supported for the input types, and the inputs could not be safely coerced to any supported types according to the casting rule ''safe'':

tide_elevations(
    x=[122.21],
    y=[-18.20],
    delta_time=pd.date_range(start="2022-09-01", end="2022-09-30", freq="1h"),
    DIRECTORY="/gdata1/data/tide_models/",
    MODEL="FES2022",
    EPSG=4326,
    TIME="datetime",
)

I downloaded the FES2022 files with aviso-fes-tides-py, and my directory looks like this:

I'm on the latest version:

[tsutterley]

Tracked this down (even though it is fixed in pyTMD 😵‍💫). The FES2022 files have an improperly set valid_max attribute. Should be 360 but is set to 0. I could fix it in the download script, but I'm undecided if it is worth the effort.

[tsutterley]

Hey @robbibt, super weird. For some reason the FES2022 netCDF files have the longitude arrays masked. I have a fix in #311.

[tsutterley]

For sure I'll do it tomorrow 👍👍

[robbibt]

One last request (should have mentioned this yesterday!) - if we could get a conda update too that would be fantastic - the last one there is 2.1.1. 🙂
https://anaconda.org/conda-forge/pytmd

[tsutterley]

for sure I'm just waiting for the PR from conda-forge to merge. I can do it manually tomorrow if I don't get an update for the recipe.

[tsutterley]

sometimes you just have to do things manually 😆 conda is bumped to 2.1.3

[robbibt]

Awesome, thanks!

[robbibt]

Just two related questions after exploring FES2022 a little more:

1. Compared to previous releases, it looks like FES2022 has done a funky thing with their "extrapolated" layers, where they have injected "0" values for inland lakes to prevent the models from running over those environments. Annoyingly for our applications, this also includes coastal wetland ecosystems that we most want to model! 😂 I just wanted to verify that PyTMD is able to handle these correctly given they're coded as "0" values and not standard nodata?

2. Currently I've been testing differences between the standard and "extrapolated" versions of FES2022 by manually renaming the "ocean_tide_extrapolated" folder to "ocean_tide". Is there an easier way via the PyTMD API? (I know there's the "extrapolate" param, but I believe that's for when points fall completely out of the modelling domain, not whether or not to use the pre-generated extrapolated data)

[robbibt]

Just some further analysis: I suspect that pyTMD isn't currently accounting for those masked-out zero values. I've compared results from the standard FES2022 data against the "extrapolated" data for some of our tide gauges that sit on top of those masked pixels (e.g. GESLA Wyndham site code 63090; 128.101, -15.4533):

And we seem to be getting back highly dampened tide amplitudes in the "extrapolated" data compared to the standard data:

Even if there's a pyTMD work-around, I might also raise this issue with AVISO - hardcoding those "lakes" in the constituent files seems to defeat one of the main purposes of the "extrapolated" data (being able to generate a best-estimate of tides outside of the modelling domain). Especially annoying when many of those "lakes" aren't lakes at all, but tide-influenced coastal ecosystems!

[tsutterley]

Hey @robbibt,
Thanks for being my FES2022 beta tester 😅 🤦 sorry for the bumpy start!

I agree with you, I'd rather not zero out these inland and coastal water bodies with the extrapolated FES data. So in #314 I'm setting the mask for these values so that that they can be rightfully interpolated (when using the extrapolation routines within pyTMD).

Good question. I currently support models of a known type (FES/GOT/OTIS etc) that aren't in io.model by using definition files. I didn't really want to "support" FES2022-extrapolated out of the box I wanted to do more work to investigate the model. But for this case, I'm including the definition files for the FES2022-extrapolated model in #314 (in both tab-delimited and JSON formats). These use the "glob" functionality to read all the model files supplied in the tide model directory. The only change you should have to do on your end is use --definition-file model_FES2022b_extrapolated.def or DEFINITION_FILE='model_FES2022b_extrapolated.def' instead of --tide FES2022 or MODEL='FES2022'. Still include the directory variable as before.

The other recent update (from a PR merged yesterday) is to enable "on-the-fly" cropping of the model files. This should greatly improve the computational times for some of these higher resolution models. I think I have covered the prime meridian/dateline cases. If you want to test those out, you can set --crop or CROP=True. Hope this helps 👍

[robbibt]

No worries, super appreciative to have FES2022 support so early - I think we're all learning about its quirks as we go! Thanks for the masking fix, I'll test the new code out on Monday.

The other two changes sound fantastic too: we already do an initial manual crop on our end (running our analysis on copies of the files cropped to Australia), but the on-the-fly cropping should make it far faster (from what I've gathered, the slowest part is actually the interpolation vs. loading the constituent files, and the interpolation runs far faster on smaller inputs).

JSON inputs for definition files sounds great! Would this potentially make it possible to define definitions "in code", e.g. pass in something like a Python dictionary containing all required params, rather than having to load the definition from file? That would be really helpful for reproducibility - at the moment we store some custom def files (e.g. for FES2012) next to our model data, but that only works for our usage, not for external users who might want to run our code on their systems.

[robbibt]

OK, looks like the latest version does a great job of resolving issues over those masked lakes: we're no longer seeing artificially dampened tides after the fix:

Compared against GESLA tide gauges, the "extrapolated" model now produces results that are very close to the standard model:

Before:

After:

[tsutterley]

Yes! Defining the JSON files as in-memory io.BytesIO objects is definitely possible and I think should already work if you json.dump your dictionary into one. That was at least the goal. Also yes, moving towards just being able to define everything as kwargs. Just moving slowly 😂

Ok perfect. Glad the dampening issue got fixed. THANK you for pointing it out! Was an interesting decision by the FES team to zero out the coastal water bodies.

[tsutterley]

based on your tests, should I put out a new release (2.1.4) so you can get things done?

[robbibt]

No rush, but that would be fantastic if you get a chance. It's all working nicely for me so far!

[tsutterley]

on it 👍