Long Queue for ERA5 download

[pbrotoisworo]

Hi, I'm just wondering if there is another way to download the required ERA5 files for correct_troposphere step? Sometimes the queue using pyaps is very slow and I have many grib files to download.

[falkamelung]

Yes, this is frustrating indeed. We have come so far that the bottleneck for InSAR processing is ERA5 download. I wonder how the ASF/Hyp3 folks deal with this. @jhkennedy , do you guys have a copy of ERA5 at at the ASF? If so it would be a great service to make it available to the MintPy community. Or does JPL has a copy that could be shared? I suggested this a while ago to WinSAR but they did not have staff at the time to investigate this. We could try again.

[yuankailiu]

Agree! The last time when I tried this, in 9 hours, I only got 38 out of 158 grb files.
But I have also been requesting from CDS, rather than ECMWF. Do you also try ECMWF and have a shorter queue time?

[pbrotoisworo]

@yuankailiu , according to the MintPy log messages ECMWF and CDS are the same? Or am i misunderstanding something? This is just using the default settings in the mintpy config file.

INFO: You are using the latest ECMWF platform for downloading datasets: https://cds.climate.copernicus.eu/api/v2
Downloading 1 of 27: C:\project\mintpy_output_small\ERA5\ERA5_S40_S20_E120_E140_20170403_21.grb 
{'product_type': 'reanalysis', 'format': 'grib', 'variable': ['geopotential', 'temperature', 'specific_humidity'], 'pressure_level': ['1', '2', '3', '5', '7', '10', '20', '30', '50', '70', '100', '125', '150', '175', '200', '225', '250', '300', '350', '400', '450', '500', '550', '600', '650', '700', '750', '775', '800', '825', '850', '875', '900', '925', '950', '975', '1000'], 'year': '2017', 'month': '04', 'day': '03', 'time': '21:00', 'area': '-20.00/120.00/-40.00/140.00'}
2022-05-01 18:30:24,627 INFO Welcome to the CDS
2022-05-01 18:30:24,627 INFO Sending request to https://cds.climate.copernicus.eu/api/v2/resources/reanalysis-era5-pressure-levels
2022-05-01 18:30:24,909 INFO Request is queued

[yuankailiu]

You are right. They are the same. Sorry.

[jhkennedy]

@falkamelung my team doesn't currently have a copy of ERA5 and generally run into the same issues downloading grb files, though maybe @Alex-Lewandowski has some thoughts?

I think ASF may be interested in hosting a copy of the dataset, but getting a full copy from CDS/ECMWF and keeping it up to date looks like a pretty big lift. Is there any alternative sources that'd work?

I know GEE has the daily aggregates here but they don't have the hourly data. Not sure if the MS Planetary Computer has it...

[yunjunz]

I would recommend reporting to ECMWF directly so that they are aware of the issue and could possibly fix it. There is a user forum at:

https://confluence.ecmwf.int/display/CUSF/C3S+Climate+Data+Store+API+User+Forum

The part of pyaps code to download ERA5 using CDSAPI is here for reference:

https://github.com/insarlab/PyAPS/blob/c4c623e85a63c51b039bab2007224a5be3ce4045/src/pyaps3/autoget.py#L105-L133

[pbrotoisworo]

Thanks for the recommendation @yunjunz and sharing the relevant pyaps code.

In my case, I'm planning on running MintPy on AWS. So my workaround for now is to run the pyaps code separately on my personal computer otherwise the AWS instance would just sit there using money.

[Alex-Lewandowski]

Joe, to answer your earlier question we do not have a good work around for quicker access to ERA5. We have plans to update the OpenSARlab MintPy notebook to optionally skip the tropospheric correction step, but that hasn't happened yet. Alex

…

On Sun, May 8, 2022 at 5:45 AM Yuan-Kai Liu ***@***.***> wrote: You are right. They are the same. Sorry. — Reply to this email directly, view it on GitHub <#766 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AJBHEYEK7VUUF35VS4SFWN3VI7AP3ANCNFSM5UZ24P6Q> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- *Alex Lewandowski* *Software Engineer* *Alaska Satellite Facility* *UAF Geophysical Institute* *text: +1-907-378-7595* *he, him, his* *I acknowledge that I am working upon the ancestral land of Troth Yeddha’, home of the Lower Tanana and Dené people. Their stewardship of the land over thousands of years makes my work at UAF possible. *

[scottstanie]

Just wondering, is the reason that PyAPS uses ERA5 because it's the best/highest resolution model? Or is it because the original version of the code chose that model?

I haven't done extensive testing between the possible weather models to correct, but when I was playing around with https://github.com/dbekaert/RAiDER , the GMAO model was much quicker to download than ERA5. I think it took 30-60 minutes to get 150 dates over a big area.

[yunjunz]

The chosen models are inherited from Jolivet et al. (2011; 2014) papers, which was ERA-Interim (and NARR and MERRA), then shifted to ERA5. Here (https://hal.archives-ouvertes.fr/hal-02414275/) is the latest publication that I am aware of from Romain's group on this topic.

As for raider, we should add the support to mintpy directly, if it is ready to use. Contributions are very welcomed. @dbekaert may have insights on this.

[jhkennedy]

Just following up on this with some updates:

• raider is now on conda-forge so should be ready to use by mintpy -- there's quite a bit of work going on right now there as well, so in another month there should be a newer version with a lot of improvements
• ERA5 is now on Google Cloud as an analysis-ready, cloud-optimized dataset: https://github.com/google-research/arco-era5

[yunjunz]

Thanks for the update @jhkennedy. Both raider and arco-era5 are very exciting developments!

Based on a quick look at RAiDER_tutorial repo, I did not find an API example to feed raster geometry to calculate the InSAR tropo delay yet. It will be great to involve the raider developers (@dbekaert, @sssangha, @piyushrpt, etc.) in the discussion to get their opinion and/or helps.

I won't have time to integrate raider with mintpy in recent months, but I am more than happy to help and answer any questions if anyone is interested in contributing to this.