HyP3 + Mintpy inquiries:- 1) How to unzip in python 2)how to run mintpy?

[MohamedAbdrabou2022]

Description of the desired feature

Is your feature request related to a problem? Please describe

Describe the solution you'd like

Describe alternatives you have considered

Additional context

Are you willing to help implement and maintain this feature?

• Yes
• No

[welcome[bot]]

👋 Thanks for opening your first issue here! Please filled out the template with as much details as possible. We appreciate that you took the time to contribute!
Make sure you read our contributing guidelines.

[asjohnston-asf]

The hyp3_sdk package supports working with HyP3 jobs using Python, and includes a utility function for unzipping the files after they've been downloaded. See the hyp3_sdk.util.extract_zipped_product() function in our SDK reference

>>> import hyp3_sdk
>>> hyp3_sdk.util.extract_zipped_product('S1AA_20220927T164913_20221102T164913_VVR036_INT80_G_ueF_306C.zip', delete=True)
PosixPath('S1AA_20220927T164913_20221102T164913_VVR036_INT80_G_ueF_306C')

That function uses the Python ZipFile module, in case you'd prefer to use the module directly.

[asjohnston-asf]

ASF also provides a Time series analysis with HyP3 and MintPy Notebook in our Tutorials that demonstrates the entire workflow in Python, from finding InSAR pairs, running those jobs in HyP3, downloading the interferograms, and generating the time series using MintPy.

[MohamedAbdrabou2022]

I downloaded the data and unzipped them, what is the next steps for generating the time series using MintPy?

[MohamedAbdrabou2022]

@asjohnston-asf

[yunjunz]

Since this is more of a Q&A (not a code bug or feature request), I converted it here to the GitHub discussion.

[asjohnston-asf]

Once you've downloaded and unzipped them, you should be ready for Step 3 in the tutorial notebook: https://nbviewer.org/github/ASFHyP3/hyp3-docs/blob/main/docs/tutorials/hyp3_insar_stack_for_ts_analysis.ipynb#3.-Time-series-Analysis-with-MintPy

[MohamedAbdrabou2022]

I have to start directly from here

[MohamedAbdrabou2022]

@asjohnston-asf

[MohamedAbdrabou2022]

I have to install gdal to run Mintpy time series?

[MohamedAbdrabou2022]

@yunjunz

[asjohnston-asf]

Yes, ASF recommends GDAL as the most straightforward way to subset the geotiffs from HyP3 to a common width and height as required my MintPy. Fortunately, if you've installed MintPy you should already have GDAL installed in your python environment.

https://mintpy.readthedocs.io/en/latest/installation/

[MohamedAbdrabou2022]

@asjohnston-asf
Now my 236 pairs downloaded from ASF on demand are unzipped in a folder SBASINSAR in D drive; I noticed that I have to make a directory structure before..can you suggest me as I did not utilize hyp3_sdk.

Do I have to install all asf_search, hyp3_sdk, pathlib, typing, osgeo, pandas, dateutil.parser?

Also, as I use Spyder Anaconda, I have to run each cell in different .p file(script) or how?

[asjohnston-asf]

If you've already downloaded your products from HyP3, you should only need GDAL to prepare those geotiffs for processing by MintPy.

I'm not familiar with Spyder Anaconda; I can't offer any guidance there.

[MohamedAbdrabou2022]

How to install gdal? Can you tell me how to run this on Jupyter?

[MohamedAbdrabou2022]

Can I run codes online on jupyter and call my unzipped files to the jupyter note?

[MohamedAbdrabou2022]

@yunjunz @asjohnston-asf

[MohamedAbdrabou2022]

Any help/suggestions to do that in any python viewer or ide

…

On Tuesday, November 8, 2022, Andrew Johnston ***@***.***> wrote: If you've already downloaded your products from HyP3, you should only need GDAL to prepare those geotiffs for processing by MintPy. I'm not familiar with Spyder Anaconda; I can't offer any guidance there. — Reply to this email directly, view it on GitHub <#885 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3CMPUDT3GVV72VSFQANTPDWHKTPRANCNFSM6AAAAAARVURYHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

-- **Disclaimer:** This E-mail represents the views of its author only and does not necessarily reflect the opinions of Faculty of Science (*SCI <http://www.sci.cu.edu.eg>*). We do not assume responsibility for any loss, damage (including direct, indirect, punitive, special, or consequential loss or damage), which you may directly or indirectly suffer from this email or its contents or through the ways of handling this email which includes sending, receiving, replying, forwarding, deleting, saving or changing and adjusting its content in any way.  

[mosyhey]

Hello. You can run MintPy even without subsetting the geotiffs, but you will get an error. Anyway, if you have problems running Jupyter, you can save all four pieces of code in section 3.1 in a Python file, and after a little manipulation run it in Anaconda:

from pathlib import Path
from typing import List, Union
from osgeo import gdal

def get_common_overlap(file_list: List[Union[str, Path]]) -> List[float]:
    corners = [gdal.Info(str(dem), format='json')['cornerCoordinates'] for dem in file_list]
    ulx = max(corner['upperLeft'][0] for corner in corners)
    uly = min(corner['upperLeft'][1] for corner in corners)
    lrx = min(corner['lowerRight'][0] for corner in corners)
    lry = max(corner['lowerRight'][1] for corner in corners)
    return [ulx, uly, lrx, lry]

def clip_hyp3_products_to_common_overlap(data_dir: Union[str, Path], overlap: List[float]) -> None:
    files_for_mintpy = ['_water_mask.tif', '_corr.tif', '_unw_phase.tif', '_dem.tif', '_lv_theta.tif', '_lv_phi.tif']
    tifnum = 1
    tifall = len(list(data_dir.glob('*/*_dem.tif'))) * len(files_for_mintpy)
    for extension in files_for_mintpy:
        for file in data_dir.rglob(f'*{extension}'):
            dst_file = file.parent / f'{file.stem}_clipped{file.suffix}'
            gdal.Translate(destName=str(dst_file), srcDS=str(file), projWin=overlap)
            print(str(tifnum) + ' of ' + str(tifall))
            tifnum += 1

data_dir = Path('D:/sbasinsar')
files = data_dir.glob('*/*_dem.tif')

overlap = get_common_overlap(files)
clip_hyp3_products_to_common_overlap(data_dir, overlap)

Replace data_dir value with your working directory address. After the successful running, six clipped geotiffs will be added to each folder of the working directory: _corr_clipped.tif, _dem_clipped.tif, _lv_phi_clipped.tif, _lv_theta_clipped.tif, _unw_phase.tif and _water_mask.tif.

This code determines the common overlap area covered by all images automatically. You can also define the subsetting area manually, by setting ulx, uly, lrx, and lry variables in the corresponding UTM coordinate system.

[MohamedAbdrabou2022]

@mosyhey
Ok
But, still have a problem with defining reference point, how to adjust that?
out of area every time error

[mosyhey]

Share maskTempCoh.png in the pic folder and also all the lines in the config file that start with the expression mintpy.reference..

[MohamedAbdrabou2022]

@mosyhey
Is there an example that used HyP3 that I can use and get the same results?

[mosyhey]

Try this: Sentinel-1 of the 2019 Ridgecrest, California earthquake sequence with HyP3

[MohamedAbdrabou2022]

@mosyhey
Thanks for your kind reply!
For the previous question
1- maskTempCoh is clear with no data in it.

2- Regarding the reference point:-

mintpy.reference.yx = auto #[257,151 / auto]
mintpy.reference.lalo = 1296137,759977 #[31.8,130.8 / auto]
mintpy.reference.maskFile = auto #[filename / no], auto for maskConnComp.h5
mintpy.reference.coherenceFile = auto #[filename], auto for avgSpatialCoh.h5
mintpy.reference.minCoherence = auto #[0.0-1.0], auto for 0.85, minimum coherence for auto method

Is there an example for HyP3 for land subsidence not earhquakes?

[MohamedAbdrabou2022]

I must write these commands on anaconda or on cmd as my operating system is windows?

@yunjunz
@asjohnston-asf

[yunjunz]

Please do not @ developer on every comment. This introduced extra unwanted email notifications.

[MohamedAbdrabou2022]

Ok.
I have an error and I could not solve it until now which is below, however I made set up for gdal

[MohamedAbdrabou2022]

The Python version on my system is 3.8, and Spyder Python 3.9...this may be the problem?
Also, I have to install mintpy on anaconda or on the system? I confused.

[MohamedAbdrabou2022]

After running this code in python shell, I found a new error..

Traceback (most recent call last):
File "D:/InSAR/GERD_2014_2022/tria1.py", line 34, in
overlap = get_common_overlap(files)
File "D:/InSAR/GERD_2014_2022/tria1.py", line 14, in get_common_overlap
ulx = max(corner['upperLeft'][0] for corner in corners)
ValueError: max() arg is an empty sequence

[MohamedAbdrabou2022]

Ok..I will try run it after modification. Is there a solution for this structure?

…

On Wednesday, November 16, 2022, mosyhey ***@***.***> wrote: Also replace '*/*_dem.tif' with '*\*\*dem.tif' in line 16. Delete or comment out these extra lines: print(files)print(data_dir)print(*list(files),sep='\n') Uncomment the last two lines and run the script. Note that you modify the code and the folder structure remains non-standard. Your files paths: D:\InSAR\GERD_2014_2022\S1AA_2 0141021T032442_...\S1AA_20141021T032442_...\..._dem.tif Standard files paths: D:\InSAR\GERD_2014_2022\S1AA_2 0141021T032442_...\..._dem.tif — Reply to this email directly, view it on GitHub <#885 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/A3CMPUBUBM6E4RKEJJ7XCSTWIWZZNANCNFSM6AAAAAARVURYHA> . You are receiving this because you were mentioned.Message ID: ***@***.***>

-- **Disclaimer:** This E-mail represents the views of its author only and does not necessarily reflect the opinions of Faculty of Science (*SCI <http://www.sci.cu.edu.eg>*). We do not assume responsibility for any loss, damage (including direct, indirect, punitive, special, or consequential loss or damage), which you may directly or indirectly suffer from this email or its contents or through the ways of handling this email which includes sending, receiving, replying, forwarding, deleting, saving or changing and adjusting its content in any way.