Fetching NASA SRTM from MPC fails

[goergen95]

Calling get_nasa_srtm() fails because MPC now requires those URLs to be signed

[goergen95]

library(terra)
#> terra 1.7.78
url1 <- "https://nasademeuwest.blob.core.windows.net/nasadem-cog/v001/NASADEM_HGT_s05e030.tif"
url2 <- "https://nasademeuwest.blob.core.windows.net/nasadem-cog/v001/NASADEM_HGT_s04e030.tif"
rast(paste0("/vsicurl/", url1))
#> Warning: HTTP response code: 409 (GDAL error 11)
#> Error: [rast] file does not exist: /vsicurl/https://nasademeuwest.blob.core.windows.net/nasadem-cog/v001/NASADEM_HGT_s05e030.tif
rast(paste0("/vsicurl?pc_url_signing=yes&url=", url2))
#> class       : SpatRaster 
#> dimensions  : 3601, 3601, 1  (nrow, ncol, nlyr)
#> resolution  : 0.0002777778, 0.0002777778  (x, y)
#> extent      : 29.99986, 31.00014, -4.000139, -2.999861  (xmin, xmax, ymin, ymax)
#> coord. ref. : lon/lat WGS 84 (EPSG:4326) 
#> source      : NASADEM_HGT_s04e030.tif 
#> name        : NASADEM_HGT_s04e030

^{Created on 2024-10-13 with reprex v2.1.1}

[fBedecarrats]

Hi @goergen95 , I indeed face this issue for the training on mapme.biodiversity and applications for impact evaluation that we are currently facilitating in Madagascar. Should I make a fork of the package, change the url by appending ?pc_url_signing=yes&url=" to it and have participant re-installing the package from my forked github repo?
Or is it more complicated than that?

[goergen95]

Why not merge those changes to main and point your students to the dev version via:

remotes::install_github("https://github.com/mapme-initiative/mapme.biodiversity", dependencies = TRUE)

Happy to receive a PR. Not sure yet when this will hit CRAN - but since we have some checks failing there, a new release might be required in the next days...

[fBedecarrats]

hmmm. I'm affraid that it is not so easy.
In , line 51, I replaced:

fps[["source"]] <- paste0("/vsicurl/", urls)

with:

fps[["source"]] <- paste0("/vsicurl?pc_url_signing=yes&url=", urls)

But when I test it after load_all(), I get:

aoi_path <- system.file("extdata", "gfw_sample.gpkg", package = "mapme.biodiversity")
aoi <- st_read(aoi_path, quiet = TRUE)

dir.create("test")
mapme_options(dir = "test")

aoi <- get_resources(
  x = aoi,
  get_nasa_srtm()
)
# R package 'progressr' recommended.
# Please install via `install.packages('progressr')`FALSE
# 
# Attaching package: ‘purrr’
# 
# The following object is masked from ‘package:testthat’:
#   
#   is_null
# 
# There were 16 warnings (use warnings() to see them)
# > warnings()
# Warning messages:
#   1: In CPL_gdaltranslate(source, destination, options, oo,  ... :
#                             GDAL Message 6: Unsupported option: pc_url_signing

And the reprex you gave does not work with me:

> library(terra)
> url1 <- "https://nasademeuwest.blob.core.windows.net/nasadem-cog/v001/NASADEM_HGT_s05e030.tif"
> url2 <- "https://nasademeuwest.blob.core.windows.net/nasadem-cog/v001/NASADEM_HGT_s04e030.tif"
> rast(paste0("/vsicurl/", url1))
Error: [rast] file does not exist: /vsicurl/https://nasademeuwest.blob.core.windows.net/nasadem-cog/v001/NASADEM_HGT_s05e030.tif
In addition: Warning message:
In new_CppObject_xp(fields$.module, fields$.pointer, ...) :
  GDAL Error 4: `/vsicurl/https://nasademeuwest.blob.core.windows.net/nasadem-cog/v001/NASADEM_HGT_s05e030.tif' does not exist in the file system, and is not recognized as a supported dataset name.
> rast(paste0("/vsicurl?pc_url_signing=yes&url=", url2))
Error: [rast] file does not exist: /vsicurl?pc_url_signing=yes&url=https://nasademeuwest.blob.core.windows.net/nasadem-cog/v001/NASADEM_HGT_s04e030.tif
In addition: Warning messages:
1: In new_CppObject_xp(fields$.module, fields$.pointer, ...) :
  GDAL Message 6: Unsupported option: pc_url_signing
2: In new_CppObject_xp(fields$.module, fields$.pointer, ...) :
  GDAL Message 6: Unsupported option: pc_url_signing
3: In new_CppObject_xp(fields$.module, fields$.pointer, ...) :
  GDAL Message 6: Unsupported option: pc_url_signing
4: In new_CppObject_xp(fields$.module, fields$.pointer, ...) :
  GDAL Message 6: Unsupported option: pc_url_signing
5: In new_CppObject_xp(fields$.module, fields$.pointer, ...) :
  GDAL Message 6: Unsupported option: pc_url_signing
6: In new_CppObject_xp(fields$.module, fields$.pointer, ...) :
  GDAL Error 4: `/vsicurl?pc_url_signing=yes&url=https://nasademeuwest.blob.core.windows.net/nasadem-cog/v001/NASADEM_HGT_s04e030.tif' does not exist in the file system, and is not recognized as a supported dataset name.
> terra::gdal()
[1] "3.4.1"

terra is indeed 1.7-78 here.
I'm affraid that I can't figure out the solution by myself.

[goergen95]

GDAL v3.4.1 is about three years old now. The pc_url_signing parameter was added in 3.5.2 In fact, we require minimum version of 3.7.0 for the usage of mapme.biodiversity as indicated in DESCRIPTION:

mapme.biodiversity/DESCRIPTION

Line 35 in cda74e6

SystemRequirements: GDAL (>= 3.7.0), PROJ (>= 4.8.0)

[fBedecarrats]

Thanks @goergen95 . I'm affraid that this outdated GDAL version is the one shipped with rocker/geospatial (we work with mapme.biodiversity on a cloud environment running on kubertnetes). I will try to solve this.

[goergen95]

You can find some images we are building off of rocker in mapme-docker.

[fBedecarrats]

You can find some images we are building off of rocker in mapme-docker.

Wonderful, thanks!