Skip to content

Kerchunk

H. Joe Lee edited this page Aug 11, 2023 · 22 revisions

Sample NASA HDF5 Products Test

Zarr fails to read the following datasets from Kerchunk files. The following list is cumulative and doesn't repeat the same dataset name from the previous file.

20220524080000-STAR-L2P_GHRSST-SSTsubskin-ABI_G17-ACSPO_V2.71-v02.0-fv01.0.nc.h5.json

  • /geostationary: IndexError: too many indices for array; expected 0, got 1
  • /crs: IndexError: too many indices for array; expected 0, got 1
  • /dt_analysis: ValueError: Shuffle buffer is not an integer multiple of elementsize #342
  • /l2p_flags: ValueError: cannot reshape array of size 106751 into shape (1,1800,3600)
  • /quality_level: ValueError: cannot reshape array of size 94601 into shape (1,2250,4500)
  • /satellite_zenith_angle: ValueError: Shuffle buffer is not an integer multiple of elementsize
  • /sea_surface_temperature: ValueError: Shuffle buffer is not an integer multiple of elementsize
  • /sses_bias: ValueError: cannot reshape array of size 393656 into shape (1,2250,4500)
  • /sses_standard_deviation: ValueError: cannot reshape array of size 270531 into shape (1,2250,4500)
  • /sst_count: ValueError: cannot reshape array of size 168432 into shape (1,2250,4500)
  • /sst_dtime: ValueError: Shuffle buffer is not an integer multiple of elementsize

20220607120000-STAR-L3S_GHRSST-SSTsubskin-LEO_PM_D-ACSPO_V2.80-v02.0-fv01.0.nc.h5.json

  • /l3s_sst_reference: ValueError: Shuffle buffer is not an integer multiple of elementsize
  • /sst_source: ValueError: cannot reshape array of size 170249 into shape (1,2250,4500)
  • /wind_speed: ValueError: cannot reshape array of size 294150 into shape (1,2250,4500)

AMSR_2_L3_DailySnow_P00_20160831.he5.json

  • /HDFEOS INFORMATION/CoreMetadata.0: IndexError: too many indices for array; expected 0, got 1
  • /HDFEOS INFORMATION/StructMetadata.0:IndexError: too many indices for array; expected 0, got 1

ATL08_20181014084920_02400109_003_01.h5.json

  • /ancillary_data/release: ValueError: When changing to a larger dtype, its size must be a divisor of the total size in bytes of the last axis of the array.
  • /ancillary_data/version: ValueError: When changing to a larger dtype, its size must be a divisor of the total size in bytes of the last axis of the array.

OMI-Aura_L2-OMNO2_2016m0215t0210-o61626_v003-2016m0215t200753.he5.json

  • /HDFEOS INFORMATION/ArchivedMetadata.0: IndexError: too many indices for array; expected 0, got 1

Future Work

Once NCZarr can handle Kerchunk, Hyrax netcdf_handler can replace dmrpp_module.