How to define an MLM input that depends on a preprocessing operation

[fmigneault]

With a STAC Item using MLM extension, it is possible to define a model input as such:

{
  "type": "Feature",
  "properties": {
    "mlm:input": [
      {
        "name": "EO Data",
        "bands": ["B01", "B02", "B03", "B04", "B08", "B8A", "B09", "B11", "B12", "NDVI", "NDWI1", "NDWI2"],
        "input": {
          "shape": [-1, 12, 1024, 1024],
          "dim_order": ["batch", "channel", "height", "width"],
          "data_type": "uint16"
        },
        "pre_processing_function": null
      }
    ],
    "raster:bands": [
      { "name": "B01", "common_name": "coastal", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 60, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B02", "common_name": "blue", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 10, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B03", "common_name": "green", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 10, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B04", "common_name": "red", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 10, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B05", "common_name": "rededge1", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 20, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B06", "common_name": "rededge2", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 20, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B07", "common_name": "rededge3", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 20, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B08", "common_name": "nir", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 10, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B8A", "common_name": "nir08", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 20, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B09", "common_name": "nir09", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 60, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B10", "common_name": "cirrus", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 60, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B11", "common_name": "swir16", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 20, "scale": 0.0001, "offset": 0, "unit": "m"},
      { "name": "B12", "common_name": "swir22", "nodata": 0, "data_type": "uint16", "bits_per_sample": 15, "spatial_resolution": 20, "scale": 0.0001, "offset": 0, "unit": "m"}
    ]
  }
}

Provided that the Item also includes a raster:bands definition that describe the B01 to B12 bands, this is valid against the schema to define mlm:input[0].bands cross-references.

However, in this specific case, the model also takes as input the NDVI, NDWI1 and NDWI2, which are themselves derived from a calculation combining the other bands:

• NDVI = (B08 - B04) / (B08 + B04)
• NDWI1 = (B03 - B08) / (B03 + B08)
• NDWI2 = (B08 - B11) / (B08 + B11)

What would be the best way to represent those calculations to define those "virtual bands".

Things considered:

• https://github.com/stac-extensions/processing#expression-object with processing:expression and format: rio-calc using above formulas
• https://github.com/stac-extensions/virtual-assets#fields with vrt:algorithm and vrt:algorithm_opts similar to what processing does

In both cases, the question remains, where to put the expression/algorithm?
Given that this STAC Item is not itself a "data provider" definition listing Asset URIs, but rather a model "using/expecting" these Assets, it seems that nesting processing:expression or vrt:algorithm/vrt:hrefs in the "assets" section is not appropriate, as the source Assets are not offered by this STAC Item. That would be misleading for a client reading the contents of the Item.

Another possibility is to define new "virtual bands" under raster:bands, which would contain a nested processing:expression.
However, that seems to be more of a hack than anything else, since that would not be validated against the processing extension. Including the extension in stac_extensions would also cause an error (due to no processing:... field found), which would require a random field just to pass validation...

The pre_processing_function (https://github.com/crim-ca/mlm-extension#processing-expression) seems like the best candidate to receive this information and avoid stretching the definition of other extensions, but it is currently available only for working over the full MLM input (ie: the set of 9 bands + 3 virtual ones all at once). Since only some "bands" receive a preprocessing, this definition becomes somewhat convoluted under a single field.

Should an array be permitted in pre_processing_function, with indices of the array corresponding to per-band operations?
Any other suggestions?

[fmigneault]

@m-mohr @emmanuelmathot @rbavery
I would like your input regarding this, to see if there are any other suggestions or considerations when involving the full STAC ecosystem.

[rbavery]

What could work though, is to have it directly like follows :

I like that example, looks succinct. Would we still not get validation by including the processing expression in this nested object because it's not at the initial property level?

[fmigneault]

In this case, I can include the $ref to processing:expression to perform the validation (though, not much to validate other than format: string since expression: Any is allowed). The key difference is that processing:expression is not shown, so users are not misled by the extensions available in the STAC Item.

[fmigneault]

To be more explicit. We can "align" the field to look like processing:expression, but we cannot "hack" the processing extension and apply it anywhere we want (i.e.: under mlm:input[].bands). We must respect its definition that limits its use under the STAC Item properties.

Using $ref to processing:expression is a shortcut to make sure the definitions are "aligned", but technically, we could rewrite them explicitly in MLM to achieve the same behavior. The processing extension is not included in stac_extension (unless some other properties for lineage would actually use it as intended).

[fmigneault]

The solution described in #7 (reply in thread) is now implemented in https://github.com/crim-ca/mlm-extension/releases/tag/v1.2.0