Configuration Suggestions for Building a Large COG

[ljstrnadiii]

I am looking for any recommendations for speeding up writing a single large (10Gi) COG on a machine only doing this task.

Assuming I have 10Gi Xarray dataset with a single band. Lets say an xarray dataset with shape (52000, 52000) of type float32 that has chunks (4096, 4096) and I am writing to_raster as follows:

array.rio.to_raster(
        tmp_path, # local path
        overview_resampling="average",
        driver="COG",
        bigtiff="yes",
        BLOCKSIZE=blocksize,
        windowed=True, # without this we were seeing OOM due to low ephemeral storage
    )

but this is taking a long time ~4 hours and only partially complete.

Questions

• would it be helpful to set lock=False to enable multi-threading?
• windowed=True seems to slow things down significantly and is likely not appropriate for cogs?
• would the multiple worker example here help using a lock?
• Any GDAL configs worth trying?
• Increase ephemeral storage (for tmp files like for .ovr files and remove windowed=True)?
• Any other suggestions to try?

Performance and Utilization

On a machine with 15gb mem and 4 cores utilization looks like:

Possibly Relevant Issues / Sources:

• speeding up windowed writes #439
• https://corteva.github.io/rioxarray/html/examples/dask_read_write.html