Multi-Spectral Remote Sensing Image Retrieval Using Geospatial Foundation Models

[brunosan]

2403.02059.pdf
4 Mar 2024

OpenAI "candid summary":

The authors are basically saying they found a way to heavily compress satellite image data into a much smaller representation that still works pretty well for training AI models to do things like classify scenes and segment objects.

Their trick is to take an existing big AI model pre-trained on satellite imagery, and fine-tune part of it to squish the data down a ton while trying not to lose too much useful information. They call this "neural embedding compression."

In their tests, they could shrink the data by 75-90% and still get the AI models to perform almost as well as training on the original uncompressed images. Even at crazy high 99.7% compression, accuracy only dropped by 5% for scene classification.

Basically, the authors are flexing that their compression scheme is way better than existing methods and claim it's a big deal for sustainably working with huge satellite datasets going forward. Less data transferred and stored means lower costs and energy usage.

[brunosan]

Gist of the approach is to find the changes in precision when finding "similar top 20" images with same labels (any of the labels if multiclass), using different embdding models and vector storages.

They compare the precision of the embeddings:

• The raw embeddings, size 768.
• Binaries (transforming all to 0 or 1 setting the optimal threshold)
• hashing, by chunking the embedding into 32 chunks, making the average on each chunk, and doing the binary on each average. Final size 32.
• LSH hashes. Randomly split the embedding space and assign 0 or 1 depending on what side the embedding falls. Do that 32 times. This maximizes that neighbor embeddings have very similar hashes. Final size 32.

Note: All input sizes are 224x224 which for Prithvi that means 6,7kmx6.7km
Note: They used Milvus as vector db with indexes as INV FLOAT and BIN IVF FLAT with the default values of 128 clusters and 8 top clusters.

They then compare results of each embedding method, on 3 different models. Prithvi (model trained on 6 input bands), Prithvi-RBG (trained on only 3 chanels RGB) and a vanilla ViT trained on ImageNet that only takes RGB.

1. I'm mostly surprised that Prithvi is not that much more precise than ViT. Less than 10% in full embeddings, and is actually better when using hashes. I suspect this is because ViT probably had more training time.
2. Prithvi is better than Prithvi-RGB, This is expected and good news, since more input meant more signal to find similar
3. Binary embeddings seem a very easy and fast way to massively reduce size and mantain signals. This is a similar result to Unary patch embeddings #201.
4. LSH seems really good at clustering embeddings (as designed, and as seen on Tsne plots) but these cluster don't seem to align to labels. I guess this would improve if you increase the length, or make the planes optimal, but in that case you are closer to a Random Forest embedding.
5. A fundamental problem we find at Learning to find similar semantics from embeddings #186 and they don't explore here is the fact that similarity is at the image level, whereas label is always a subset of the image. It might be that a top similar result is so because it also has very similar forest and roads and buildings, but the query has an airport and the results don't. This is partially addressed when they consider a positive hit on multiclass when any of the classes is present.

Relevance for Clay:

• This is a great exploration of what labels and datasets and embeddings storage to use. We should run these with Clay.