The goal of image retrieval is to retrieve the most relevant images to the query image. Imagine we are looking for more pictures depicting a specific geographical location, such as the Gullfoss waterfall in Iceland.

In a standard image retrieval system, our query will be encoded into high-dimensional vectors using image encoders such as ResNet, ViT, etc., and using some similarity metric (e.g., cosine similarity), the most relevant images from a large database of candidate images will be retrieved.

In a multimodal image retrieval system, the goal remains the same, but the query is a combination of the image and some other modality such as text, image metadata, etc.

In our scenario, we'll be working with images taken from the ISS (International Space Station). The goal of our image retrieval system is to retrieve relevant candidate images for our queries, where "relevant" means there is a non-zero overlap between the query and the retrieved images. Briefly, there are three sources of data:

• 1. The query image taken from the ISS
• 2. The candidate images taken by the Sentinel-2 satellite in the year 2021
• 3. The image metadata (nadir) for both the query and candidate images

The nadir is the point on Earth's surface directly below the satellite. It is useful metadata for the ISS images, as knowing it can provide a rough estimation of where the image was taken (we can be sure that the image is within ~2500km of the nadir point). The picture below helps illustrate this concept better.

The question remains, however: how to use the metadata to improve retrieval performance? We've opted for a simple approach partially inspired by the recent CVPR paper by Dhakal et al. [1] and the GeoCLIP paper [2]. We use a pretrained DINOv2 foundation model as the image encoder combined with a SALAD aggregator to improve the embeddings' performance. For the metadata encoder, we pass the nadir coordinates (latitude, longitude) through a combination of Fourier Features and standard MLP layers to obtain a high-dimensional embedding. After having both the image and metadata embeddings, we simply concatenate them and use this as our final multimodal embedding. We train using a multi-similarity loss on the concatenated embeddings. The picture below illustrates the architecture better.

Experimental Setup

We performed two experiments to prove whether multimodal embeddings improve retrieval performance. In the first experiment, no metadata is used—only the image embeddings are used for retrieval. In the second experiment, we use the image embeddings concatenated with the metadata embeddings for both the query and candidate images.

Results

The results are reported on each of the evaluation datasets (Alps, Texas, Toshka Lakes, Amazon, Napa, Gobi) using the standard Recall@K metric. It's worth noting that all 161,496 candidate images are used for retrieval, making the task more challenging. Below is the coverage of the query and candidate images used for retrieval.

Image + Coordinate Embeddings

Only Image Embeddings

All models were trained on a single NVIDIA H100 100GB GPU. We used the DINOv2-based model with register tokens and SALAD aggregator. The learning rate was set to 1e-4. The models were trained for 20 epochs with a batch size of 96. For more details, consult the GitHub repository here.