Search papers, labs, and topics across Lattice.
This paper introduces two novel multimodal contrastive learning architectures, MELT and SALT, designed to enhance location encoders by leveraging unpaired geospatial data. The study reveals that while these methods match the performance of the leading two-modality baseline (SATCLIP) across four spatial prediction tasks, increasing modality diversity does not consistently yield better results, indicating that the effectiveness of the location encoder is a critical factor. Notably, MELT demonstrates more stable training dynamics compared to SALT, laying a stronger groundwork for future developments in this area.
Expanding beyond traditional two-modality frameworks, MELT and SALT reveal that more modalities don't always translate to better performance in spatial predictions.
Spatial prediction tasks are often limited by a lack of high-quality labelled ground-truth observations. To overcome this challenge, self-supervised pre-training is a possible solution, with contrastive learning dominant for location encoders. Those approaches usually align geographic coordinates with just one additional modality. We propose two multimodal contrastive learning architectures: Multimodal Embedding via Location Tying (MELT) and Sequential Alternating Location Training (SALT). These architectures expand this framework beyond two modalities by utilising unpaired geospatial data. Both methods are technically viable and match the performance of the strongest two-modality baseline (SATCLIP) across four downstream tasks. However, increasing the number of modalities does not consistently improve performance, suggesting that the chosen location encoder is the main limitation - the contrastive objective reaches its peak early, regardless of modality diversity or pre-training volume. MELT provides more stable training than SALT and presents a stronger foundation for future scaling.
