Search papers, labs, and topics across Lattice.
This study probes self-supervised learning (SSL) representations of satellite imagery to evaluate their ability to encode statistical associations with environmental variables, using co-located ERA5 reanalysis data. By analyzing models trained under identical conditions, including DINO, MAE, and MoCo, the authors demonstrate that representation-level metrics can effectively differentiate models with similar downstream performance, revealing insights into their geometric properties and encoding of environmental signals. The findings indicate that the linear accessibility of these signals correlates with performance on environmentally dependent tasks, underscoring the importance of representation analysis beyond traditional benchmarks.
SSL representations of satellite imagery can reveal hidden environmental associations that significantly impact downstream task performance.
Self-supervised learning (SSL) is designed to learn generic, transferable representations rather than representations optimized for a single task. Most geospatial benchmarks evaluate representations solely through downstream tasks, providing limited insight into the information encoded within the representation itself. We ask a different question: do SSL representations of satellite imagery preserve statistical associations with environmental variables that co-vary with the imaging process? To answer this question, we probe SSL representations using co-located ERA5 reanalysis variables, a global dataset of physically consistent environmental variables, including temperature, precipitation, surface solar radiation, surface pressure, and volumetric soil water. These variables are physically related to the spectral reflectance and radar backscatter recorded by Sentinel-1 and Sentinel-2, making them meaningful evaluation targets despite not being used during SSL pretraining. We complement this probing analysis with intrinsic representation metrics to characterize representation geometry and investigate how these properties relate to downstream performance and the encoding of environmental signals. Using DINO, MAE, and MoCo models trained under identical conditions, we show that representation-level metrics distinguish models with similar downstream benchmark performance, providing complementary information beyond task-driven benchmarks. We further find that the linear accessibility of environmental signals is associated with performance on environmentally dependent tasks in the PANGAEA benchmark. Finally, we release ERA5 annotations co-located with the SSL4EO dataset to enable physically grounded representation evaluation for future geospatial foundation models.
