Search papers, labs, and topics across Lattice.
This study evaluates the calibration of Geospatial Foundation Models (GeoFMs) beyond mere accuracy metrics, revealing that these models exhibit significant overconfidence under distribution shifts, regardless of their pretraining source. The analysis, conducted across multiple datasets and stress conditions, shows that EO-pretrained models do not outperform ImageNet-pretrained models in terms of stability or calibration when faced with data corruption. Furthermore, common uncertainty quantification methods fail to mitigate the degradation in model performance, highlighting the need for a more comprehensive evaluation framework for GeoFMs in real-world applications.
GeoFMs are overconfident and equally fragile under stress, regardless of whether they are pretrained on EO or ImageNet data, challenging conventional evaluation practices.
Geospatial Foundation Models (GeoFMs) are most commonly ranked and selected by accuracy on standard benchmark conditions via averaged ranks. We show that this protocol is too narrow: the promised deployment in critical EO tasks requires further angles of analysis, mainly calibration, the agreement between a model's confidence and its correctness. Across 16 frozen encoders, four classification and five segmentation datasets, and two orthogonal stress axes, every encoder degrades as corruption intensifies, and the ranking changes as well. Across the four classification benchmarks, EO-pretrained and ImageNet-pretrained encoders are indistinguishable on clean accuracy and clean calibration, and EO pretraining provides no more stability under shift than ImageNet pretraining. Under shift the GeoFMs drift further into overconfidence than the ImageNet-pretrained encoders, at every grade and in every corruption family. A centered kernel alignment (CKA) analysis ties this to representational rigidity: EO-pretrained embeddings move less under corruption while losing just as much task information and remaining overconfident. We apply three commonly explored uncertainty quantification methods and find that temperature scaling and deep ensembles cannot counteract the degradation, while a Gaussian-process probe roughly halves ECE under severe cloud only by tripling it on clean data. In selective prediction experiments, we find that confidence-based abstention cannot defer around confidently wrong predictions, and advocate that benchmark rankings and evaluations should therefore operate across a multitude of conditions and metrics to more holistically evaluate model development progress and close the gap to real world deployment scenarios.
