Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper systematically evaluates uncertainty quantification (UQ) methods in the context of foundation models for semantic segmentation, specifically using a fine-tuned DPT decoder atop the pretrained SAM2 encoder. The study benchmarks four UQ approaches—Monte Carlo Dropout, Deep Sub-Ensemble, Test-Time Augmentation, and Evidential Deep Learning—across various datasets, including Cityscapes and NYUv2, while analyzing trade-offs in segmentation accuracy, calibration, and computational cost. The findings underscore the potential and limitations of integrating UQ into foundation models, emphasizing the need for future research to balance accuracy, robustness, and efficiency for practical applications.
Uncertainty-aware foundation models can significantly improve segmentation reliability, but they come with trade-offs that challenge real-world deployment.
Abstract. Foundation models are increasingly breaking what seemed to be impossible not long ago by enabling unprecedented accuracy and cross-domain generalization. Yet their lack of interpretability, tendency to be overconfident, and sensitivity to real-world domain shifts pose critical challenges for safety- and mission-critical applications. Uncertainty quantification (UQ) offers a principled way to address these issues, but its integration into segmentation foundation models has yet to be explored. In this paper we present the first systematic evaluation of UQ methods applied to a foundation model for semantic segmentation. We fine-tune a lightweight DPT decoder on top of the pretrained SAM2 encoder to establish a simple yet competitive baseline and benchmark four representative UQ approaches – Monte Carlo Dropout, Deep Sub-Ensemble, Test-Time Augmentation, and Evidential Deep Learning – across Cityscapes, NYUv2, and two challenging out-of-domain settings. Our analysis compares segmentation accuracy, calibration, uncertainty quality, and inference time, revealing clear trade-offs between predictive performance, reliability, and computational cost. These results highlight both the promise and the current limitations of uncertainty-aware foundation models, pointing to the need for future work that jointly optimizes accuracy, robustness, and efficiency for real-world deployment.