Search papers, labs, and topics across Lattice.
This paper investigates the effects of test-time adaptation (TTA) on the calibration of vision-language models, revealing that while TTA can enhance recognition accuracy, it often leads to unreliable predictive confidence due to increased entropy. The authors introduce Zero-Shot-Anchored Entropy Calibration (ZAEC), a novel label-free method that restores the zero-shot entropy of sharpened predictions using minimal temperature scaling, effectively mitigating the calibration degradation observed with TTA. Experimental results demonstrate that ZAEC achieves the lowest post-hoc macro-average Expected Calibration Error (ECE) across multiple TTA methods and datasets, highlighting its effectiveness in maintaining reliable confidence levels without requiring labeled data.
TTA can boost accuracy but often at the cost of calibration, and ZAEC is the key to restoring reliable confidence without labeled data.
Test-time adaptation (TTA) can improve the recognition accuracy of vision-language models under distribution shift, but often degrades calibration, making predictive confidence unreliable for downstream decision-making. Many existing label-free calibration approaches are either coupled to prompt optimization or rely on logit-range statistics that provide only a coarse characterization of the predictive distribution. We show that TTA can increase confidence and reduce entropy even when the top-1 prediction and its correctness remain unchanged, a failure mode we term prediction-preserving sharpening. Across diverse TTA methods and benchmarks, larger entropy reductions relative to paired zero-shot predictions are associated with greater increases in Expected Calibration Error (ECE). On entropy-reduced samples, confidence gains also tend to exceed accuracy gains. Based on these findings, we propose Zero-Shot-Anchored Entropy Calibration (ZAEC), a label-free post-hoc method that uses zero-shot entropy as a sample-specific uncertainty reference. ZAEC selectively restores the zero-shot entropy of sharpened predictions through minimal temperature scaling while leaving all other predictions unchanged. It requires no labeled calibration data or learned parameters and preserves class rankings and classification accuracy. Across five TTA methods and 15 datasets, ZAEC achieves the lowest post-hoc macro-average ECE on ViT-B/16, with consistent gains on RN50.
Tsinghua AI