Search papers, labs, and topics across Lattice.
This paper introduces ARGTCA, a novel method that enhances confidence estimation in vision-language models (VLMs) by leveraging a Symbolic Attribute Graph to represent class and attribute pairs as interconnected nodes. By employing a Graph Attention Network (GAT) with contrastive objectives, the method captures inter-attribute dependencies, addressing the limitations of prior approaches that treated attributes independently. Experimental results demonstrate that ARGTCA-DIV significantly reduces Expected Calibration Error (ECE) by ~37% compared to baselines, indicating that structured modeling of attributes can substantially improve test-time adaptation reliability in VLMs.
Structurally informed embeddings derived from a Symbolic Attribute Graph can slash calibration errors in vision-language models by up to 37%.
Reliable confidence estimation remains a key limitation of test-time adaptation in vision-language models (VLMs), where prompt tuning improves zero-shot accuracy but often degrades calibration due to entropy-driven overconfidence. Prior approaches mitigate this using LLM-derived class attributes and contrastive regularization, yet treat attributes independently, ignoring their relational structure. We propose ARGTCA, which represents (class, attribute) pairs as nodes in a Symbolic Attribute Graph and trains a Graph Attention Network (GAT) using contrastive objectives to produce structurally informed embeddings that capture inter-attribute dependencies. We introduce two attribute selection strategies: ARGTCA-DIV for intra-class diversity and ARGTCA-DISC for inter-class discrimination. Experiments across nine benchmarks show that ARGTCA-DIV reduces average Expected Calibration Error (ECE) by approximately ~37% over baselines, while ARGTCA-DISC consistently performs as the second-best variant, reducing average ECE by approximately ~17% over baselines. These results suggest that modeling symbolic attribute interactions provides a principled approach for reliable test-time adaptation in VLMs.
