Search papers, labs, and topics across Lattice.
This paper introduces Counterfactual Evidence Disentanglement (CED), a novel training-time evidence audit method for Vision-Language Models (VLMs) that ensures responses are grounded in relevant image evidence rather than relying on language priors or irrelevant visual context. By neutralizing object-centric Evidence Regions and assessing the impact on answer correctness, CED effectively distinguishes between valid evidence-based responses and those based on shortcuts. The method demonstrates superior performance across nine benchmarks and four model backbones, significantly improving the reliability of VLM outputs in visual reasoning tasks.
CED reveals that VLMs can be trained to prioritize evidence-based reasoning over language shortcuts, leading to more reliable visual understanding.
Vision-Language Models (VLMs) should answer from concrete image evidence rather than language priors, dataset shortcuts, or irrelevant visual context. Existing perception-aware post-training methods encourage image use through global perturbations or attention proxies, but they do not test whether a sampled answer causally depends on the local evidence that supports it. We propose Counterfactual Evidence Disentanglement (CED), a training-time evidence audit for VLM grounding. For each response, CED neutralizes an object-centric Evidence Region and compares the resulting support drop against matched non-evidence Regions. We combine this signal with answer correctness inside GRPO, rewarding correct answers that rely on the evidence path rather than shortcut or nuisance paths. CED uses weak object-level proposals, requires no question-specific evidence annotations, and adds no inference-time overhead. Across nine public benchmarks and four backbones, CED outperforms prior RL-based post-training methods, with targeted analyses verifying its object-centric signal.
NUS
Tsinghua AI