Search papers, labs, and topics across Lattice.
This paper introduces VLADriveBench, a novel framework designed to evaluate the relationship between chain-of-thought (CoT) reasoning and driving actions in vision-language-action (VLA) models. By integrating observational metrics with a CoT intervention protocol, the authors reveal that traditional trajectory quality assessments can obscure critical insights about the relevance and causal nature of CoT reasoning. The findings indicate that while ORION achieves high observational alignment, its CoT lacks causal impact, contrasting with Alpamayo v1.5, which, despite lower scores, demonstrates a strong causal relationship influenced by visual salience.
Observational metrics can mislead evaluations of VLA models, revealing that high trajectory quality does not guarantee meaningful CoT reasoning.
Vision-language-action (VLA) models generate chain-of-thought (CoT) reasoning alongside driving trajectories, but existing benchmarks evaluate only trajectory quality and do not assess whether the CoT is relevant, consistent, or causally connected to the driving action. We introduce VLADriveBench, a framework that combines observational metrics (mentioning, hallucination, contradiction, action alignment) with a CoT intervention protocol to provide complementary views of the CoT-action relationship. Applying VLADriveBench to three models across two architectures, we find that the two analyses can diverge sharply: ORION scores highest on observational alignment yet its CoT is epiphenomenal, while Alpamayo v1.5 scores lower yet its CoT is strongly causal, with visual salience gating the extent of CoT influence.
