Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To address confounding correlations in standard representation probing, this work introduces an evaluation framework that decouples state prediction accuracy, target-state responsiveness, and context stability in Vision-Language-Action (VLA) models using counterfactually crossed physical observations. The authors find that high natural-trajectory decoding accuracy routinely coexists with weak controlled responsiveness, as readouts exploit spurious contextual features like robot configuration and task progress. Crucially, factoring in target-state responsiveness and context stability metrics significantly improves downstream policy failure prediction on novel initializations compared to standard error baselines.
High probe accuracy in VLAs is often an illusion: internal state decoders frequently track ambient scene context rather than the target object itself, hiding causal blind spots that trigger policy failures.
Accurately decoding object states from the internal representations of vision-language-action (VLA) models does not establish that the predictions respond faithfully to changes in the target physical state. In natural observations, object state, robot configuration, occlusion, and task progress vary together, allowing contextual cues to contribute to prediction. In this paper, we introduce an evaluation framework that separates prediction accuracy, target-state responsiveness, and context stability using physically validated observations that cross target coordinates with robot contexts. We demonstrate that high natural-trajectory accuracy can coexist with weak controlled target-state responsiveness in fixed representation-readout pairs. Comparisons and interventions involving representations, readouts, and training data show that the three properties provide distinct diagnostic information. Furthermore, adding responsiveness and context sensitivity to a failure predictor based on initial state error and physical variables reduces policy-failure prediction error on new initializations relative to the specified baseline while same-observation controlled MAE is also informative. These findings motivate evaluating target-state responsiveness and context stability alongside natural prediction accuracy, and examining their relationship to actual policy behavior and task outcomes.