Search papers, labs, and topics across Lattice.
This paper critically examines the temporal video question answering benchmark, revealing that current scoring methods conflate task and channel questions, which can mislead evaluations of model performance. The authors introduce a novel label-free metric, the reversal-drop, which measures the accuracy loss when visual sequences are reversed, effectively distinguishing between models that rely on positional encoding and those that interpret visual order. Their findings highlight that models with similar aggregate scores can exhibit fundamentally different failure modes, underscoring the need for more nuanced evaluation metrics in temporal understanding tasks.
Models that seem equally capable on temporal benchmarks can actually rely on drastically different mechanisms for understanding video sequences, revealing hidden vulnerabilities in their evaluations.
A score on a temporal video question answering benchmark is meant to measure that a model has temporal understanding, but it conflates two questions. 1. The task question: is the question even temporal, does it need several frames and their order? and 2. The channel question, when it does, does the model recover the order from the pixels, or read it off the positional encoding (RoPE)? Most of a temporal score answers neither, a single frame and answer priors often carry it. The field's validity checks, frame-shuffle sensitivity and the accuracy gained from the full video, speak only to the task question. We contribute a label-free screen for the channel question, the reversal-drop: the accuracy lost when the visual sequence is reversed while RoPE remains forward. It can be applied to compatible temporal benchmarks without new annotations. Paired reverse labels, or tasks whose labels transform deterministically under reversal, distinguish models that follow reversed content from those merely disrupted by the conflict. Molmo2 answers the forward event reading order off positions, while Qwen3-VL answers the reversed event it actually sees, reading visual order (comparatively). We call them position-dominant and visual-sequence-dominant. The split holds across two benchmarks and several temporal tasks at two scales, and activation patching shows it is a real internal property, not an artifact of the conflict. The distinction matters, the two channels fail on opposite inputs so two models with similar score are not interchangable, i.e. an aggregate score does not reflect potential failure modes.
