Search papers, labs, and topics across Lattice.
This study investigates the dual failures of video large language models (LLMs) when determining the temporal order of events, specifically how they can either accept false claims or reject true ones. By distinguishing between the effects of availability and weighting through innovative interventions, the authors demonstrate that while availability is necessary for accurate event recognition, it is insufficient on its own, as evidenced by the models' performance in a controlled reversal test. The proposed method significantly improves order accuracy, achieving rates between 0.92 and 1.00 for models capable of reading order, while also introducing a strategy to abstain from guessing when evidence is insufficient.
Video LLMs can misjudge event order by either succumbing to false claims or dismissing true ones, revealing a critical flaw in their reasoning capabilities.
When asked which of two events came first, video large language models can fail in two opposite ways: cave to a false claim, or reject a true one. Prior video sycophancy work measures only the first and mitigates it by teaching the model to trust the user less, a fix known in text and image models to worsen the second. In video, both failures come from two causes the literature treats as one: availability, whether the sparse sampled frames contain the two events, and weighting, whether that evidence is trusted over the user. We separate them with two interventions that keep the claim fixed: a frame-preserving reorder that flips the claim's truth, and a sampling-offset shift that captures or misses both events at a fixed frame budget. When the events are missed, the two twins present identical frames, so each of the nine models we evaluate accepts a true and a false claim at the same rate, making Youden's $J=0$ by construction. Availability is necessary but not sufficient. Five of the nine read the order, yet four of those five still cave to the false claim, so their deference hits a weighting ceiling. Since trust cannot be calibrated over evidence that was never sampled, we propose a reversal test that cancels the model's order prior by scoring the sampled frames forward and reversed, then answers, resamples, or abstains without reading the claim. The test raises the order accuracy to 0.92-1.00 on the models that read the order and abstains rather than guesses on those that cannot.
