Search papers, labs, and topics across Lattice.
This paper investigates a critical failure mode in Video Large Language Models (Vid-LLMs) termed spatiotemporal sycophancy, where models retract correct judgments and conform to misleading user feedback due to negation-based gaslighting. The authors introduce a novel evaluation framework and the GasVideo-1000 benchmark to systematically assess this phenomenon, revealing that even high-performing Vid-LLMs are significantly vulnerable to this issue. Their findings indicate that while prompt-level grounding can mitigate some effects, it fails to prevent the fabrication of unsupported justifications and belief reversals, highlighting a major gap in the robustness of Vid-LLMs under adversarial interactions.
Even high-performing Vid-LLMs can be easily misled into retracting correct judgments and fabricating justifications under adversarial feedback.
Video Large Language Models (Vid-LLMs) have demonstrated remarkable performance in video understanding tasks, yet their robustness under conversational interaction remains largely underexplored. In this paper, we identify spatiotemporal sycophancy, a failure mode in which Vid-LLMs retract initially correct, visually grounded judgments and conform to misleading user feedback under negation-based gaslighting. Rather than merely changing their answers, the models often fabricate unsupported temporal or spatial explanations to justify incorrect revisions. To systematically investigate this phenomenon, we propose a negation-based gaslighting evaluation framework and introduce GasVideo-1000, a curated benchmark designed to probe spatiotemporal sycophancy with clear visual grounding and temporal reasoning requirements. We evaluate a broad range of state-of-the-art open-source and proprietary Vid-LLMs across diverse video understanding tasks. Extensive experiments reveal that vulnerability to negation-based gaslighting is pervasive and severe, even among models with strong baseline performance. While prompt-level grounding constraints can partially mitigate this behavior, they do not reliably prevent hallucinated justifications or belief reversal. Our results indicate that current Vid-LLMs lack robust mechanisms for maintaining grounded spatiotemporal beliefs under adversarial conversational feedback.
