Search papers, labs, and topics across Lattice.
This paper introduces VT-Contrast, a novel representation-level temporal counterfactual objective designed to enhance temporal understanding in Video Language Models (VideoLMs). By contrasting order-preserving video token representations with reordered counterfactuals, the method effectively exposes temporal dynamics that are often overlooked in traditional training setups. The results demonstrate significant improvements in performance across various temporal understanding benchmarks, highlighting the importance of integrating temporal supervision in VideoLMs.
VideoLMs can significantly enhance their temporal understanding by leveraging a counterfactual approach that reveals hidden dynamics in video data.
Seeing frames in order does not mean representing time. Modern VideoLMs receive ordered video streams, yet their main supervision acts on generated text rather than video-token representations where event dynamics should first emerge. This mismatch allows models to learn temporal answers from shortcuts such as objects, scenes, and language priors, without requiring internal video representations to capture event progression. To address this, we propose VT-Contrast, a representation-level temporal counterfactual objective for VideoLMs. Its design asks where temporal supervision should act and what temporal differences it should expose. VT-Contrast supervises selected late-layer last-frame video tokens, where temporal information is expected to be integrated before language generation, and contrasts order-preserving views with same-video reordered counterfactuals graded by Kendall tau distance. It requires no architectural changes, is compatible with diverse VideoLM training tasks, and improves overall performance across temporal understanding benchmarks. Our code is available at https://github.com/ANDgate99/VT-Contrast.
