Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
VideoLLMs exhibit a critical failure in temporal reasoning, where reversing frame order often yields unchanged predictions due to a progressive fading of temporal information from intermediate layers to the output. This paper quantifies this phenomenon using a temporal divergence vector and introduces Temporal Activation Injection (TAI), a training-free method that reinjects learned temporal signals at inference time. TAI significantly enhances temporal reasoning across multiple VideoLLMs and benchmarks without compromising non-temporal performance.
VideoLLMs forget up to 40% of temporal context by the output layer, but a simple inference-time injection can recover it.
Video Large Language Models (VideoLLMs) receive frames in sequential order and interpret how visual content evolves along the temporal axis, yet temporal reasoning remains a persistent weakness across architectures. Reversing the frame order of a video, a transformation that should invert temporal answers, often leaves the final prediction unchanged. We investigate where this failure originates by defining the temporal divergence vector $\tau_l$, the layer-wise representational difference induced by reversing temporal order. Tracking its magnitude across layers reveals a consistent temporal divergence profile where the divergence peaks at intermediate layers and progressively diminishes toward the output. We confirm this peak is specific to temporal reasoning and functionally critical for predictions, establishing that VideoLLMs acquire temporal information at intermediate layers but fail to maintain it to the output. This progressive fading motivates our method, Temporal Activation Injection (TAI), which extracts $\tau_l$ at the peak of the profile for each input and reinjects it into subsequent layers following the measured decay. TAI requires no training and consistently improves temporal reasoning across three VideoLLMs and four benchmarks with negligible impact on non-temporal tasks. Code is available at https://github.com/Youngwoo-git/Before-It-Fades.