Search papers, labs, and topics across Lattice.
The paper introduces CoPE-VideoLM, a video language model that leverages video codec primitives (motion vectors and residuals) to reduce computational overhead and improve temporal coverage compared to keyframe sampling methods. They train lightweight transformer encoders to aggregate codec primitives and align their representations with image encoder embeddings using a pre-training strategy. Results show significant reductions in time-to-first-token (up to 86%) and token usage (up to 93%) while maintaining or exceeding performance on 14 video understanding benchmarks.
Video Language Models can achieve up to 86% faster time-to-first-token and 93% token reduction by ditching full-image encoding in favor of motion vectors and residuals from video codecs.
Video Language Models (VideoLMs) empower AI systems to understand temporal dynamics in videos. To fit to the maximum context window constraint, current methods use keyframe sampling which can miss both macro-level events and micro-level details due to the sparse temporal coverage. Furthermore, processing full images and their tokens for each frame incurs substantial computational overhead. To address these limitations, we propose to leverage video codec primitives (specifically motion vectors and residuals) which natively encode video redundancy and sparsity without requiring expensive full-image encoding for most frames. To this end, we introduce lightweight transformer-based encoders that aggregate codec primitives and align their representations with image encoder embeddings through a pre-training strategy that accelerates convergence during end-to-end fine-tuning. Our approach reduces the time-to-first-token by up to 86% and token usage by up to 93% compared to standard VideoLMs. Moreover, by varying the keyframe and codec primitive densities we are able to maintain or exceed performance on 14 diverse video understanding benchmarks spanning general question answering, temporal reasoning, long-form understanding, and spatial scene understanding.
Google Research