Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to long video understanding by utilizing Neural Knowledge Representations (NKR), which encapsulate a video's semantic content as optimized network weights rather than traditional token streams or databases. The method employs Agentic Knowledge Distillation (AKD) to generate dense descriptions and question-answer pairs, allowing the NKR to serve as a lightweight, portable asset that can be mounted onto a frozen Vision-Language Model (VLM) for efficient query-based understanding. Experimental results on the LVBench benchmark demonstrate that this approach not only matches state-of-the-art performance but also reduces end-to-end latency by over two orders of magnitude, significantly enhancing interactive long-video analysis.
Transforming long videos into lightweight, reusable knowledge representations could redefine the efficiency of video understanding systems.
We propose a new paradigm for long video understanding by treating a long video as a Neural Knowledge Representation (NKR). NKR represents video contents neither as a stream of tokens nor pre-organized databases, but as an individual small portion of network weights attached to the VLM backbone. The NKR weights are optimized to encapsulate the video's semantic content via a novel Agentic Knowledge Distillation (AKD) process, where an agent automatically synthesizes dense descriptions and question-answer pairs to distill the video's knowledge into the NKR. While AKD serves as a comprehensive, one-time encoding phase, the resulting NKR transforms the video into a portable, reusable asset. At inference, the lightweight NKR is mounted onto a frozen Vision-Language Model (VLM), enabling direct, query-based understanding without reloading or re-encoding the original video. This approach decouples video length from inference cost, offering high amortized efficiency for multi-turn video understanding. Experiments on the LVBench benchmark show our method achieves performance comparable to state-of-the-art approaches while reducing end-to-end latency by over two orders of magnitude, opening new possibilities for interactive long-video understanding.
Microsoft Research