Search papers, labs, and topics across Lattice.
SLVMBench is introduced as the first benchmark to evaluate video large language models (video-LLMs) on their ability to learn skills from extensive video memory and apply them to real-time tasks. By presenting models with lengthy video streams that include tutorial content amidst irrelevant footage, the benchmark assesses the complete process of memorizing, extracting, and utilizing procedural knowledge. Results indicate that current video-LLMs significantly struggle with skill acquisition and application, particularly when the relevant information is embedded in long video contexts, highlighting a critical limitation in their design.
Video-LLMs fail to effectively learn and apply skills from long video memories, revealing a fundamental gap in their capabilities.
We introduce Skill Learning from Video Memory (SLVMBench), the first benchmark that jointly evaluates whether video large language models (video-LLMs) can learn skills from long video memory and apply them to real-time tasks. SLVMBench presents models with 2-3 hour video streams that contain a tutorial video embedded in a stream of arbitrary irrelevant videos, resembling real-world human learning practices. Video-LLMs are asked to apply the acquired skill to answer real-time questions about an ongoing video. Unlike long-video understanding benchmarks that emphasize passive comprehension and skill-learning benchmarks that rely on short, immediate demonstrations, SLVMBench tests the full pipeline of memorizing and extracting procedural knowledge, as well as transferring it to real-time tasks. Moreover, rigorous human annotations feature sub-second-level temporal calibration, manually engineered questions eliminating common-sense guessing, and collated tutorials to ensure coverage of the required skills. Evaluations on state-of-the-art proprietary and open-source video LLMs show that video-LLMs struggle substantially with learning and applying skill knowledge from videos. Moreover, performance degrades markedly when the skill knowledge is placed within a long video memory. These results reveal a key limitation of existing video LLMs and position SLVMBench as the first benchmark for studying real-time skill acquisition and application from long-context video memory.
Tsinghua AI