Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
This paper introduces FastBench, a novel benchmark designed to evaluate streaming Video Large Language Models (VLMs) on high-dynamic real-world video streams, addressing the limitations of existing benchmarks that focus on low-dynamic scenarios. The benchmark's trajectory-grounded pipeline, involving QA generation, answer verification via SAM3 and CoTracker3, and human inspection, reveals significant performance gaps, with even the strongest model achieving only 50.7%. The study also presents ProactiveFrame, a training-free baseline that demonstrates the challenges VLMs face in adaptively focusing on critical temporal information within streams.
Even state-of-the-art streaming VLMs miss crucial fast-paced events in real-world video, scoring below 51% on a new high-dynamic benchmark.
Streaming Video Large Language Models (VLMs) enable continuous video understanding, yet existing benchmarks focus on low-dynamic scenarios. Under bounded context budgets, models must balance temporal history, spatial resolution, and temporal granularity; sparse sampling at 1--2 FPS misses fast events. We introduce FastBench to evaluate high-dynamic perception in real-world video streams. Its trajectory-grounded pipeline combines QA generation from high-FPS clips, filtering of questions answerable at 2 FPS, answer verification using SAM3 and CoTracker3 trajectories, and three rounds of human inspection. FastBench contains 306 QA pairs across eight domains, six capabilities, and forward, instant, and backward temporal scopes, with human-annotated evidence intervals. We also present ProactiveFrame, a training-free baseline that adjusts incoming frame rates through text tokens. A dual-tier sliding window retains recent high-FPS observations while downsampling older ones into sparse history. Experiments reveal substantial limitations: the strongest model, Gemini-3.5-Flash, scores only 50.7%. Denser sampling improves Qwen3-VL-8B from 32.9% at 2 FPS to 44.6% at 24 FPS, but gains saturate as history is compressed. ProactiveFrame outperforms sparse uniform sampling by 5.4 and 1.5 percentage points, yet remains well below oracle-guided focusing, showing that current VLMs struggle to determine from the stream alone when finer temporal perception is needed. FastBench provides a testbed for high-dynamic streaming video understanding. Code and data: https://github.com/Ashone3/FastBench.
Tsinghua AI