Search papers, labs, and topics across Lattice.
This paper introduces ViCoStream, a stage-wise coordinated streaming framework designed to optimize the inference of VideoLLMs for real-time applications. By integrating techniques such as chunk-wise execution and visual token control, the framework achieves an impressive throughput of 134 FPS while keeping query latency below 50 ms on a single A100 GPU. The systematic analysis of bottleneck migration highlights how various factors like chunk size and attention locality influence the trade-off between throughput and accuracy, demonstrating that ViCoStream can maintain performance close to full-history baselines.
Achieving 134 FPS with under 50 ms latency, ViCoStream redefines the capabilities of VideoLLMs for real-time streaming applications.
Streaming VideoLLMs must continuously process incoming video while maintaining low query latency, making both video-ingestion throughput and query-time responsiveness critical for real-time deployment. Existing methods largely focus on accelerating individual modules, such as visual encoding, token pruning, or KV-cache compression, but provide limited insight into whether the resulting system can sustain real-time streaming performance. We formulate streaming VideoLLM inference as a coordinated pipeline spanning visual preprocessing, visual encoding, token dropping, and LLM prefilling/decoding. Building on this formulation, we propose ViCoStream (Video Coordinated Streaming), a stage-wise coordinated streaming framework that combines chunk-wise execution, CUDA-stream overlap, visual token control, bounded visual attention, and query-side retrieval to bound per-chunk computation and memory costs. We further provide a systematic study of bottleneck migration, revealing how chunk size, token retention, attention locality, and retrieval scope shape the throughput-accuracy trade-off. Experiments with Qwen2.5-VL-3B/7B-Instruct across multiple streaming benchmarks show that ViCoStream achieves 134 FPS video throughput and less than 50 ms TTFT on a single A100 GPU while maintaining accuracy close to full-history baselines.
