Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Multi-Stream Scene Script (MTSS), a new paradigm for video captioning that factorizes videos into complementary streams (Reference, Shot, Event, and Global) and reconnects them through explicit identity and temporal links. By decoupling visual, auditory, and identity information, MTSS enhances video understanding, evidenced by a 25% error reduction on Video-SALMONN-2 and a 67% performance gain on Daily-Omni. Furthermore, MTSS improves multi-shot video generation, yielding significant human-rated improvements in identity consistency, audio-visual alignment, and temporal controllability.
Stop treating videos as monolithic narratives: factorizing video descriptions into streams grounded by identity and time unlocks better video understanding and generation, even closing the gap between small and large MLLMs.
Advances in Multimodal Large Language Models (MLLMs) are transforming video captioning from a descriptive endpoint into a semantic interface for both video understanding and generation. However, the dominant paradigm still casts videos as monolithic narrative paragraphs that entangle visual, auditory, and identity information. This dense coupling not only compromises representational fidelity but also limits scalability, since even local edits can trigger global rewrites. To address this structural bottleneck, we propose Multi-Stream Scene Script (MTSS), a novel paradigm that replaces monolithic text with factorized and explicitly grounded scene descriptions. MTSS is built on two core principles: Stream Factorization, which decouples a video into complementary streams (Reference, Shot, Event, and Global), and Relational Grounding, which reconnects these isolated streams through explicit identity and temporal links to maintain holistic video consistency. Extensive experiments demonstrate that MTSS consistently enhances video understanding across various models, achieving an average reduction of 25% in the total error rate on Video-SALMONN-2 and an average performance gain of 67% on the Daily-Omni reasoning benchmark. It also narrows the performance gap between smaller and larger MLLMs, indicating a substantially more learnable caption interface. Finally, even without architectural adaptation, replacing monolithic prompts with MTSS in multi-shot video generation yields substantial human-rated improvements: a 45% boost in cross-shot identity consistency, a 56% boost in audio-visual alignment, and a 71% boost in temporal controllability.