Search papers, labs, and topics across Lattice.
This paper introduces SlotNarrative, a novel slot-based interface for video large language models (Video-LLMs) that organizes video content into persistent object narratives using compact object-state tokens. By grouping visual features into object-like slots and utilizing a lightweight memory system for temporal correspondence, SlotNarrative significantly reduces the token count required for video representation while maintaining accuracy. Experimental results demonstrate that this approach achieves a favorable balance between visual-token efficiency and performance across multiple datasets, marking a substantial advancement in video understanding tasks.
SlotNarrative streamlines video language modeling by reducing token usage to just 144 positions, without sacrificing accuracy.
Video large language models (Video-LLMs) have made strong progress in open-ended video understanding. However, their visual interfaces remain token-intensive and provide limited explicit structure for linking recurring object evidence across time. We introduce SlotNarrative, a slot-based interface that organizes a video into persistent object narratives represented by compact object-state tokens. Rather than compressing frame-wise features before establishing temporal correspondence, SlotNarrative first groups visual features into object-like slots and then associates recurring observations with clip-level object entries through a lightweight, parameter-free memory that integrates multiple complementary matching cues. Each retained entry is serialized into two token types: an identity token that summarizes persistent object appearance and a set of state tokens that encode segment-level appearance, geometry, visibility, and trajectory information. This design yields an interface of only 144 allocated visual-token positions for a frozen Video-LLM, independent of the number of sampled frames. Across multiple datasets, SlotNarrative achieves a favorable trade-off between accuracy and visual-token count compared with prior compact Video-LLM interfaces. Experimental results establish persistent object narratives as a compact, structured, and temporally organized visual interface for Video-LLMs. Our code will be made publicly available.
