Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This paper introduces Parallel Tube Decoding (PTD) for spatio-temporal video grounding, addressing the inefficiencies of autoregressive localization methods that lead to increased decoding latency and error propagation. By decomposing the grounding process into a temporal block and time-conditioned spatial blocks that are decoded simultaneously, PTD achieves a significant reduction in both token-level and trajectory-level dependencies. The method demonstrates a 79x reduction in Tube Completion Latency and a 92x increase in spatial decoding throughput on the VidSTG benchmark, while also enhancing grounding accuracy with a compact 4B model backbone.
Achieving a staggering 79x reduction in decoding latency, Parallel Tube Decoding redefines efficiency in spatio-temporal video grounding.
Spatio-temporal video grounding (STVG) requires models to identify when a referred event occurs and localize the target entity throughout that interval. Existing multimodal large language models typically serialize dense localization trajectories autoregressively, causing decoding latency to grow with tube length and allowing localization errors to propagate across time. We introduce Parallel Tube Decoding (PTD), a generative formulation that decomposes grounding into a temporal block followed by time-conditioned spatial blocks decoded simultaneously. This removes both token-level and trajectory-level dependencies, reducing the sequential decoding depth to a fixed 1 + 1 rounds, independent of tube length. To enable parallel spatial generation, we introduce Decoupled Block Attention, which preserves access to shared video-query context while eliminating cross-box dependencies, together with localization-aware policy optimization for temporal boundaries and spatial geometry. On VidSTG, PTD reduces Tube Completion Latency by 79x and increases spatial decoding throughput by 92x over standard autoregressive decoding, while also improving grounding accuracy. With a compact 4B backbone, our model performs favorably well on VidSTG and HC-STVG, and generalizes zero-shot to temporal grounding, grounded VideoQA, and referring video object tracking. Our results show parallel tube generation is an efficient and effective alternative to autoregressive localization in videos.