Search papers, labs, and topics across Lattice.
This paper adapts Masked Diffusion Vision-Language Models (MDVLMs) to the task of Temporal Action Localization (TAL) to overcome limitations of autoregressive decoders. They introduce a Planned Training Objective with boundary-aware masking and step-weighted reconstruction, along with a Step-Level IoU Reward to address TAL-specific mismatches in standard masked diffusion training. Experiments on ActivityNet-RTL, ActivityNet-1.3, and THUMOS-14 demonstrate that MDVLM-TAL improves temporal reasoning and boundary localization, especially under stricter temporal IoU criteria, compared to autoregressive vision-language baselines.
Ditching left-to-right thinking boosts video understanding: Masked diffusion models let AI revise its guesses about when actions happen in videos, leading to more precise timing.
Temporal action localization (TAL) requires recognizing the target event and localizing its start and end times precisely in untrimmed videos. Recent vision-language formulations improve semantic reasoning and support language-conditioned outputs, but their autoregressive decoders still generate tokens from left to right, preventing later semantic evidence from revising earlier timestamp predictions. We adapt masked diffusion vision-language models (MDVLMs) to TAL so that semantic tokens and boundary tokens remain editable throughout iterative denoising with bidirectional attention, allowing temporal boundaries and semantic content to be refined jointly. Direct adaptation, however, creates two TAL-specific mismatches: standard masked diffusion training corrupts all positions uniformly at random, but the time tokens are more reliable when enough semantic context is available; and token-level cross-entropy does not reflect temporal IoU. To address these mismatches, we introduce a Planned Training Objective that uses boundary-aware masking and step-weighted reconstruction to rehearse the late recovery of time tokens, together with a Step-Level IoU Reward that provides overlap-aware supervision during denoising. A standard sequence-level cross-entropy term provides the base reconstruction signal. Experiments on ActivityNet-RTL, ActivityNet-1.3, and THUMOS-14 show that MDVLM-TAL improves both temporal reasoning and boundary localization over autoregressive vision-language baselines, with especially strong gains under stricter temporal IoU criteria.
