Search papers, labs, and topics across Lattice.
This paper introduces TimeSteer, a framework for inference-time speech scheduling in joint audio-visual diffusion models, enabling precise control over when utterances occur without the need for model fine-tuning. By leveraging intrinsic properties of the denoising process, such as a timing-sensitive text-to-audio cross-attention head and the organization of latent content, TimeSteer allows for effective localization and remapping of audio-visual content within user-defined intervals. Experimental results demonstrate that TimeSteer significantly enhances interval controllability compared to existing training-free methods while preserving high-quality generation outputs.
Achieving precise timing control in audio-visual generation without model retraining could revolutionize how we synchronize speech and visuals in AI applications.
Although pretrained joint audio-visual diffusion models offer rich control over \emph{what} to generate, they provide no explicit control over \emph{when} an utterance should occur. To address this, we study \emph{inference-time speech scheduling}, a novel task that places coupled speech and visual articulation within user-specified begin--end intervals without finetuning the backbone model. We uncover two intrinsic properties of the denoising process that enable this task. First, a timing-sensitive text-to-audio cross-attention head exposes each utterance's model-implied source span along the latent timeline. Second, the predicted clean latent already organizes coupled speech and visual articulation, allowing their temporal placement to be edited without regenerating the content. Building on these discoveries, we propose \textbf{TimeSteer}, a training-free framework that localizes each utterance's source span through \textbf{Source Span Localization} and transfers the associated audio-visual latent content from the source interval to the specified target interval through \textbf{Region-Aware Latent Remapping}. We further introduce \textbf{SpeechShift}, the first benchmark for interval-level speech scheduling in joint audio-visual generation. Experiments across two representative backbones show that TimeSteer substantially improves interval controllability over training-free baselines while maintaining competitive overall generation quality.
