Search papers, labs, and topics across Lattice.
TurboT2VA introduces a novel distillation and inference framework designed to accelerate a 19B-parameter joint text-to-video-audio model, addressing challenges such as modality-imbalanced optimization and continuous-time consistency training. By employing a progressive curriculum that includes discrete and continuous consistency training, the framework achieves a remarkable 20.1脳 speedup in generator latency while maintaining high-quality output across visual, audio, and synchronization dimensions. Additionally, an architecture-aware inference stack further enhances performance, yielding a 54.67脳 speedup at high resolution, underscoring the potential for efficient multimodal generation.
Achieving a staggering 54.67脳 speedup in text-to-video-audio generation without sacrificing quality could revolutionize real-time multimedia applications.
Joint text-to-video-audio generation produces synchronized visual and acoustic content, but the long sampling trajectories and heterogeneous multimodal computation of large models make inference prohibitively expensive. We present TurboT2VA, a distillation and inference framework for accelerating a 19B-parameter joint video-audio model. Large-scale T2VA distillation is challenged by modality-imbalanced optimization, the difficulty of continuous-time consistency training at scale, and the quality--diversity trade-off. TurboT2VA addresses these issues with per-modality normalization and a progressive curriculum comprising discrete consistency warm-up, continuous consistency refinement, and joint consistency--distribution matching. The curriculum first establishes a stable, diverse generation trajectory and only then introduces distribution-level refinement. On LTX-2, four-step distillation reduces generator latency from 50.52s to 2.51s at the standard evaluation resolution of 512$\times$768, achieving a 20.1$\times$ speedup while maintaining strong visual quality, audio fidelity, diversity, and video-audio synchronization. We further develop an architecture-aware inference stack that combines guarded W8A8 and fused operators, padded-text compaction, and modality-aware sparse attention while preserving dense cross-modal and text-conditioning paths. Under the high-resolution deployment setting at 1024$\times$1792, the complete stack reduces generator latency from 318.74s to 5.83s on one NVIDIA H20, achieving a 54.67$\times$ generator-only speedup. Inference code and generation demos are available at https://github.com/thu-ml/TurboDiffusion/tree/main/turbot2va.
