Search papers, labs, and topics across Lattice.
Ripple introduces a real-time joint audio-video generation system that leverages a cross-modal recurrent memory mechanism to enhance streaming performance. This approach combines fixed-length sliding-window attention with modality-specific memory states, allowing for efficient long-form generation while maintaining audio-visual synchronization. The system achieves approximately 28 frames per second at 480P resolution, significantly outperforming existing methods in both short and long video benchmarks.
Ripple achieves real-time audio-video generation at 28 FPS while maintaining coherent long-form content, a breakthrough for streaming applications.
Audio-video generative models achieve impressive quality but suffer from high latency, making them unsuitable for real-time applications. Although several streaming audio-video generation methods have been proposed, they remain costly and fail to support long-form generation. To address this, we propose \textbf{Ripple}, a real-time joint audio-video generation system with a cross-modal recurrent memory mechanism. To enable efficient streaming inference while preserving long-term context, Ripple combines a fixed-length sliding-window attention with modality-specific memory states that continuously summarize audio and video context. Cross-modal memory interaction is further introduced to enhance audio-visual synchronization. To learn this memory-augmented model effectively, we devise a three-stage training recipe: (1) adapting a bidirectional audio-video teacher to block-wise causal attention with simulated memory, (2) optimizing the memory construction and interaction pipeline through end-to-end distillation, and (3) applying online reinforcement post-training tailored for streaming audio-video generation. As a result, Ripple achieves ~28 FPS at 480P resolution, over faster than the teacher, while capable of coherent long-form generation. Extensive experiments on both short-video and long-video benchmarks demonstrate our superior performance over existing offline and online joint audio-video generation methods.
