Search papers, labs, and topics across Lattice.
This paper introduces StreamChar, a novel framework for real-time streaming audio-video generation that effectively addresses the challenges of transcript-audio misalignment and visual drift in character animation. By decoupling long-horizon orchestration from short-window audio-video denoising, StreamChar leverages an LLM-based orchestrator for frame-aligned audio conditions and a joint audio-video DiT for enhanced denoising. Experimental results demonstrate that StreamChar achieves real-time performance on a single H100 GPU while significantly improving transcript fidelity, audio-visual synchronization, and visual quality compared to existing methods.
StreamChar achieves real-time audio-video generation with unprecedented fidelity and synchronization, overcoming the limitations of traditional autoregressive models.
Real-time streaming joint audio-video generation for character animation requires a generator to speak the requested transcript, maintain visual identity across chunks, and run within a strict playback budget. These requirements are difficult to satisfy simultaneously: chunk-wise autoregressive generation can accumulate transcript-audio misalignment and visual drift, while the few-step distillation needed for low latency often degrades spatial diversity and temporal quality. We present StreamChar, a streaming framework that separates long-horizon orchestration from short-window audio-video denoising. An LLM-based orchestrator uses the transcript and historical context to produce frame-aligned audio conditions, and a joint audio-video DiT performs local bidirectional denoising with reference and motion-frame conditioning. For efficient deployment, we use a two-stage distillation pipeline that first compresses the sampler and then fine-tunes the student under online chunk rollouts. A progress-aware pointer aligns partial transcripts with generated audio during rollout training, and a sink-chunk memory provides a persistent visual anchor for reducing long-horizon drift. Experiments on short-clip and long-horizon protocols show that StreamChar runs in real time on a single H100 GPU and provides a favorable system-level trade-off among transcript fidelity, audio-visual synchronization, visual quality, and streaming stability compared with recent joint and audio-driven baselines.
DAMO