Search papers, labs, and topics across Lattice.
The authors decouple streaming talking-head generation into a lightweight causal autoregressive motion transformer and a block-causal diffusion renderer, moving conditioning out of the pixel latent space. To resolve autoregressive exposure bias without a dedicated motion teacher, a single frozen bidirectional video diffusion model is used to distill the causal renderer while simultaneously supervising the motion generator via rendered video rollouts. This parallel causal pipeline eliminates the traditional fidelity penalty of two-stage models, delivering full-fidelity video at 15.4 FPS with 1.3 seconds of latency.
Directly conditioning video diffusion on audio wastes massive capacity on static background and identity pixels鈥攔outing control transitively through a causal motion latent distilled under a single frozen video teacher achieves real-time streaming at 15.4 FPS with zero fidelity loss.
Streaming talking-head generation produces each frame as its driving audio arrives, yet fidelity and efficiency have so far pulled in opposite directions: end-to-end methods condition a video diffusion model on audio directly and achieve high quality but only at large scale, while cheaper two-stage methods generate an intermediate motion representation and trail in fidelity. We argue the cost of the former lies in the target of fusion: the video latent is dominated by identity, appearance and background, none of which audio bears on, so coupling audio to every pixel blurs detail and wastes capacity. We instead fuse conditions in a low-dimensional identity-disentangled motion space, routing audio and motion captions by their temporal granularity, and generate motion latents with a small causal autoregressive transformer that a pretrained diffusion renderer turns into video. Conditions thus control video transitively, and high fidelity no longer requires a large backbone. Streaming this decomposition needs both models to be causal, and the exposure-bias problem could be solved by self-forcing given a bidirectional teacher. But there is no such teacher in motion space. Our decoupled self-forcing distillation resolves both models under one frozen teacher: conditioned on motion, it distills the renderer into a block-causal student; unconditionally, it scores rendered rollouts against real videos, supervising motion by the video it produces. This lifts the fidelity ceiling from the motion generator onto the stronger renderer. The two models run as parallel causal streams, reaching 15.4 FPS at 1.3 s latency with no quality degradation.
