Search papers, labs, and topics across Lattice.
This paper addresses the issue of dynamic collapse in self-forcing distillation for audio-driven avatar generation, where models converge to static outputs at the expense of temporal dynamics. The authors identify the causes of this collapse as the reverse KL objective in Distribution Matching Distillation (DMD) and unanchored self-conditioning, which leads to a detrimental feedback loop. They introduce DynaForcing, a novel training framework that employs Hybrid Forcing, Dynamics-Aware Reward Regularization, and Reference Perturbation to significantly enhance motion dynamics and visual quality, achieving teacher-comparable performance without early stopping.
DynaForcing not only recovers dynamic motion in avatars but also improves visual quality, resolving a critical trade-off in real-time streaming applications.
Audio-driven avatar generation requires realistic lip-sync, expressive motion, and real-time streaming. Recent work achieves the latter via self-forcing with Distribution Matching Distillation (DMD), but this paradigm suffers from a critical failure that has not been systematically characterized: dynamic collapse, where the student model converges to a near-static optimum with high perceptual quality but severely suppressed temporal dynamics. We trace this to two causes: the reverse KL objective in DMD, which biases toward low-motion modes, and unanchored self-conditioning, which creates a feedback loop that amplifies collapse. This is especially harmful for avatars, where even subtle motion loss breaks lip-sync and expression. To address this, we propose DynaForcing, a training framework with three complementary strategies applied at different levels. Specifically, Hybrid Forcing anchors rollouts to ground-truth dynamics at the data level to break the feedback loop. Dynamics-Aware Reward Regularization introduces explicit motion rewards via the RL interpretation of DMD to counteract the reverse KL bias at the loss level. Reference Perturbation perturbs reference images to decouple identity from static details, forcing the model to rely on audio for motion at the conditioning level. We further introduce computation graph pruning and gradient replay, reducing the GPU footprint of self-forcing by over an order of magnitude. Experiments show that DynaForcing recovers dynamics to teacher-comparable levels (Dyn-Deg: 0.31 ->0.73, Sync-C: 7.03 ->7.68) while improving visual quality, resolving the quality-dynamics trade-off throughout training without early stopping.
