Search papers, labs, and topics across Lattice.
The paper introduces Ex-Omni-2D, an omni-modal dialogue framework that integrates text, personalized speech, and reference-conditioned video to generate visually coherent responses. By predicting a structured Visual Thought Plan (VTP) that encompasses scene, emotion, and motion, the model effectively aligns speech and video outputs without requiring extensive multi-modal supervision. The implementation achieves a competitive real-time factor of 1.293 while maintaining high-quality output, showcasing a significant advancement in expressive omni-modal dialogue systems.
Ex-Omni-2D generates visually coherent dialogue responses that seamlessly integrate text, speech, and video, all while avoiding the need for extensive multi-modal training data.
Omni-modal dialogue models can understand multimodal inputs and synthesize spoken replies, yet their responses remain visually disembodied. We introduce \textbf{Ex-Omni-2D}, an omni-modal dialogue framework that generates a coordinated response comprising text, personalized speech, and reference-conditioned video. Given a multimodal query, reference image, and reference audio, the model predicts a structured \textit{Visual Thought Plan} (VTP) describing scene, emotion, and motion, followed by response text and native multi-codebook speech units. These units form a shared acoustic-temporal interface: they are decoded into speech and aligned online with video frames. This interface enables the response and avatar pathways to be learned from heterogeneous speech, dialogue, and avatar-video data, avoiding the need for large-scale query--text--speech--video supervision. A full-sequence Video Generator serves as the primary Teacher. For efficient incremental generation, we further distill it into a few-step block-causal \emph{Streaming Student} whose Prefix Streaming mechanism carries a clean latent across consecutive chunks to reduce cumulative late-chunk degradation. With four-step inference, the complete four-GPU pipeline achieves an end-to-end RTF of 1.293 at $400\times720$/$720\times400$, providing a practical quality--efficiency operating point.
