Search papers, labs, and topics across Lattice.
This paper introduces HelloWorld, a video world model designed to facilitate natural social interactions between users and in-world characters. By employing a self-distillation pipeline for fine-tuning and a training-free module for temporal localization, HelloWorld enables characters to respond to user prompts with gestures and speech in a coherent manner. Experimental results show that HelloWorld outperforms existing models in interaction quality while preserving high visual fidelity and accurate camera motion tracking.
HelloWorld allows users to engage in seamless social interactions with video characters, achieving superior quality without sacrificing visual aesthetics.
Despite the remarkable recent progress of video world models, social interaction between users and the characters within these worlds remains unsupported. To fill this gap, we present HelloWorld, a video world model that enables social interaction with in-world characters. With a single button press, users can prompt the on-screen character to respond toward the camera, e.g., turning to the viewer, waving, nodding, or speaking a short greeting. To make these interactions natural, we propose a self-distillation pipeline that finetunes the video generation model on data synthesized by itself. Each synthesized clip contains both social interactions and camera motion, allowing the model to learn camera-pose conditioning without degrading interaction quality. At inference, we further introduce a training-free module that determines when the interaction occurs. Upon a button press, it modulates the cross-attention masks of the DiT so that the interaction-related text prompt attends only to the frames within the press window, temporally localizing the character's response. We further build HelloWorldBench, a 400-sample benchmark with three social interaction metrics alongside three conventional metrics, for evaluation. Experiments demonstrate that HelloWorld surpasses a variety of baselines in interaction quality, while maintaining state-of-the-art picture aesthetics and camera-pose following. Project page: https://github.com/AlayaLab/HelloWorld
