Search papers, labs, and topics across Lattice.
This paper introduces Align4D, a novel framework that enables the generation of coherent video-3D pairs from arbitrary modality inputs, addressing the challenges of dataset diversity and scalability in X-to-4D generation. The authors implement three innovative techniques鈥擮bject Distance Alignment, Motion-Geometry Joint Alignment, and Asynchronous Optimization鈥攖o enhance the fidelity of motion and geometry in generated content. Experimental results on the newly proposed X4D dataset demonstrate that Align4D outperforms existing methods, achieving state-of-the-art quality and consistency in 4D generation tasks.
Align4D transforms any input modality into high-fidelity 4D content, setting a new standard for multimodal generative models.
Generative diffusion models excel at synthesizing high-quality images, videos, and 3D content under multimodal control. However, arbitrary user-defined modality-to-4D (X-to-4D) generation remains challenging due to the high cost of constructing diverse datasets and the limited scalability of existing methods. This paper presents Align4D, a flexible framework that translates any-modal input into coherent video-3D pairs, using video to guide 4D motion and 3D data to shape 4D geometry. Align4D introduces three key techniques: (1) Object Distance Alignment, which searches Video-Aligned and Multiview-Aligned Object Distances (VAOD/MAOD), respectively, to reconcile 4D renderings with video and the priors of multiview diffusion models; (2) Motion-Geometry Joint Alignment, which constrains known and unknown views through synchronized video and 3D inputs, ensuring consistent 4D generation; and (3) Asynchronous Optimization, which decouples Gaussian attribute and deformation network training to enhance motion and geometry fidelity. We further propose the X4D dataset, which integrates prompt, image, video, and 3D data for benchmarking. Experiments on X4D and Consistent4D demonstrate that Align4D achieves state-of-the-art quality and consistency in X-to-4D generation. Project page: https://miaoqiaowei.github.io/Align4D/.
