Search papers, labs, and topics across Lattice.
This paper introduces Hallo4D, a novel framework designed to address spatiotemporal hallucinations in 3D and 4D content generation, which are prevalent due to the reliance on 2D diffusion supervision. By employing a generation-detection-correction paradigm that utilizes large multimodal language models (LMMs) for identifying inconsistencies, the framework optimizes image-space consistency through a consensus-driven approach without the need for retraining. Experimental results show that Hallo4D significantly outperforms existing methods, providing a scalable solution for generating consistent visual content across multiple viewpoints and timeframes.
Spatial hallucinations in 3D generation can be effectively mitigated using a model-agnostic framework that leverages multimodal language models for consistency optimization.
While recent advances in 3D generation have enabled impressive visual synthesis, existing methods often rely on 2D diffusion supervision without explicit mechanisms for geometric consistency, leading to spatial hallucinations such as duplicated structures and misaligned geometry. These issues become more severe in 4D generation, where maintaining consistency across viewpoints and temporal evolution introduces additional challenges, including jitter, identity flicker, and structural drift. We present \textbf{Hallo4D}, a unified and model-agnostic framework for mitigating spatiotemporal hallucinations in 3D and 4D content generation. Hallo4D introduces a generation-detection-correction paradigm that leverages large multimodal language models (LMMs) to identify and summarize spatial and temporal inconsistencies from multi-view and multi-frame renderings. These insights guide a consensus-driven image-space consistency optimization, where an LMM-based selector evaluates candidate corrections through multi-model voting, without requiring retraining or architectural modifications. To further improve temporal consistency and optimization efficiency, Hallo4D incorporates motion-aware keyframe sampling, LMM-guided initialization, and appearance alignment. We additionally introduce exposure-aware optimization and visibility pruning to enhance robustness under challenging viewpoints. Extensive experiments demonstrate that Hallo4D consistently outperforms strong baselines across diverse 3D and 4D generation settings, providing a scalable and generalizable solution for consistency-aware content generation.
