Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces the Hash-Atlas network, which reformulates 3D scene editing as operations on 2D atlas images, allowing for a decoupled workflow between 2D editing and 3D reconstruction. Building on this, the authors present CE3D++, a dialogue-based editing system powered by a large language model (LLM) that interprets arbitrary textual inputs and autonomously invokes corresponding visual models. Experimental results show that CE3D++ can effectively manage multiple visual models for diverse editing effects while demonstrating strong scene comprehension and multi-round dialog capabilities.
CE3D++ enables flexible, dialogue-driven 3D and 4D scene editing by leveraging a large language model to interpret user intentions and coordinate visual tools seamlessly.
Recent work on image content manipulation based on vision-language pre-training models has been effectively extended to text-driven 3D scene editing. However, existing schemes for 3D scene editing still have certain shortcomings, hindering their further development as interactive design tools. Such schemes typically adhere to fixed input patterns, limiting flexibility in text input. Furthermore, their editing capabilities are constrained by a single or a few 2D visual models and require intricate pipeline design to integrate these models into 3D reconstruction processes. To address the aforementioned issues, we propose the Hash-Atlas network, which reformulates 3D scene editing as operations on 2D atlas images, thereby achieving a workflow decoupling of the 2D editing and 3D reconstruction processes. Building on this foundation, we introduce a dialogue-based 3D scene editing approach, termed CE3D++, which is centered on a large language model (LLM) that allows arbitrary textual input from users and interprets their intentions, subsequently facilitating the autonomous invocation of the corresponding visual models. Additionally, we extend CE3D++ to monocular 4D scenes by imposing motion constraints on moving objects and further fine-tuning the LLM by creating a trajectory dataset related to editing tasks, which enables the smaller LLM to schedule up to 30 different visual tools accurately. Experimental results demonstrate that CE3D++ effectively integrates multiple visual models to achieve diverse visual editing effects, possessing strong scene comprehension and multi-round dialog capabilities. The source codes and trained models are available at https://github.com/Fangkang515/CE3D.