Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This paper introduces RL3DEdit, a reinforcement learning framework for multi-view consistent 3D scene editing that leverages 2D diffusion model priors. The method uses the 3D foundation model VGGT to provide reward signals based on confidence maps and pose estimation errors, guiding the RL agent to generate 3D-consistent edits. Experiments show that RL3DEdit achieves state-of-the-art editing quality and multi-view consistency with high efficiency, addressing the challenge of scarce 3D-consistent editing data.
Forget painstakingly curating 3D-consistent datasets: this RL approach uses a 3D foundation model to *verify* consistency, turning 3D scene editing into a tractable reinforcement learning problem.
Leveraging the priors of 2D diffusion models for 3D editing has emerged as a promising paradigm. However, maintaining multi-view consistency in edited results remains challenging, and the extreme scarcity of 3D-consistent editing paired data renders supervised fine-tuning (SFT), the most effective training strategy for editing tasks, infeasible. In this paper, we observe that, while generating multi-view consistent 3D content is highly challenging, verifying 3D consistency is tractable, naturally positioning reinforcement learning (RL) as a feasible solution. Motivated by this, we propose \textbf{RL3DEdit}, a single-pass framework driven by RL optimization with novel rewards derived from the 3D foundation model, VGGT. Specifically, we leverage VGGT's robust priors learned from massive real-world data, feed the edited images, and utilize the output confidence maps and pose estimation errors as reward signals, effectively anchoring the 2D editing priors onto a 3D-consistent manifold via RL. Extensive experiments demonstrate that RL3DEdit achieves stable multi-view consistency and outperforms state-of-the-art methods in editing quality with high efficiency. To promote the development of 3D editing, we will release the code and model.