Search papers, labs, and topics across Lattice.
This paper introduces FixAnything, a versatile model that refines 3D scene renderings by leveraging a pretrained video generative model with minimal modifications. By formulating the cleanup process as video-to-video translation and utilizing a binary mask for scene structure preservation, FixAnything effectively mitigates rendering artifacts across various 3D representations. The approach not only enhances rendering quality but also supports downstream reconstruction through direct preference optimization based on camera pose accuracy, showcasing the model's capability to replace multiple specialized pipelines with a single generalist solution.
A single model can replace multiple specialized pipelines for rendering refinement, achieving significant quality improvements with minimal finetuning.
Rendering views using 3D scene representations such as Gaussian Splatting (3DGS), Neural Radiance Fields (NeRF), meshes, or even point clouds produces artifacts when input views are sparse or target views lie far from the input. Recent work mitigates these artifacts using diffusion-based generative priors, but is specialized to individual representations and require custom architectures or extensive retraining. We present FixAnything, a single model for fixing a wide range of rendering artifacts. It does so by repurposing a pretrained video generative model, leveraging its implicit multi-view priors with only minimal modification and lightweight finetuning. Our key insight is that even noisily-rendered sequences preserve camera motion and coarse scene structure, allowing cleanup to be formulated as video-to-video translation. To control what scene structure should be preserved, we introduce a binary mask denoting the clean pixels, enabling the model to anchor its output to high-quality inputs (e.g. training views) while refining the rest. To encourage FixAnything to produce 3D-consistent renderings that support downstream reconstruction, we use camera pose accuracy (recovered via structure-from-motion) as a reward signal for direct preference optimization (DPO). Across four distinct 3D representations, FixAnything consistently improves rendering quality with lightweight finetuning, demonstrating that a single generalist video prior can replace multiple specialist refinement pipelines. The simplicity of the framework enables immediate adoption of stronger future video models without architectural redesign.
