Search papers, labs, and topics across Lattice.
The paper introduces Diff3R, a framework that integrates a differentiable 3D Gaussian Splatting (3DGS) optimization layer into feed-forward networks, enabling the prediction of optimal initializations for test-time optimization. To address the computational challenges of backpropagation, they use the Implicit Function Theorem and a matrix-free PCG solver. They also incorporate a data-driven uncertainty model to prevent overfitting and enhance robustness during optimization, leading to improved rendering quality and generalization.
Feed-forward 3D Gaussian Splatting gets a huge boost in rendering quality and generalization by learning to predict optimal initializations for test-time optimization.
Recent advances in 3D Gaussian Splatting (3DGS) present two main directions: feed-forward models offer fast inference in sparse-view settings, while per-scene optimization yields high-quality renderings but is computationally expensive. To combine the benefits of both, we introduce Diff3R, a novel framework that explicitly bridges feed-forward prediction and test-time optimization. By incorporating a differentiable 3DGS optimization layer directly into the training loop, our network learns to predict an optimal initialization for test-time optimization rather than a conventional zero-shot result. To overcome the computational cost of backpropagating through the optimization steps, we propose computing gradients via the Implicit Function Theorem and a scalable, matrix-free PCG solver tailored for 3DGS optimization. Additionally, we incorporate a data-driven uncertainty model into the optimization process by adaptively controlling how much the parameters are allowed to change during optimization. This approach effectively mitigates overfitting in under-constrained regions and increases robustness against input outliers. Since our proposed optimization layer is model-agnostic, we show that it can be seamlessly integrated into existing feed-forward 3DGS architectures for both pose-given and pose-free methods, providing improvements for test-time optimization.
