Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces ReNoV, a diffusion-based novel view synthesis framework that leverages external visual representations as conditions to improve geometric consistency. The authors analyze the correspondence capabilities within spatial attention of external representations and use this to guide the diffusion process via representation projection modules. Experiments demonstrate that ReNoV achieves significant improvements in reconstruction fidelity and inpainting quality compared to existing diffusion-based methods, particularly in scenarios with sparse, unposed images.
Exploiting geometric and semantic correspondence in external visual representations unlocks high-fidelity novel view synthesis with diffusion models, even from sparse and unposed images.
We propose a novel framework for diffusion-based novel view synthesis in which we leverage external representations as conditions, harnessing their geometric and semantic correspondence properties for enhanced geometric consistency in generated novel viewpoints. First, we provide a detailed analysis exploring the correspondence capabilities emergent in the spatial attention of external visual representations. Building from these insights, we propose a representation-guided novel view synthesis through dedicated representation projection modules that inject external representations into the diffusion process, a methodology named ReNoV, short for representation-guided novel view synthesis. Our experiments show that this design yields marked improvements in both reconstruction fidelity and inpainting quality, outperforming prior diffusion-based novel-view methods on standard benchmarks and enabling robust synthesis from sparse, unposed image collections.