Search papers, labs, and topics across Lattice.
This paper introduces SPoILeR, a novel method for generating multi-view consistent renderings of unconventional imaging modalities, such as infrared and polarimetric data, using primarily RGB input. By leveraging a multimodal pre-training phase, the model learns the intermodal correlations, enabling it to predict accurate renderings even when no direct samples from the unconventional modalities are available. Experimental results demonstrate that SPoILeR can effectively synthesize these modalities, significantly reducing the need for expensive sensors and calibrated setups in novel view synthesis tasks.
Achieving accurate infrared and polarimetric renderings from just RGB images could revolutionize how we approach multimodal scene reconstruction.
Neural rendering techniques allow for accurate reconstruction of the geometry and color appearance of 3D scenes. Some methods have extended their use to additional imaging modalities, such as multispectral, infrared, or polarimetric data. However, all of these approaches require expensive sensors and calibrated setups to capture new multimodal frames for each new scene. We propose Spectral and Polarimetric Implicit Learned Representation (SPoILeR), a novel method to obtain multi-view consistent renderings of unconventional modalities for scenes where either only RGB frames or very few of the additional modalities are available. Thanks to a multimodal pre-training phase, the model learns the mutual correlation between different modalities. This step allows predicting accurate renderings of unconventional modalities during a fine-tuning phase supervised only by RGB images. Experimental results show that the approach can accurately render infrared, polarimetric, and multispectral frames for scenes where no input sample captured by these types of sensors is provided.
