Search papers, labs, and topics across Lattice.
This paper addresses the challenge of RGB-D semantic segmentation in scenarios where one modality is missing, a common occurrence in real-world applications due to sensor failures. The authors introduce a novel training approach called Condition Dropout (ConD), which enhances model robustness by simulating missing modalities during continued training while maintaining accuracy with complete inputs. Experimental results on benchmark datasets demonstrate that ConD not only improves performance under missing modalities but also achieves slight accuracy gains when both modalities are present.
Condition Dropout transforms RGB-D models by enabling them to maintain performance even when one modality is absent, effectively turning a limitation into an advantage.
RGB-D semantic segmentation has achieved remarkable progress, yet most models assume that RGB and depth are always available. In practice, failures or occlusions of surveillance sensors often remove one modality. Although RGB or depth alone can contain sufficient cues, models trained only on full-modality inputs fail to exploit the remaining modality once one is missing, causing severe degradation. We tackle this issue with a simple continued-training paradigm, \emph{Condition Dropout (ConD)}, which mitigates degradation while preserving full-modality accuracy. Starting from a pretrained RGB-D model, ConD adds a second stage that randomly simulates complete, RGB-missing, and depth-missing inputs, freezes the original encoders, and trains copied encoders with zero-initialized feature injection. Experiments on NYU-Depth V2 and SUN RGB-D show that ConD improves robustness under missing modalities and even yields slight gains when modalities are complete. Our code will be made publicly available upon acceptance.
