Search papers, labs, and topics across Lattice.
This paper addresses the challenge of adapting existing remote sensing models to new satellite sensors and modalities with minimal retraining. The authors introduce DeluluNet, a modular architecture capable of handling modality transfer, addition, and peeking scenarios through an innovative approach called modality hallucination, which predicts missing modality representations from available data. The key finding is that DeluluNet maintains predictive performance despite changes in input modalities, offering a significant advantage in dynamic remote sensing environments.
DeluluNet can adapt to new satellite sensors without extensive retraining, maintaining predictive accuracy in a rapidly evolving landscape.
Machine learning models for remote sensing are trained and deployed on a static set of modalities. However, as we equip newer satellites with novel sensors and retire old ones, practitioners may wish to deploy an existing model on a substitution, superset, or subset of modalities with minimal retraining given data availability or practical computational constraints. We study the setting of updating existing models to changing modalities and identify three main scenarios: Modality Transfer (substitution), Addition (superset), and Peeking (subset). We propose DeluluNet, an architecture with modular components for all three changing modality scenarios. DeluluNet is trained end-to-end, learning a multi-modal model from a unimodal teacher and unlabeled multimodal data via modality hallucination--predicting missing modality representations from those that are present. As a result, DeluluNet can keep predicting even when input modalities change, providing a practical alternative to re-labeling and re-training in a changing world.
