Search papers, labs, and topics across Lattice.
This paper introduces the Spatio-temporal Conditional Denoising Transformer (SCDT) for RGBT tracking, addressing the challenges of missing modalities that lead to degraded multimodal feature representations. By integrating spatial cues and temporal context, SCDT effectively reconstructs missing information and enhances weak modalities within a unified framework, showcasing its adaptability to both incomplete and complete data scenarios. Experimental results on three public benchmark datasets reveal that SCDT consistently outperforms existing state-of-the-art methods, highlighting its robustness in modality-missing situations.
SCDT outperforms existing methods by unifying feature learning for both modality-missing and complete scenarios without altering architecture or parameters.
Missing modalities in RGBT tracking often lead to incomplete and unstable multimodal feature representations that greatly degrade the performance. Existing methods typically attempt to recover missing modalities from available ones, but the quality of data generated in challenging scenarios might be unsatisfactory. In addition, current approaches exhibit limited flexibility in processing both missing and complete data. To overcome these limitations, we propose a Spatio-temporal Conditional Denoising Transformer (SCDT), which integrates the spatial cues and the temporal context to adaptively perform information reconstruction of missing modalities and feature enhancement of weak modalities in a unified framework, for robust modality-missing RGBT tracking. In particular, SCDT leverages the short-term temporal cues from recent historical frames to capture the fine-grained temporal correlations and the long-term temporal cues encoding modality evolution to capture the global context. By jointly exploiting long short-term temporal contexts as the conditions, SCDT progressively guides noisy features of available modalities to learn reliable and temporally consistent multimodal representations. Furthermore, SCDT introduces a noisemodulated adaptation mechanism that dynamically adjusts its behavior according to the modal availability, enabling a single framework to unify feature learning under both modality-missing and complete scenarios without changing the architecture or parameters. Extensive experiments on three public benchmark datasets demonstrate that our method consistently outperforms state-of-the-art methods. The code is available here.
