Search papers, labs, and topics across Lattice.
This paper introduces a State-Aware Representation Learning Approach (SARLA) for cross-modal UAV object tracking, addressing challenges posed by modality switching that leads to appearance changes and spatial shifts. The method incorporates two innovative modules: the Modality State Aware Representation Module (MSARM) for learning appearance correlations and the Spatial State Aware Representation Module (SSARM) for modeling spatial correlations across frames. Extensive experiments on a newly established benchmark, CM-UOT, demonstrate that SARLA outperforms 20 existing tracking methods, showcasing its effectiveness in robust tracking scenarios.
Robust UAV tracking is revolutionized with SARLA, which adeptly manages sudden appearance and spatial shifts caused by modality switching.
Unmanned Aerial Vehicle (UAV) object tracking has emerged as a popular research field with broad practical applications. Modern UAVs are increasingly equipped with both visible light and thermal infrared sensors. However, due to constraints in communication bandwidth, computational resources and power consumption, current systems often activate one modality and switch between modalities to maintain robust tracking in complex scenarios. Such modality switch inevitably leads to significant appearance change and sudden spatial shift, posing great challenges for existing tracking algorithms. To handle this problem, we propose a novel State-Aware Representation Learning Approach called SARLA, which perceives the inconsistent modality states of current frame with template and last frame in the target representations to adapt to the sudden changes in both appearance and position, for robust cross-modal object tracking. In particular, we propose the Modality State Aware Representation Module (MSARM) and Spatial State Aware Representation Module (SSARM). MSARM guides the model to learn appearance correlation, bridging the modality gap, while SSARM models cross-frame spatial correlation to mitigate sudden spatial shift impacts. In addition, we design a spatial shift prediction loss to further handle the effects of spatial variation caused by modality switch. To promote the development of this research field, we establish a large-scale video benchmark called CM-UOT, which consists of 1079 cross-modal sequences with an average video length greater than 621 frames and encompasses over 671K frames in total. Extensive experiments on CM-UOT dataset demonstrate the superior performance of the proposed SARLA against 20 excellent tracking methods. The source code, datasets, and evaluation protocols associated with this work are publicly available at: https://github.com/hongsmile365/sarla-.
