Search papers, labs, and topics across Lattice.
This paper introduces CmIVTP, a cross-modal framework for maritime vessel trajectory prediction that integrates AIS and CCTV data to overcome limitations of single-source data. The core of CmIVTP is a cross-modal interaction transformer that uses attention mechanisms to capture both intra-modal semantics and inter-modal interactions between vessel motion and environmental features. Experiments on a new large-scale Maritime-MmD$^+$ dataset demonstrate that CmIVTP achieves improved performance on multimodal trajectory prediction benchmarks.
Fusing AIS and CCTV data with a cross-modal transformer unlocks more accurate maritime vessel trajectory prediction, even when one data source is sparse or unavailable.
Maritime intelligent transportation systems (MITS) are essential for ensuring navigation safety and efficiency in busy waterways. However, accurate vessel trajectory prediction remains challenging due to the limitations of single-source data. Automatic identification system (AIS) data is often sparse or unavailable for small vessels, while closed-circuit television (CCTV) data alone cannot fully capture dynamic vessel behavior. To mitigate these challenges, we propose a cross-modal interaction-based vessel trajectory prediction (named CmIVTP) framework to model the intricate interactions between vessel dynamics and environmental constraints. Specifically, we introduce a target-aware scene encoder to extract scene semantic features, effectively capturing vessel-environment interactions and enhancing trajectory prediction accuracy. In addition, we propose a cross-modal interaction transformer, which integrates AIS-derived motion features, CCTV-based environmental features, and scene representations. It leverages cross-modal attention mechanisms to simultaneously capture intra-modal semantics and inter-modal interactions, ensuring dynamically consistent and environmentally feasible predictions. Furthermore, we construct a vessel group trajectory bank by clustering historical AIS trajectories into representative motion patterns, providing an efficient and scalable approach for candidate trajectory generation. Additionally, we introduce the maritime multimodal dataset plus (named Maritime-MmD$^+$), a large-scale dataset that synchronizes AIS data and CCTV video data, providing robust support for multimodal trajectory prediction research. Extensive experiments demonstrate that CmIVTP achieves better performance on multimodal-driven vessel trajectory prediction benchmarks. The code resources for this work can be available at https://github.com/LouisYxLu/CmIVTP.
