Search papers, labs, and topics across Lattice.
The paper introduces Omnidirectional Referring Multi-Object Tracking (ORMOT), a new task extending Referring Multi-Object Tracking (RMOT) to omnidirectional imagery to address the limited field-of-view of existing datasets. To facilitate research in this area, the authors create ORSet, a dataset comprising 27 omnidirectional scenes with 848 language descriptions and 3,401 annotated objects. They also propose ORTrack, a Large Vision-Language Model (LVLM)-driven framework, demonstrating its effectiveness on the ORSet dataset.
Omnidirectional imagery + language unlocks robust multi-object tracking that overcomes the field-of-view limitations plaguing conventional video datasets.
Multi-Object Tracking (MOT) is a fundamental task in computer vision, aiming to track targets across video frames. Existing MOT methods perform well in general visual scenes, but face significant challenges and limitations when extended to visual-language settings. To bridge this gap, the task of Referring Multi-Object Tracking (RMOT) has recently been proposed, which aims to track objects that correspond to language descriptions. However, current RMOT methods are primarily developed on datasets captured by conventional cameras, which suffer from limited field of view. This constraint often causes targets to move out of the frame, leading to fragmented tracking and loss of contextual information. In this work, we propose a novel task, called Omnidirectional Referring Multi-Object Tracking (ORMOT), which extends RMOT to omnidirectional imagery, aiming to overcome the field-of-view (FoV) limitation of conventional datasets and improve the model's ability to understand long-horizon language descriptions. To advance the ORMOT task, we construct ORSet, an Omnidirectional Referring Multi-Object Tracking dataset, which contains 27 diverse omnidirectional scenes, 848 language descriptions, and 3,401 annotated objects, providing rich visual, temporal, and language information. Furthermore, we propose ORTrack, a Large Vision-Language Model (LVLM)-driven framework tailored for Omnidirectional Referring Multi-Object Tracking. Extensive experiments on the ORSet dataset demonstrate the effectiveness of our ORTrack framework. The dataset and code will be open-sourced at https://github.com/chen-si-jia/ORMOT.
