Search papers, labs, and topics across Lattice.
This paper introduces a novel approach to language-guided aerial perception that addresses the challenges of identifying tiny targets in streaming UAV videos. By creating the DroneEyes dataset, the authors provide a comprehensive resource for training models on pixel-level and open-vocabulary referring-segmentation tasks, which is crucial for understanding small objects in complex environments. The proposed SkyAnchor model employs a Semantics-Aware Token Router and a Hierarchical Memory Bank to effectively manage visual data and maintain contextual understanding across sequential frames, significantly improving performance in real-time applications.
Tiny aerial targets can now be accurately identified in streaming videos, thanks to a new dataset and a model that preserves critical visual details while managing memory constraints.
Language-guided aerial perception aims to understand user-specified tiny targets in complex unmanned aerial vehicle (UAV) scenes. In real UAV deployment, the UAV must respond while it flies, so such perception runs in an online streaming manner, where frames arrive sequentially and the model responds to each one without access to future frames. However, applying current Multimodal Large Language Models (MLLMs) to this setting raises two challenges. First, targets viewed from the air are often tiny, yet the visual compression in existing MLLMs treats all regions equally and discards their fine-grained details. Second, understanding a continuous stream requires past-frame context, yet retaining the entire history is infeasible on resource-constrained onboard hardware, whereas discarding it causes the target to drift or disappear. We address the tiny object and streaming challenges from both data and method perspectives. From the data perspective, we present \textbf{DroneEyes}, the \textbf{first} pixel-level and open-vocabulary referring-segmentation dataset for tiny aerial targets, comprising $2,140$ high-definition videos and $176,623$ pairs across Object Description and Referring Expression tasks, with dense per-frame masks. From the method perspective, we propose \textbf{SkyAnchor}, an MLLM with two designs to the above challenges: a Semantics-Aware Token Router that preserves small-target under a reduced visual-token budget, and a Hierarchical Memory Bank that keeps the target consistently understood on streams.
