Search papers, labs, and topics across Lattice.
This paper introduces Active Panoramic Referring Segmentation (APRS), a novel task that enables agents to actively perceive and segment objects in 360掳 environments based on user instructions. The authors present PanoSeeker, a memory-augmented agent that combines a Vision-Language Model with an explicit spatial visual memory called EgoSphere, allowing for efficient exploration and segmentation. Experimental results on a new APRS benchmark show that PanoSeeker significantly outperforms existing state-of-the-art models in both search efficiency and segmentation accuracy.
PanoSeeker achieves unprecedented search efficiency and segmentation accuracy in dynamic 360掳 environments, redefining how agents can actively perceive and interact with their surroundings.
Existing referring segmentation models passively process static images captured from fixed perspectives, limiting their applicability in Embodied AI, where agents must perform active perception in the continuous 360$^\circ$ environments. To bridge this gap, we introduce a novel task: Active Panoramic Referring Segmentation (APRS). In this setting, an agent is required to adjust its viewing direction ($\Delta\theta, \Delta\phi$) to explore the 360$^\circ$ environment, seeking the object specified by a user instruction for segmentation. To tackle this challenging task, we propose PanoSeeker, a memory-augmented agent for efficient APRS. Rather than relying on heuristic scanning, PanoSeeker integrates a Vision-Language Model (VLM) with EgoSphere, an explicit spatial visual memory. By progressively integrating sequential local observations into a unified 360$^\circ$ representation, EgoSphere enables the agent to plan efficient and non-redundant search trajectories. Once the target is found, the agent performs active viewpoint alignment and outputs the segmentation mask. Furthermore, we curate an expert-annotated search trajectory dataset with memory timelines for Supervised Fine-Tuning, followed by Reinforcement Learning post-training to explicitly optimize PanoSeeker's exploration efficiency. Extensive experiments on our newly established APRS benchmark demonstrate that PanoSeeker achieves superior search efficiency and segmentation accuracy, significantly outperforming adapted state-of-the-art baselines.
