Search papers, labs, and topics across Lattice.
This paper introduces USS, an end-to-end framework for Embodied Visual Tracking (EVT) that integrates unified spatial-semantic prompts, allowing agents to utilize text, point, bounding box, and mask inputs for more effective target tracking in dynamic environments. By employing modality-specific encoders and hybrid attention mechanisms, USS enhances the agent's ability to ground targets accurately, particularly in cluttered scenes with multiple similar objects. Real-robot experiments show that spatial cues significantly outperform text-only prompts in challenging scenarios, while simulation results indicate state-of-the-art performance compared to existing methods, highlighting the framework's efficiency and robustness.
Spatial-semantic prompting outperforms traditional text-only methods in embodied visual tracking, especially in complex environments with similar distractors.
Embodied Visual Tracking (EVT) requires an agent to continuously follow a specified target while actively moving through dynamic environments. However, prevailing EVT paradigms predominantly rely on language-based target indication. While language is expressive and convenient, cluttered scenes often contain multiple objects that satisfy the same semantic description, leading to ambiguous target grounding. We therefore propose a paradigm shift, reframing target indication in EVT from text-only specification to unified spatial-semantic prompting. Based on this paradigm, we introduce Unified Spatial-Semantic Prompts for Embodied Visual Tracking with Latent Dynamics Learning, USS, an end-to-end embodied tracking framework that supports text, point, bounding box, and mask prompts within a unified architecture. USS encodes heterogeneous prompts with modality-specific encoders, fuses prompt tokens with visual features through hybrid attention, and decodes compact prompt-conditioned representations into egocentric waypoints. To further improve temporal robustness, USS incorporates a latent world model that predicts future representations through self-supervised alignment. Real-robot experiments demonstrate that explicit spatial target cues yield higher success rates than text-only prompts, particularly in scenarios involving similar distractors and longer-horizon tracking where maintaining instance-level target identity is critical. In the simulation benchmark, USS also achieves state-of-the-art performance among non-MLLM-based methods and competitive results against recent MLLM-based approaches with faster inference speed. Our findings reveal that spatial-semantic prompting provides a more precise and flexible target indication interface for embodied visual tracking. Project site: https://arescheah.github.io/uss-project-page/.
