Search papers, labs, and topics across Lattice.
This paper introduces REAL, an innovative framework for open-world mobile manipulation that addresses the challenges of real-world deployment by integrating active exploration, visual grounding, and interactive intent disambiguation. By establishing sim-to-real-consistent environment APIs and employing a simulated user for human-in-the-loop interaction, the framework enables systematic optimization of agent performance through diverse task compositions and a hierarchical training pipeline. Experimental results show that the trained agent achieves a 78.3% success rate in real-world trials, significantly outperforming leading commercial models on interactive tasks, demonstrating effective zero-shot transferability to novel household scenarios.
Achieving a 78.3% success rate in real-world mobile manipulation, this framework bridges the reality gap with zero-shot transferability to unseen tasks.
Real-world deployment of embodied agents requires active exploration, visual grounding, and interactive intent disambiguation. However, existing frameworks often rely on privileged simulator states or assume complete instructions, bypassing realistic deployment challenges. To bridge this gap, we present REAL, an agentic framework for open-world mobile manipulation. REAL establishes sim-to-real-consistent environment APIs without oracle perception and integrates a simulated user to enable human-in-the-loop interaction. Within this environment, we design diverse task compositions to drive data collection, supervised fine-tuning, and online reinforcement learning, systematically optimizing agent performance. To comprehensively evaluate this approach, we introduce REAL-Bench, a benchmark spanning 241 tasks across active exploration, visual distraction, articulated manipulation, and interactive disambiguation. Experimental results demonstrate that our trained agent outperforms leading commercial closed-source VLMs on interactive tasks with a 56.9% success rate. Further empirical analysis reveals that our hierarchical training pipeline successfully aligns the model's tool-use capabilities while maintaining robust open-vocabulary reasoning under extended exploration horizons. Finally, we deploy and evaluate our framework on a physical dual-arm mobile robot, where it achieves a 78.3% end-to-end success rate over 60 real-world episodes. These physical trials demonstrate robust zero-shot transferability to unseen household scenarios, validating that our sim-to-real-consistent design successfully bridges the reality gap for long-horizon mobile manipulation. Code is available at https://github.com/InternRobotics/REAL.
