Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
ROSClaw is introduced as a novel agent framework that unifies policy learning and task execution for heterogeneous robots within a vision-language model (VLM) controller. It uses e-URDF representations for sim-to-real topological mapping, providing real-time access to physical states and enabling a data collection mechanism for iterative policy optimization. The framework demonstrates improved robustness in multi-policy execution by maintaining semantic continuity and dynamically assigning task-specific control, reducing reliance on robot-specific workflows.
Forget robot-specific pipelines: ROSClaw's unified vision-language framework lets you control heterogeneous robots with a single agent, enabling rapid skill transfer and continuous improvement.
The integration of large language models (LLMs) with embodied agents has improved high-level reasoning capabilities; however, a critical gap remains between semantic understanding and physical execution. While vision-language-action (VLA) and vision-language-navigation (VLN) systems enable robots to perform manipulation and navigation tasks from natural language instructions, they still struggle with long-horizon sequential and temporally structured tasks. Existing frameworks typically adopt modular pipelines for data collection, skill training, and policy deployment, resulting in high costs in experimental validation and policy optimization. To address these limitations, we propose ROSClaw, an agent framework for heterogeneous robots that integrates policy learning and task execution within a unified vision-language model (VLM) controller. The framework leverages e-URDF representations of heterogeneous robots as physical constraints to construct a sim-to-real topological mapping, enabling real-time access to the physical states of both simulated and real-world agents. We further incorporate a data collection and state accumulation mechanism that stores robot states, multimodal observations, and execution trajectories during real-world execution, enabling subsequent iterative policy optimization. During deployment, a unified agent maintains semantic continuity between reasoning and execution, and dynamically assigns task-specific control to different agents, thereby improving robustness in multi-policy execution. By establishing an autonomous closed-loop framework, ROSClaw minimizes the reliance on robot-specific development workflows. The framework supports hardware-level validation, automated generation of SDK-level control programs, and tool-based execution, enabling rapid cross-platform transfer and continual improvement of robotic skills. Ours project page: https://www.rosclaw.io/.