Search papers, labs, and topics across Lattice.
The paper introduces CoEnv, a compositional environment framework that integrates real-world and simulation components to facilitate multi-agent embodied collaboration. CoEnv uses real-to-sim scene reconstruction, VLM-driven action synthesis for planning, and validated sim-to-real transfer with collision detection. Experiments on multi-arm manipulation benchmarks show CoEnv achieves high task success rates and execution efficiency, demonstrating a new approach for multi-agent embodied AI.
Coordinating embodied multi-agent systems doesn't require end-to-end training; instead, offload planning to a VLM in simulation and transfer back to the real world for execution.
Multi-agent embodied systems hold promise for complex collaborative manipulation, yet face critical challenges in spatial coordination, temporal reasoning, and shared workspace awareness. Inspired by human collaboration where cognitive planning occurs separately from physical execution, we introduce the concept of compositional environment -- a synergistic integration of real-world and simulation components that enables multiple robotic agents to perceive intentions and operate within a unified decision-making space. Building on this concept, we present CoEnv, a framework that leverages simulation for safe strategy exploration while ensuring reliable real-world deployment. CoEnv operates through three stages: real-to-sim scene reconstruction that digitizes physical workspaces, VLM-driven action synthesis supporting both real-time planning with high-level interfaces and iterative planning with code-based trajectory generation, and validated sim-to-real transfer with collision detection for safe deployment. Extensive experiments on challenging multi-arm manipulation benchmarks demonstrate CoEnv's effectiveness in achieving high task success rates and execution efficiency, establishing a new paradigm for multi-agent embodied AI.
