Search papers, labs, and topics across Lattice.
The paper introduces ACE (Agentic Control for Embodied Manipulation), a zero-shot workflow reasoning framework designed for tabletop pick-and-place tasks that integrates natural language understanding with dynamic environment adaptation. By employing a mask-mediated vision-action interface and a multi-timescale memory system, ACE enables agents to verify and adjust their actions in real-time, thereby effectively handling execution failures and user corrections. The framework achieves significant task-level zero-shot generalization, with a 50% success rate in complex equation formation and a 70% success rate in constraint-based object retrieval, outperforming standard end-to-end baselines.
ACE achieves a remarkable 70% success rate in constraint retrieval tasks without any task-specific retraining, showcasing the power of zero-shot workflow reasoning in robotic manipulation.
Open-ended tabletop manipulation requires agents to not only understand natural language but also adapt to dynamic environments and execution failures. We present ACE (Agentic Control for Embodied Manipulation), a zero-shot workflow reasoning framework for tabletop pick-and-place from natural language. Rather than relying on direct low-level action mapping, ACE combines agentic workflow reasoning with two robot-facing executable skills: a visual grounding interface and a reusable pick-and-place primitive. To bridge semantic reasoning and physical control, the active sub-goal is grounded into a mask-mediated vision-action interface. This unified mask specifies the target object and destination, is tracked over time, exposed for human verification, and ultimately passed to a task-agnostic downstream policy for execution. Crucially, ACE operates in a closed loop supported by a multi-timescale memory. After an action is executed, the system automatically verifies whether the intended sub-goal succeeded, using the outcome to advance, retry, repair, or replan. This enables online adaptation to user corrections, scene changes, and physical failures. We evaluate ACE on logically complex, long-horizon tasks, including zero-shot multi-step equation formation with number cubes and constraint-based object retrieval. ACE demonstrates task-level zero-shot generalization on novel semantic constraints and randomized tabletop scenes without task-specific retraining. Specifically, while standard end-to-end baselines struggle to complete these logically demanding tasks, ACE achieves a 50% success rate in equation formation and a 70% success rate in constraint retrieval. This contrast demonstrates that explicit workflow reasoning and mask-mediated control offer a robust, practical route toward adaptable robotic manipulation.
Tsinghua AI