Track manipulation, locomotion and embodied systems.
Search papers, labs, and topics across Lattice.
RoboAware enhances embodied coding agents by learning a state-conditioned responsibility coordinator that leverages counterfactual outcomes to select optimal modular robot skills. This approach addresses the challenge of anticipating policy family success in dynamic physical states by introducing State-Locked Counterfactual Branching (SCB) to generate unobserved outcomes and Execution-Aware Learning (EAL) to distill these into family-conditioned values. RoboAware achieves state-of-the-art performance across diverse benchmarks, including RoboSuite, LIBERO-Pro, and RoboTwin, demonstrating significant improvements in task success rates.
Counterfactual outcomes reveal that existing embodied agents miss 20% of potential success by not exploring alternative skill branches.
Embodied coding agents can combine modular robot skills with frozen end-to-end policies, yet effective composition requires anticipating which policy family will succeed in the current physical state. We present RoboAware, which builds on coding agents' skill orchestration by learning only a state-conditioned responsibility coordinator from counterfactual outcomes. Inspired by the success of REPL, we propose the $P^5$ schema and formulate a hierarchical MDP based on it. $P^5$ organizes skills uniformly into five semantic stages, defining where responsibility can be compared. To address the lack of counterfactual branch outcomes in existing work, we introduce State-Locked Counterfactual Branching (SCB), which restores the same training state to generate and execute a code block from each admissible family, exposing outcomes that selected-branch experience leaves unobserved. Building on this, we propose Execution-Aware Learning (EAL), which combines Monte Carlo tree search with Q-learning to distill these outcomes into family-conditioned values. At deployment, the coordinator selects the policy family according to observable context, and the frozen coding agent generates the next local code block. Comprehensive single-episode evaluations on 100 tasks show that RoboAware reaches a 77.0% overall success rate, with SOTA averages of 90.0% on RoboSuite, 73.8% on diverse LIBERO-Pro task clusters, and 90.0% on challenging RoboTwin bimanual tasks, outperforming existing code-as-policy and VLA-harness baselines.