Track the frontier, from condensed matter to cosmology.
Search papers, labs, and topics across Lattice.
This paper introduces Semantically UNified (SUN) Programs, which integrate geometric and contact relations into a unified framework for model-based control and learned policies in long-horizon manipulation tasks. The authors present Kuafu, a system that synthesizes these programs from language and scene semantics, enabling the retention of task semantics during policy training while achieving high success rates across multiple tasks. Notably, Kuafu outperforms existing methods, achieving an 82.03% macro-success rate and significantly improving simulation and physical robot performance without the need for manual dense rewards or demonstrations.
Kuafu's ability to synthesize robust policies from language and task semantics achieves over 82% success in complex manipulation tasks, revolutionizing how we approach control-to-learning paradigms.
Bridging model-based control and learned policies in long-horizon manipulation has harbored a silent disagreement: control executes specified objectives, learning amortizes that behavior into a reactive policy, yet existing protocols discard task semantics, leaving rewards hand-crafted and behavior drifting from what control verified.We introduce Semantically UNified (SUN) Programs, typed executables where geometric and contact relations are defined once and compiled into aligned Model Predictive Control (MPC) costs, satisfaction predicates, RL rewards, transition guards, and diagnostics. Our system, Kuafu, driven by large vision language systems, automatically synthesizes SUN Programs from language and scene semantics, screens feasibility via MPC, and retains semantics while training stage-conditioned policies. Across nine tasks, Kuafu achieves 82.03% macro-success, outperforming sparse-reward (35.67%) and Stage-BC (24.75%) baselines. At 8192-way scale, it generates 10.57x the successful trajectory time per hour of human teleoperation. With 500 trajectories per task, Kuafu data trains DP3 policies to 46.0% simulation success (vs. 22.4% for alternatives) and 34.7% on physical Franka and Kinova robots. These results establish that simulation-screened task semantics can effectively amortize control into robust policies, without demonstrations or manual dense rewards, unifying symbolic planning and data-driven execution.