Search papers, labs, and topics across Lattice.
CMU MLAffiliation:CornellSt. Mildred's-Lightbourn SchoolUniversity of MonastirAug 17, 2026arXiv:2608.16978This paper introduces VLCP, a novel approach that utilizes a frozen vision-language model (VLM) to generate control policies for robot manipulation by writing Python functions without fine-tuning or demonstrations. By implementing a closed-loop system where the VLM rewrites the control code every K steps based on real-time observations, the method significantly improves the robot's ability to recover from failures during execution. Evaluated across 57 tasks, VLCP achieves a pooled success rate of 35.1%, demonstrating a tenfold improvement over traditional open-loop methods that fail to adaptively correct errors within an episode.
Closing the loop on robot manipulation failures, VLCP rewrites control code in real-time, achieving a tenfold increase in success rates over traditional methods.
Turning a frontier vision-language model into a robot policy usually means fine-tuning it to emit an action representation it never saw in pretraining, which throws away much of the reasoning that made the model worth reaching for. We go the other way and keep the VLM frozen. It writes the policy as a short Python control function, with no demonstrations and no fine-tuning. Writing that code once is open-loop, though. Existing closed-loop methods react at the wrong level: they retry a fixed policy or pick a different subtask, but never rewrite the code that failed. VLCP closes the loop where the failure actually lives, on the control code, within a single episode. Every $K$ steps the VLM re-observes the scene from multi-view RGB, proprioceptive state, and a state delta, then rewrites the control function from what it just saw, so a failure is caught before it compounds. We evaluate on a 57-task MuJoCo/RoboVerse sweep. This training-free policy reaches $35.1\%$ pooled success, against $3.5\%$ for the identical system queried once per episode. That tenfold gap holds with non-overlapping confidence intervals in every scene family. The gain traces to a $27.3\%$ within-episode recovery rate on failed grasps: a miss an open-loop controller would carry to the end of the episode gets re-observed and fixed at the next replan. And the loop stays cheap. A median $84\%$ of input tokens hit cache, an episode needs only about $10$ compact queries, and control blocks written during any replan persist to a cross-episode skill library reused in later prompts.
