Search papers, labs, and topics across Lattice.
This paper introduces VLLR, a dense reward framework for long-horizon robotic tasks that combines extrinsic rewards from LLMs and VLMs with an intrinsic reward based on policy self-certainty. VLLR uses LLMs to decompose tasks into verifiable subtasks and VLMs to estimate progress, initializing the value function for efficient warm-up. Experiments on the CHORES benchmark demonstrate that VLLR significantly outperforms both pretrained policies and state-of-the-art RL finetuning methods, achieving up to 56% absolute success rate gains without manual reward engineering.
Forget hand-crafted rewards: VLLR leverages LLMs and VLMs to automatically generate dense rewards, boosting robotic task success rates by up to 56% on long-horizon tasks.
Existing robotic foundation policies are trained primarily via large-scale imitation learning. While such models demonstrate strong capabilities, they often struggle with long-horizon tasks due to distribution shift and error accumulation. While reinforcement learning (RL) can finetune these models, it cannot work well across diverse tasks without manual reward engineering. We propose VLLR, a dense reward framework combining (1) an extrinsic reward from Large Language Models (LLMs) and Vision-Language Models (VLMs) for task progress recognition, and (2) an intrinsic reward based on policy self-certainty. VLLR uses LLMs to decompose tasks into verifiable subtasks and then VLMs to estimate progress to initialize the value function for a brief warm-up phase, avoiding prohibitive inference cost during full training; and self-certainty provides per-step intrinsic guidance throughout PPO finetuning. Ablation studies reveal complementary benefits: VLM-based value initialization primarily improves task completion efficiency, while self-certainty primarily enhances success rates, particularly on out-of-distribution tasks. On the CHORES benchmark covering mobile manipulation and navigation, VLLR achieves up to 56% absolute success rate gains over the pretrained policy, up to 5% gains over state-of-the-art RL finetuning methods on in-distribution tasks, and up to $10\%$ gains on out-of-distribution tasks, all without manual reward engineering. Additional visualizations can be found in https://silongyong.github.io/vllr_project_page/
