Search papers, labs, and topics across Lattice.
This paper introduces DenseReward, a novel dense reward model for robotic manipulation that synthesizes diverse failure data in simulation without human intervention. By generating realistic failure trajectories and predicting frame-level reward scores from visual and language inputs, DenseReward provides fine-grained feedback that significantly enhances policy optimization. Experimental results demonstrate that DenseReward surpasses existing vision-language models and robotic reward systems in both simulated and real-world tasks, offering effective guidance for reinforcement learning applications.
DenseReward synthesizes diverse failure trajectories automatically, enabling robots to learn from a rich array of failure modes without human labeling.
Reinforcement learning holds great promise for improving robot policies beyond the limits of imitation learning. However, its practical adoption remains bottlenecked by the lack of reliable vision-language reward models that provide dense and informative feedback. Two key challenges remain: acquiring diverse failure data at scale and obtaining fine-grained reward signals beyond sparse trajectory-level success labels. Collecting failure trajectories typically requires laborious human effort, while pseudo-failures constructed by relabeling successful demonstrations fail to capture the diverse physical failure modes that arise during robot execution. Meanwhile, existing reward models often predict sparse binary or trajectory-level rewards, which provide limited guidance for efficient policy optimization. We introduce DenseReward, a dense robotic reward model that addresses both challenges. To train DenseReward, we develop an automated failure data generation pipeline that synthesizes physically realistic failure trajectories in simulation without human labeling, covering diverse failure modes such as collisions, missed grasps, object drops, and recovery behaviors. DenseReward predicts dense frame-level reward scores from visual observations and language instructions, enabling fine-grained estimation of task progress throughout an episode. Experiments show that DenseReward outperforms general-purpose VLMs and existing robotic reward models in dense reward prediction across both simulated and real-world manipulation. We further demonstrate that DenseReward provides effective reward guidance for downstream model predictive control and reinforcement learning. We release the dataset, trained reward models, and evaluation suite to support the development of failure-aware dense reward modeling for robot learning.
