Search papers, labs, and topics across Lattice.
This paper introduces Robo-Dopamine 2.0, a novel history- and out-of-distribution (OOD)-aware process reward model designed to enhance robotic manipulation by addressing the limitations of existing visual reward models. By employing history-conditioned pairwise rewards and an OOD-aware signed progress space, the model effectively distinguishes between valid progress and failures, leading to improved visual order consistency and robustness in execution. The results demonstrate a significant increase in mean visual order consistency and successful real-world insertions, showcasing the model's effectiveness in refining reinforcement learning for robotic tasks.
Achieving a mean visual order consistency of 0.9872, Robo-Dopamine 2.0 significantly outperforms traditional reward models, demonstrating its superior ability to handle OOD scenarios in robotic manipulation.
Vision-language-action (VLA) models improve robotic manipulation but remain vulnerable to compounding errors, scene changes, and off-trajectory states. Reinforcement learning can refine pretrained VLA policies, yet sparse success signals hinder exploration, while engineered dense rewards are costly and task-specific. Existing learned visual reward models often rely on static before-after observations, causing temporal ambiguity and weak discrimination between robustness-preserving variations and task-invalid failures under out-of-distribution (OOD) execution. We introduce Robo-Dopamine 2.0, a history- and OOD-aware process reward model with a pairwise prediction interface. It combines (1) history-conditioned pairwise rewards that use source-aligned reference panels for synthetic OOD queries and observed rollout history for online queries, while preserving the queried endpoints, and (2) an OOD-aware signed progress space that represents valid progress, robustness, failure, and recovery. A Signed-Hop Curriculum with transition-aware replay learns coarse execution ordering before fine-grained progress calibration. We also construct an OOD trajectory dataset and a five-family benchmark. Reference panels improve mean visual order consistency (VOC) from 0.967 to 0.986 and OOD-robust VOC from 0.906 to 0.958. With the same 400K pairwise-reward budget, Signed-Hop training with 25% replay reaches 0.9872 mean VOC, compared with 0.9858 for a matched-pool shuffled control. In downstream reinforcement learning, the full model achieves 86.8% mean RoboTwin success and 71/80 successful real-world insertions.
