Search papers, labs, and topics across Lattice.
This paper introduces CoRe, a novel hybrid framework that combines Formal Rewards (FR) and Residual Rewards (RR) using vision-language model (VLM) feedback to enhance preference-aligned reinforcement learning. By leveraging VLMs, the Formal Reward Module (FRM) iteratively optimizes FR based on task knowledge and preference feedback, while the Residual Reward Module (RRM) captures nuanced preferences from video-level observations. Experimental results reveal that CoRe significantly outperforms existing reward design methods in both simulation and real-world robotic manipulation tasks, demonstrating its effectiveness and efficiency in policy learning.
CoRe achieves preference-aligned reinforcement learning by seamlessly integrating human-like reward decomposition, outperforming traditional methods in both simulated and real-world environments.
Reward design remains a central challenge in reinforcement learning (RL). Hand-crafted rewards are often difficult to specify and may lead to suboptimal policies, while learned rewards from preferences can suffer from inefficiency and unstable training. Inspired by the dual nature of human learning explored in cognitive science, we decompose rewards into two complementary components: Formal Rewards (FR), explicitly designed based on task knowledge, and Residual Rewards (RR), learned from observations to capture implicit and nuanced preferences. Based on this decomposition, we propose CoRe, a hybrid framework that integrates FR and RR with vision-language models (VLMs) feedback to achieve preference-aligned policies without human involvement. Our contributions are twofold: (1) We propose a Formal Reward Module (FRM) that leverages VLMs to iteratively design and optimize FR based on task knowledge and preference feedback, enabling the continual improvement of policy during training; (2) We introduce a Residual Reward Module (RRM) that learns RR from video-level preference by employing VLMs to generate preference labels and capturing nuanced rewards that complement FR, ensuring alignment with human intent. Through the synergy of FRM and RRM, CoRe enables the automatic construction of reliable rewards that are efficient and preference-aligned. Extensive experiments demonstrate that CoRe outperforms existing approaches in terms of policy learning effectiveness and efficiency on ten robotic manipulation tasks in simulation and five real-worlds. Videos can be found on our project website: https://core-2026.github.io/
