Search papers, labs, and topics across Lattice.
This paper introduces Experiential Learning (EL), a novel approach that transforms an LLM's feedback mechanism from a rubric-based evaluator to a coaching model, allowing for the distillation of rich, contextual feedback into transferable knowledge. By leveraging this higher-bandwidth feedback channel, EL provides dense supervision and maintains fine-grained preferences among high-quality responses, significantly outperforming traditional scalar reward methods in reinforcement learning. The results demonstrate that EL not only enhances performance on held-out and unseen tasks but also improves generalization and reduces the risk of reward hacking, highlighting the potential of experiential knowledge in non-verifiable task training.
Experiential Learning outperforms traditional reinforcement learning by providing richer feedback, leading to better generalization and reduced reward hacking in LLM training.
Reinforcement learning (RL) on open-ended tasks compresses an LLM's rubric-based evaluation into a scalar reward, discarding rich textual feedback and conflating responses with distinct quality profiles. We propose Experiential Learning (EL), which repurposes the feedback model from an LLM-as-a-Judge into an LLM-as-a-Coach. The coach distills its assessment of each on-policy response into transferable experiential knowledge, which conditions a teacher model and is internalized by the policy through on-policy context distillation. Compared with scalar rewards, this higher-bandwidth feedback channel provides dense supervision and preserves fine-grained preferences among high-quality responses. Across two policy families, with feedback from the policy itself or a proprietary model, EL consistently outperforms rubric-based RL on held-out and unseen open-ended tasks. Notably, EL generalizes better beyond the training distribution, and mitigates reward hacking. These findings establish experiential knowledge as a richer and more generalizable learning signal for post-training on non-verifiable tasks.
Tsinghua AI