Search papers, labs, and topics across Lattice.
The paper introduces Memory-Enhanced Dynamic reward Shaping (MEDS), a novel reinforcement learning framework that addresses the issue of reduced sampling diversity in LLMs by incorporating historical behavioral signals into reward design. MEDS uses density-based clustering of intermediate model representations from past rollouts to identify and penalize frequently recurring error patterns. Experiments across five datasets and three base models demonstrate that MEDS consistently improves average performance and behavioral diversity compared to existing baselines, achieving gains of up to 4.13 pass@1 points and 4.37 pass@128 points.
LLMs can learn to avoid repeating mistakes by remembering and penalizing frequently recurring error patterns in past rollouts.
Despite the success of reinforcement learning for large language models, a common failure mode is reduced sampling diversity, where the policy repeatedly generates similar erroneous behaviors. Classical entropy regularization encourages randomness under the current policy, but does not explicitly discourage recurrent failure patterns across rollouts. We propose MEDS, a Memory-Enhanced Dynamic reward Shaping framework that incorporates historical behavioral signals into reward design. By storing and leveraging intermediate model representations, we capture features of past rollouts and use density-based clustering to identify frequently recurring error patterns. Rollouts assigned to more prevalent error clusters are penalized more heavily, encouraging broader exploration while reducing repeated mistakes. Across five datasets and three base models, MEDS consistently improves average performance over existing baselines, achieving gains of up to 4.13 pass@1 points and 4.37 pass@128 points. Additional analyses using both LLM-based annotations and quantitative diversity metrics show that MEDS increases behavioral diversity during sampling.
Tsinghua AI