Search papers, labs, and topics across Lattice.
This paper introduces Self-Review Reinforcement Learning (SRRL), a novel framework that integrates a self-review mechanism within each reinforcement learning episode to enhance learning from sparse feedback. By generating self-reviews after failed attempts, the model can identify errors and improve its responses through policy distillation, ensuring that these improvements are retained in future episodes. Evaluated against a standard RLVR baseline, SRRL demonstrates superior performance and learning efficiency on the GSM8K benchmark with two language models, Qwen 3-4B and OLMo-3-7B.
Self-Review Reinforcement Learning transforms failure into a learning opportunity, enabling models to internalize improvements and significantly boost performance on complex tasks.
Reinforcement Learning is commonly used to train large language models using environmental feedback. In applied settings, the environment usually provides sparse or delayed feedback. This makes it difficult for the model to pinpoint which actions in its reasoning led to success or failure. So, learning effectively from these signals is hard because the model must determine how each failure should inform meaningful behavioral corrections in subsequent iterations. We introduce a training framework, Self-Review Reinforcement Learning, that embeds an explicit self-review step into each RL episode. When a first-pass response fails, the model generates a self-review to identify what went wrong, which conditions an improved second attempt. Unlike inference-time reflection approaches, such as Reflexion, the framework optimizes self-review with policy gradients and internalizes improvements into the base policy via selective distillation, ensuring they persist across future episodes. A cross-episode memory keeps successful self-reviews for reuse when encountering similar tasks in future episodes during training. We evaluate SRRL against a standard RLVR baseline using the GRPO optimizer across two language models, Qwen 3-4B and OLMo-3- 7B, on GSM8K benchmark. SRRL consistently outperforms the RLVR in final reward performance and achieves greater learning efficiency by successfully transforming feedback into behavioral improvement.
