Search papers, labs, and topics across Lattice.
This paper introduces OPERA, a novel approach to aligning large language models (LLMs) for open-ended reasoning tasks by utilizing intrinsic rewards derived from perplexity dynamics instead of relying on biased external judges. By synthesizing data through guiding words and employing perplexity-prioritized rollouts, OPERA generates a diverse dataset of 20,000 reasoning trajectories that enhances the stability and quality of supervision. The method achieves state-of-the-art performance on open-ended tasks with the Qwen3-8B model, matching or exceeding the capabilities of proprietary models like Gemini2.5 and MiniMax-M2.5.
OPERA's intrinsic reward mechanism enables LLMs to achieve new heights in open-ended reasoning, outperforming proprietary models without the pitfalls of biased supervision.
Reinforcement Learning (RL) has enabled LLMs to excel in objective reasoning tasks such as mathematics and code generation. However, applying RL to open-ended tasks, such as creative writing, remains challenging because LLM-as-a-judge reward models often exhibit stylistic biases and positional inconsistencies, leading to unstable supervision. To address this, we propose OPERA (Objective Perplexity-based Reflective Alignment), which replaces unreliable external judges with intrinsic rewards derived from perplexity dynamics. Specifically, we derive an intrinsic reward signal from perplexity dynamics, quantifying uncertainty reduction at critical reflective states. During the cold-start phase, we introduce a data synthesis method that leverages carefully designed guiding words to generate diverse reasoning traces, along with perplexity-prioritized rollouts that utilize internal log-probabilities to identify logically consistent reasoning branches. This pipeline yields a large-scale dataset comprising 20,000 high-quality reasoning trajectories. Empirical evaluations consistently demonstrate the scalability and efficacy of our approach in alignment for open-ended tasks. Implementing OPERA on Qwen3-8B establishes a new state-of-the-art among open-source models, achieving parity with or surpassing proprietary models like Gemini2.5 and MiniMax-M2.5 in some open-ended tasks. The code is available at https://github.com/pangpang-xuan/OPERA.
