Search papers, labs, and topics across Lattice.
This paper introduces the Thinking Checklist Reward (TCR), a novel process-oriented reward mechanism designed to enhance LLM preference alignment by evaluating reasoning trajectories rather than just final outcomes. By converting preference pairs into sample-specific thinking checklists, TCR provides more nuanced guidance for credit assignment during reinforcement learning. Experimental results across five models from three families demonstrate that TCR significantly improves alignment performance, highlighting the effectiveness of its exponential moving average (EMA) residual formulation in isolating additional thinking contributions beyond outcome predictions.
TCR transforms LLM preference alignment by focusing on the reasoning process, leading to substantial improvements in model performance across diverse benchmarks.
LLM preference alignment aims to optimize models toward human preferences across diverse user instructions. Reinforcement learning has become a major post-training approach for this goal, but existing proxy rewards are often outcome-level, mainly evaluating the final response while providing limited guidance for the reasoning trajectory. This can make credit assignment coarse when multiple responses receive similar final scores, leaving trajectory-level preferences under-specified. To address this limitation, we propose Thinking Checklist Reward (TCR), a process-oriented reward for RL-based preference alignment. TCR converts preference pairs into sample-specific thinking checklists and uses them to evaluate whether the generated reasoning trace addresses the preference-implied considerations. To reduce overlap with outcome-level supervision, TCR further introduces an exponential moving average (EMA) residual formulation to isolate a complementary thinking surplus beyond what is predictable from the outcome reward. Experiments on five models from three model families show that TCR consistently improves alignment performance across diverse benchmarks, with ablations further validating the importance of EMA-based residual formulation and sample-specific checklist supervision.
