Search papers, labs, and topics across Lattice.
This paper identifies instruction quality as a critical bottleneck in preference learning, where low-quality instructions limit the informativeness of response pairs. By employing Best- and Worst-of-N analyses, the authors demonstrate that refining instructions significantly enhances the quality of sampled responses and preference signals. Their instruction-refinement pipeline, which utilizes reward signals and rubric-guided feedback from large language models (LLMs), leads to substantial improvements in alignment across various models and benchmarks.
Instruction quality is the hidden bottleneck in preference learning, and refining it can dramatically enhance model alignment.
Preference learning optimizes models using response pairs, yet the informativeness of these pairs is fundamentally shaped by the instructions from which they are generated. We identify instruction quality as a hidden bottleneck in preference learning: low-quality or ambiguous instructions restrict the response-quality distribution, limiting strong chosen responses and weakening preference signals. Through Best- and Worst-of-N analyses, we show that instruction quality constrains both the ceiling and floor of sampled response quality. Motivated by this observation, we introduce an instruction-refinement pipeline that selects weak instructions using reward signals and revises them with rubric-guided LLM feedback, improving preference data without discarding examples. Across offline and online preference learning settings, experiments on multiple models and benchmarks show broad alignment improvements over original data and alternative data-improvement strategies. Further analyses indicate that instruction refinement raises achievable response quality and complements response-centric preference data curation. Overall, instruction quality emerges as a key factor governing how informative preference signals are formed for LLM alignment. Code is available at: https://github.com/01choco/instruction-refinement/
