Track synthesis, materials and catalysis as it is published.
Search papers, labs, and topics across Lattice.
This paper introduces WEval, a fine-grained evaluation pipeline for writing reward models, and WRL, a reinforcement learning training framework that selectively drops instruction requirements to create positive and negative samples. WEval enables systematic evaluation by measuring the correlation between reward model rankings and gold rankings across diverse writing tasks and requirements. Experiments demonstrate that WRL significantly improves performance on writing benchmarks and exhibits strong generalization compared to existing coarse-grained approaches.
Fine-grained reward modeling, achieved by selectively dropping instruction requirements, unlocks substantial improvements in writing-centric generation tasks.
Large language models have achieved remarkable progress in text generation but still struggle with generative writing tasks. In terms of evaluation, existing benchmarks evaluate writing reward models coarsely and fail to measure performance from the perspective of specific requirements. In terms of training, existing training methods either use LLM-as-a-judge approaches or train coarse-grained reward models, lacking fine-grained requirement-adherence reward modeling. To address these issues, we propose a fine-grained evaluation pipeline WEval for writing reward models and a fine-grained reinforcement learning training framework WRL. The evaluation data of WEval covers multiple task categories and requirement types, enabling systematic evaluation of writing reward models by measuring the correlation between the rankings of the reward model and gold rankings. WRL constructs positive and negative samples by selectively dropping instruction requirements, allowing for more precise reward model training. Experiments show that our models achieve substantial improvements across various writing benchmarks and exhibit strong generalization. The code and data are publicly available at \href{https://github.com/Rainier-rq1/From_Coarse_to_Fine}{https://github.com/Rainier-rq1/From\_Coarse\_to\_Fine}.