Search papers, labs, and topics across Lattice.
This paper introduces WEval, a fine-grained evaluation pipeline for writing reward models, and WRL, a reinforcement learning training framework that selectively drops instruction requirements to create targeted training samples. WEval enables systematic evaluation by correlating reward model rankings with gold rankings across diverse writing tasks and requirements. Experiments demonstrate that WRL significantly improves performance on writing benchmarks and generalizes well, addressing the limitations of coarse-grained evaluation and training methods.
Current LLMs struggle with writing tasks because existing reward models are too coarse, but a new fine-grained reward modeling approach closes the gap.
Large language models have achieved remarkable progress in text generation but still struggle with generative writing tasks. In terms of evaluation, existing benchmarks evaluate writing reward models coarsely and fail to measure performance from the perspective of specific requirements. In terms of training, existing training methods either use LLM-as-a-judge approaches or train coarse-grained reward models, lacking fine-grained requirement-adherence reward modeling. To address these issues, we propose a fine-grained evaluation pipeline WEval for writing reward models and a fine-grained reinforcement learning training framework WRL. The evaluation data of WEval covers multiple task categories and requirement types, enabling systematic evaluation of writing reward models by measuring the correlation between the rankings of the reward model and gold rankings. WRL constructs positive and negative samples by selectively dropping instruction requirements, allowing for more precise reward model training. Experiments show that our models achieve substantial improvements across various writing benchmarks and exhibit strong generalization. The code and data are publicly available at \href{https://github.com/Rainier-rq1/From_Coarse_to_Fine}{https://github.com/Rainier-rq1/From\_Coarse\_to\_Fine}.
