Search papers, labs, and topics across Lattice.
This paper introduces a method for improving creative writing in LLMs by addressing the challenges of reward modeling and automatic evaluation in the absence of verifiable reference answers. They design a multi-agent collaborative workflow based on Grounded Theory to dynamically produce interpretable fine-grained criteria for evaluation. The core of their approach is Memory-augmented Replay Policy Optimization (MRPO), which combines self-reflection based on dynamic criteria with a supervised fine-tuning and reinforcement learning paradigm to convert evaluation criteria into reward signals for end-to-end optimization, leading to a Writer-R1 model that outperforms strong baselines.
LLMs can now write better stories, poems, and scripts, thanks to a new training method that uses AI to automatically generate its own writing feedback criteria.
As a typical open-ended generation task, creative writing lacks verifiable reference answers, which has long constrained reward modeling and automatic evaluation due to high human annotation costs, evaluative bias, and coarse feedback signals. To address these challenges, this paper first designs a multi-agent collaborative workflow based on Grounded Theory, performing dimensional decomposition and hierarchical induction of the problem to dynamically produce interpretable and reusable fine-grained criteria. Furthermore, we propose the Memory-augmented Replay Policy Optimization (MRPO) algorithm: on the one hand, without additional training, MRPO guides models to engage in self-reflection based on dynamic criteria, enabling controlled iterative improvement; on the other hand, we adopt the training paradigm that combines supervised fine-tuning with reinforcement learning to convert evaluation criteria into reward signals, achieving end-to-end optimization. Experimental results demonstrate that the automatically constructed criteria achieve performance gains comparable to human annotations. Writer-R1-4B models trained with this approach outperform baselines across multiple creative writing tasks and surpass some 100B+ parameter open-source models.
