Search papers, labs, and topics across Lattice.
This paper introduces a method for learning reusable, natural-language rubrics from inline comments on text drafts, addressing the challenge of capturing tacit, context-dependent criteria for LLM writing and review support. The approach iteratively refines rubrics by minimizing mismatches between rubric-conditioned comment predictions and actual comments. Experiments in real-world and controlled settings demonstrate the effectiveness of distilling inline comments into rubrics that facilitate comment prediction, rubric comprehension, and automated text revision.
Unlock expert-level writing feedback: this method distills tacit knowledge from inline comments into reusable rubrics, enabling more effective LLM-driven writing support.
Large language models (LLMs) are increasingly used for writing and review support, but their usefulness depends on context-dependent criteria, such as expert preferences or organization-specific conventions, that are often tacit, undocumented, and difficult to elicit directly. We propose a problem setting for learning reusable natural-language rubrics from accumulated inline comments on artifacts such as human-written or LLM-generated drafts. Our method infers rubrics from these comments and iteratively refines them by observing comment-wise mismatches between rubric-conditioned predictions and reference comments. We evaluate the proposed method in real-world review settings and in controlled settings with reference rubrics. These results show that inline comments can be distilled into reusable rubrics that support comment prediction, rubric understanding, and automatic artifact revision.
