Search papers, labs, and topics across Lattice.
This paper introduces UNSPECIFIC, a framework designed to synthesize general constraints from similar reference documents to combat the issue of copy-pasting in LLM instruction following. By selectively hardening trivially satisfied constraints, the framework balances the difficulty and naturalness of instructions, leading to a significant drop in superficial compliance among LLMs. The results indicate that the satisfaction rate of GPT-5 Mini decreases from 90% to 78%, highlighting the effectiveness of UNSPECIFIC in promoting deeper understanding and adherence to complex instructions.
LLMs exhibit a 12% drop in constraint satisfaction when faced with more challenging, synthesized instructions that prevent trivial copy-pasting.
Large language models (LLMs) are increasingly expected to follow long lists of constraints in complex instructions, and synthesizing instructions from a reference document (i.e., back-translation) is a widely used method to measure/enhance LLMs' ability to follow complex instructions. However, this method introduces a critical loophole: the constraint synthesis model copies text from the reference as a very specific constraint and the evaluated LLM trivially satisfies the constraint by copying its text in the response. To address these issues, we propose UNSPECIFIC, a novel framework that synthesizes constraints common to two similar reference articles to reduce copy-pasting, selectively hardens only trivially satisfied constraints to balance difficulty and naturalness, and evaluates satisfaction on both the generated article and its summary to penalize superficial instruction following. Consequently, we built the UNSPECIFIC benchmark on news, story, and blog domains to analyze the copy-pasting behavior of LLMs. Our results show that our synthesized constraints are not only more challenging (e.g., the satisfaction rate of GPT-5 Mini drops from 90% to 78%) and natural (LLM win-rate gap improves by 30%) from a human perspective but also mitigate the copy-pasting. We also find that a large portion of constraints are satisfied superficially (i.e., not satisfied in the core narrative of the article). The code and datasets are released at https://github.com/JeetDSharma/UNSPECIFIC.
