Search papers, labs, and topics across Lattice.
This paper performs a large-scale empirical evaluation of natural language instruction files used to guide AI coding agents, using 679 files from GitHub and SWE-bench Verified. Surprisingly, random rules improve performance as much as expert-curated ones, suggesting a context priming effect. The study also finds that negative constraints are the only individually beneficial rule type, while positive directives often hurt performance, highlighting a risk in relying on seemingly helpful guidance.
Expert-written rules for coding agents are often useless or even harmful, with random constraints working just as well and negative constraints outperforming positive directives.
Developers increasingly guide AI coding agents through natural language instruction files (e.g., CLAUDE.md, .cursorrules), yet no controlled study has measured whether these rules actually improve agent performance or which properties make a rule beneficial. We scrape 679 such files (25,532 rules) from GitHub and conduct the first large-scale empirical evaluation, running over 5,000 agent runs with a state-of-the-art coding agent on SWE-bench Verified. Rules improve performance by 7--14 percentage points, but random rules help as much as expert-curated ones -- suggesting rules work through context priming rather than specific instruction. Negative constraints ("do not refactor unrelated code") are the only individually beneficial rule type, while positive directives ("follow code style") actively hurt -- a pattern we analyze through the lens of potential-based reward shaping (PBRS). Moreover, individual rules are mostly harmful in isolation yet collectively helpful, with no degradation up to 50 rules. These findings expose a hidden reliability risk -- well-intentioned rules routinely degrade agent performance -- and provide a clear principle for safe agent configuration: constrain what agents must not do, rather than prescribing what they should.
