Search papers, labs, and topics across Lattice.
This study introduces Harness-IF, a novel evaluation framework for coding agents that distinguishes between genuine instruction compliance and coincidental task success. By employing Against-Prior Accuracy (AP-Acc), the authors assess the effectiveness of 256 operational rules across 60 multi-turn coding tasks, revealing that all tested models perform worse when adhering to rules that contradict their default behaviors. The findings indicate that existing compliance metrics may overestimate a model's instruction-following capabilities, with significant implications for the evaluation of coding agents in practical applications.
Coding agents may appear compliant, but they actually underperform when faced with rules that challenge their default behaviors, revealing a critical flaw in current evaluation metrics.
When a coding agent obeys a rule, it may simply have been going to do that anyway. Existing instruction-following benchmarks cannot tell the difference: they concentrate rules in the user turn, while coding-agent benchmarks emphasize final task success. We introduce Harness-IF, which scores operational rules one at a time from execution evidence: 60 realistic multi-turn coding items drawn from a 642-rule library, 256 rules receiving verdicts, placed on the five configurable surfaces a deployed agent reads. To separate compliance from coincidence we introduce Against-Prior Accuracy (AP-Acc), which scores only rules labeled as opposing unprompted defaults, observed by re-running tasks with the rule withheld across nine probe builds and curated otherwise. Across 12 frontier models, accuracy spans 72.1-85.9% and AP-Acc 66.1-78.6%; every model is worse on against-prior rules, by 3.6 to 7.4 points (mean 5.81), and the direction survives a common-support analysis with item-clustered intervals. Aggregate scores therefore overstate compliance by a model-specific margin: prior control leaves the top build unchanged and exchanges three adjacent rank pairs. A counterbalanced conflict pilot on nine separate builds adds a second result: pooled precedence does not follow prompt depth, with system prompts, project files, and user instructions ahead of tool and skill descriptions.
