Search papers, labs, and topics across Lattice.
This paper introduces SpecPath, a diagnostic evaluation framework that assesses coding agents' ability to handle specification-path sensitivity, where equivalent requirement histories can lead to different programming outcomes. The study reveals that while aggregate accuracy remains stable across various coding tasks, a significant portion of successful implementations fails when subjected to alternative specification paths. This highlights a critical gap in understanding how coding agents interpret evolving requirements, emphasizing the need for rigorous evaluation of their robustness to specification history changes.
Coding agents can achieve high accuracy on final specifications, yet 35% of them falter when faced with equivalent requirement histories, exposing a hidden vulnerability in their design.
Modern coding agents increasingly appear capable of following complex software requirements, yet their success leaves a critical ambiguity: do they resolve the active specification, or merely follow the most salient path by which it was stated? We identify specification-path sensitivity, a failure mode in which requirement histories that are equivalent in their final meaning lead the same agent system to produce behaviorally different programs. This reframes evolving-requirement evaluation as active-contract resolution: before writing code, an agent must determine which requirements still count. Building on this view, we introduce SpecPath, a diagnostic evaluation that holds the repository, final contract, verifier, agent system, and execution budget fixed while changing only the revision path that leads to the contract. Rather than treating each patch as an isolated pass or failure, SpecPath uses paired executable outcomes to reveal whether an agent realizes the same tested behavior across contract-equivalent histories. Across five calibrated software tasks and fourteen coding-agent configurations, aggregate direct and revision-history accuracy is nearly unchanged; nevertheless, 35 of 100 complete blocks that succeed on the direct specification fail on at least one equivalent history. These results show that implementation success on a consolidated request does not guarantee specification-path invariance. Evaluating evolving requirements therefore calls for controlled tests of whether agents are robust to the path by which a specification becomes final.
