Search papers, labs, and topics across Lattice.
This paper investigates the challenges faced by coding agents in maintaining coherence across repository-scale tasks, specifically focusing on the concept of coherence debt, which arises when necessary facts are unavailable during code edits. By modeling the task as a coupled-fact graph and experimenting with seven models and five harnesses, the authors demonstrate that the absence of critical information leads to systematic failures across all models, particularly when faced with unseen APIs or changes in library names. The findings reveal that the cost of missing facts directly correlates with the complexity of the task, emphasizing the need for agents to have access to relevant information during code modifications to ensure successful outcomes.
Missing critical facts during coding tasks leads to systematic failures across multiple models, highlighting the importance of coherence in repository-scale coding agents.
Repository-scale coding requires an agent to keep tests, imports, configuration, and migration rules consistent within a bounded context window. We model this as reconstructing a coupled-fact graph: at each edit, a required fact comes from recent context or parametric memory, and the facts covered by neither form coherence debt. We supply and withhold each channel and inject faults across seven models and five harnesses. As expected, no model completes a task on an unseen API with both channels empty, and putting the facts in the prompt restores success. When a rename defeats what models memorized about a real library, all seven fail in the same place, passing and missing the same tests. Availability decides the outcome and distance does not: withholding a fact costs exactly the work it supports, and a supplied fact works as well far from the edit as next to it. Harnesses pay unequal prices for it: configurations that all pass every test differ more than tenfold in tokens consumed because they rebuild the same content at different rates, and spending more recovers nothing when facts are withheld. A missing fact produces wrong work rather than absent work: an agent asked to act acts, fabricating the file or guessing the value, so instruments built on reads look for a hole already filled. How often it says it is blocked instead is a property of the model, from every trial to none. Availability does not settle every edit: where standard and code disagree, agents follow the standard even when it prescribes the worse code, so a stale convention file costs more than no file. Because parametric memory substitutes for reading, on SWE-bench, where models likely know the repositories, reads no longer predict success. Harnesses should keep the facts an edit depends on available when the agent writes, and check that availability against what the agent produces rather than what it reads.
