Search papers, labs, and topics across Lattice.
This paper establishes a measurement framework to differentiate between action prediction, task uncertainty, and step attribution in the evaluation of coding agents. By employing SCAE, a replay-based estimator grounded in a structural causal model, the authors reveal that next actions are predominantly influenced by execution provenance rather than code-graph transitions. The findings indicate that existing process evaluations often conflate semantic relevance with causal contribution, leading to potential misinterpretations of agent performance.
Current evaluations of coding agents may mismeasure performance by conflating action, task, and step levels, revealing a critical flaw in how we assess agent execution.
Coding agents are increasingly evaluated not only by whether they solve a task, but also by how they execute it. However, existing process-level evaluations often treat action prediction, task uncertainty, and step attribution as if they were the same problem, which makes it unclear what such evaluations actually measure. In this paper, we introduce a measurement framework for process evaluation in coding agents and instantiate step-level causal attribution with SCAE, a replay-based estimator derived from a structural causal model of agent execution. Our framework combines prefix-conditioned identification, replay/intervention-based estimation, and controlled judge-information manipulation to study process evaluation at the action, task, and step levels. Experiments on 499 file-localization episodes from 12 repositories show that next actions are driven primarily by execution provenance rather than code-graph transitions, execution uncertainty is structured at the task rather than step level, and full-trace judges exhibit systematic collider bias, suggesting that current process evaluation often measures semantic relevance rather than certified causal contribution.
