Search papers, labs, and topics across Lattice.
This paper introduces PTA-IRT, a novel framework that integrates process-level data from historical execution trajectories with traditional outcome-based evaluation methods to enhance the benchmarking of software engineering agents. By leveraging detailed information about how agents navigate tasks, such as explored context and attempted edits, PTA-IRT improves the selection of calibration subsets and the estimation of agent abilities. The results demonstrate that PTA-IRT significantly outperforms existing IRT baselines in both score and ranking recovery across multiple benchmarks, even under low calibration budgets.
PTA-IRT reveals that incorporating detailed problem-solving trajectories can dramatically enhance the evaluation accuracy of software engineering agents.
Evaluating software engineering agents on realistic benchmarks is costly, since each task may require multi-step code exploration, modification, and test execution. Existing efficient evaluation methods select representative subsets to estimate full-benchmark performance, but are largely result-only: they fit historical pass/fail response matrices or static task semantics, discarding how agents solve problems. We propose PTA-IRT, a Privileged Trajectory-Aware Item Response Theory framework that fuses process and outcome signals. Historical execution trajectories supply process-level evidence beyond pass/fail, such as explored context, attempted edits, and solving paths, which PTA-IRT uses as privileged information for calibration subset selection and ability estimation. Under low calibration budgets, PTA-IRT consistently outperforms prior IRT baselines on score and ranking recovery across four SWE benchmarks. Code and data are publicly available at https://github.com/DeepSoftwareAnalytics/PTA-IRT.
