Search papers, labs, and topics across Lattice.
This paper analyzes real-world software engineering sessions from JetBrains IDEs, revealing that interactive coding agent benchmarks often fail to capture the diverse and dynamic task flows characteristic of actual user interactions, which involve a broad mix of task types and frequent switching. To address this, the authors introduce SWE-TaskFlow, a method to adapt existing issue-derived benchmarks by steering them towards specific target task flows using prompt splitting and repository QA, and propose a TaskFlow Alignment Score (TFAS) for trajectory selection. Evaluating SWE-TaskFlow on SWE-Bench Pro tasks showed that adopting a multi-step interaction protocol, mimicking realistic task flows, significantly increases agent cost without a commensurate improvement in task resolution rates.
Real-world coding agent usage reveals that task flow, not just task completion, dramatically impacts agent cost and effectiveness, suggesting current benchmarks are misaligned with user behavior.
The evaluation of coding agents generally strives to be as realistic as possible. In our study, we collect 4,782 agent sessions of real software engineers in JetBrains IDEs, which we call Production Sessions. Since our subject is interactive agents, we study the sessions with at least three user messages (33% of the sample). These long sessions differ from issue-derived benchmark tasks in two ways: (i) user requests span a far wider mix of task types - questions about the project's code, planning, review, refactoring, execution - and (ii) users switch between types throughout a session. Long-session samples from three public interaction corpora exhibit markedly different Task Flows (the distributions of session lengths, task types, and type-to-type transitions), so no single interaction distribution is universally realistic: benchmarks should name a target use case and calibrate to measurements from it. We present SWE-TaskFlow, an approach for transforming any issue-derived benchmark: it preserves the verified tasks and tests while steering the interaction toward a target Task Flow through prompt splitting and verifiable repository QA, with a TaskFlow Alignment Score (TFAS) for selecting among generated trajectories. In a pilot on 700 SWE-Bench Pro tasks, solving the task sequentially in several steps approximately doubles agent cost without a stable change in resolve rate: the interaction protocol itself is an important dimension of evaluation.
