Search papers, labs, and topics across Lattice.
This paper introduces SWE-Together, a novel multi-turn benchmark designed to evaluate coding agents in interactive user sessions by reconstructing real user-agent interactions from recorded coding sessions. By curating 109 repository-level tasks from over 11,000 sessions, the authors create a reactive LLM-based user simulator that accurately reflects user intents and feedback. The findings reveal that more advanced coding agents not only achieve higher final correctness but also require fewer corrective interventions, indicating a significant enhancement in user experience during coding tasks.
Stronger coding agents can achieve higher success rates while requiring fewer user interventions, reshaping our understanding of effective coding assistance.
Most coding-agent benchmarks are static: an agent receives a complete task description up front and is judged only by its final code. Real coding assistance is interactive, with users clarifying goals, adding constraints, and correcting mistakes over multiple turns. We introduce SWE-Together, a multi-turn benchmark reconstructed from real user-agent coding sessions. To make real interactions verifiable, we curate 109 repository-level tasks from 11,260 recorded sessions, selecting sessions with recoverable repository states, clear user goals, and observable outcomes. To replay these interactions across agents, we build a reactive LLM-based user simulator that preserves the original users'intents and provides feedback when the coding agent's progress requires it. To evaluate agents as collaborators, we measure both final repository correctness and the number of corrective feedback turns required during the interaction. Experiments with frontier coding agents show that stronger agents generally achieve higher final success rates while requiring fewer interventions, suggesting an improved user experience.
Meta AI