Search papers, labs, and topics across Lattice.
This study investigates the construction processes of Software Engineering (SE) agents through semi-structured interviews and an online survey, revealing how development practices are evolving as these agents become more prevalent. It identifies a seven-stage workflow and highlights a shift towards evaluation-driven development, where the evaluation of agent outputs is crucial for iteration. Key challenges include unreliable evaluation signals and comprehension debt, which arise as code complexity increases and model updates alter behavior.
As SE agents become mainstream, the shift to evaluation-driven development reveals new challenges that could redefine engineering workflows.
The rise of Software Engineering (SE) agents, i.e., LLM-based agents that can understand large codebases and carry out engineering tasks with limited human intervention, has been marked by rapid advances and adoption, but little is known about how developers build these systems in practice: existing studies mine repositories or examine deployment, but few investigate how SE agents are constructed. Through semi-structured interviews with 20 practitioners from 12 organizations and an online survey of 80 practitioners, this paper is the first to study how SE processes are changing in the development of SE agents and what challenges developers face. We find that as implementation becomes cheaper, bottlenecks shift rather than disappear: long-standing non-coding work such as requirements, coordination, review, and deployment becomes more visible, while reviewing and evaluating agent output becomes new and central. We characterize a seven-stage workflow and a shift toward evaluation-driven development, in which evaluation steers iteration and specifications become versioned artifacts read by both humans and agents. We further identify six challenges that teams face, together with the practices they adopt to address them, including unreliable evaluation signals, comprehension debt as code outpaces understanding, and behavioral changes introduced by provider-side model updates.
