Search papers, labs, and topics across Lattice.
This paper evaluates the impact of varying levels of AI autonomy on software development by having a team of developers reimplement a full-stack web application through three phases: partial AI assistance, full AI assistance with GitHub Copilot, and full AI assistance with AWS Kiro. The study found that increased AI autonomy led to reduced development effort, improved requirement adherence, and lower cognitive workload, although developer frustration did increase slightly. Notably, the AWS Kiro phase outperformed the other phases across most metrics, indicating that the architecture of AI tools can significantly influence development outcomes beyond just the level of autonomy provided.
Higher AI autonomy in software development can cut effort and enhance adherence to requirements, but it may also raise developer frustration levels.
This paper presents an exploratory evaluation of how increasing levels of AI autonomy affect software development productivity, requirement adherence, and developer cognitive workload. A team of four developers reimplemented the same full-stack web application across three sequential phases: partial AI-assisted development using GitHub Copilot, an AI-exclusive workflow using GitHub Copilot, and an AI-exclusive workflow using AWS Kiro. Evaluation metrics included development effort (hours), requirement adherence (RITM score), AI-interaction efficiency, and NASA-TLX workload measures. Across phases, higher levels of AI autonomy were associated with reduced development effort, improved requirement adherence, and lower self-reported mental workload, while developer frustration increased modestly. The AWS Kiro phase achieved the strongest overall performance on most measured dimensions, suggesting that tooling architecture may influence outcomes independently of AI autonomy level.
