Search papers, labs, and topics across Lattice.
This paper analyzes the reliability of AI bots (Claude, Devin, Cursor, Copilot, and Codex) operating within GitHub Actions CI/CD workflows using a dataset of 61,837 workflow runs triggered by AI-generated pull requests. The study reveals significant agent-dependent differences in workflow success rates, with Copilot and Codex showing the highest reliability (~93-94%), and a negative correlation between AI agent contribution frequency and workflow success at the repository level. A taxonomy of failure categories further highlights a shift from functional to non-functional PR failures over time, suggesting evolving challenges in AI agent integration.
AI bots like Copilot and Codex can reliably trigger CI/CD workflows, but flooding repos with their PRs may actually *decrease* overall workflow success.
Continuous Integration and Deployment (CI/CD) workflows are central to modern software delivery, yet the reliability of agentic AI bots operating within these workflows remain underexplored. Using pull requests (PRs), commits, and repositories from the AIDev dataset, we retrieved associated CI/CD workflow runs via the GitHub Actions API and analyzed 61,837 runs from 2,355 repositories, all triggered by PRs generated by five AI bots: Claude, Devin, Cursor, Copilot, and Codex. We observed substantial agent-dependent differences in workflow reliability, with Copilot and Codex achieving the highest success rates ~93% and ~94% respectively. At the repository level, we find a negative correlation between AI agent contribution frequency and workflow success rate, suggesting that a higher frequency of Agentic PRs may hinder CI/CD workflow reliability. We defined a taxonomy of 13 categories against 3,067 agentic PRs whose associated workflows failed, and observed a trend analysis that indicates visually observable shifts from functional to non-functional PR categories over time, although these trends are not statistically significant. Our findings motivate the need for actionable guidance on integrating AI agents into CI/CD workflows and prioritizing safeguards in workflows where failures are most likely to occur.
