Search papers, labs, and topics across Lattice.
Phoenix is a multi-agent LLM system designed to automate the resolution of GitHub issues, utilizing a coordinated approach with six specialized agents and seven layered safety controls. The system achieved a 75% oracle resolution rate on a benchmark slice without regressions, while a pilot study on real issues demonstrated 100% correctness preservation, highlighting the effectiveness of its safety mechanisms. However, challenges remain, such as the localization limitations of the planner, which resulted in some pull requests targeting incorrect code paths.
Phoenix resolves GitHub issues with 75% accuracy while ensuring safety through a multi-agent system, but still faces challenges in planner localization.
We present Phoenix, a multi-agent LLM system that resolves GitHub issues from triage through pull-request creation, combining seven layered safety controls with a baseline-aware test evaluation strategy. Phoenix decomposes the work across six specialized agents. Planner, reproducer, coder, tester, failure analyst and Pull Request (PR) agent, all coordinated by a label-based GitHub webhook state machine. Every change is checked against a baseline test run before a pull request is opened. On a 24-instance slice of SWE-bench Lite. run on the production webhook path, Phoenix oracle-resolves 75% of instances with no pass-to-pass regressions on successful runs; this curated slice is not directly comparable to full-split leaderboard results, and we discuss the limits of the comparison. A complementary pilot on 42 real issues across 14 repositories yields 100% correctness preservation (CP; mean 122s on the hard tier). Manual inspection shows that about half of the resulting pull requests are well-targeted fixes. The other half place code at incorrect paths, a planner localization limitation we are addressing with retrieval. We also report the deployment failure modes (WAF filtering, token expiry, permission boundaries, flaky CI) that motivated each safety mechanism.
CMU ML