Search papers, labs, and topics across Lattice.
This study investigates the predictability of acceptance and review effort for pull requests (PRs) submitted by both human developers and AI agents, using a leakage-aware prediction pipeline on the AIDev dataset. The authors find that acceptance predictions can be made effectively at the time of submission, with tree-based models achieving F1 scores exceeding 0.95, highlighting the importance of textual clarity and metadata as key predictors. Conversely, predicting review effort proves more challenging, as submission-time features only modestly account for factors like comment counts and time-to-merge, indicating that external workflow dynamics significantly influence these outcomes.
Early signals from pull requests can predict acceptance with over 95% accuracy, but estimating review effort remains elusive due to complex external factors.
Pull requests (PRs) are a central mechanism for reviewing and integrating code changes in modern software repositories. As AI coding agents begin to submit more code changes alongside human developers, maintainers face a new challenge: deciding which PRs are likely to be accepted and which ones may require substantial review effort. This paper studies whether such outcomes can be estimated at the time a PR is opened, before reviewer discussion, CI feedback, or merge decisions are available. Using the AIDev dataset, we construct a leakage-aware prediction pipeline for human- and agent-authored PRs. The feature set is limited to submission-time information, including PR text characteristics, metadata, repository context, temporal signals, and lightweight diff statistics. We evaluate classical machine-learning models, including Logistic Regression, Random Forests, Gradient Boosting, Extra Trees, and MLPs, across pooled, human-only, agent-only, and balanced contributor views. Our results show that acceptance prediction is feasible from early signals: tree-based models achieve F1 scores above 0.95, with textual clarity and metadata among the most influential predictors. Review-effort prediction is more difficult. Comment counts and time-to-merge are only modestly explained by submission-time features, suggesting that reviewer availability, project workflow, and team-specific review practices play a major role. These findings indicate that early PR models can support triage and reviewer prioritization, but should be used as advisory tools rather than automated decision-makers.
