Search papers, labs, and topics across Lattice.
This paper introduces BulkPR-Bench, a novel benchmark designed to evaluate the governance of interacting pull requests (PRs) in software development by requiring agents to jointly decide on the merging order of changes. The benchmark consists of 581 candidate PRs from 18 real repositories and employs metrics like Relational Delivery Score (RDS) and Global Safety-Gated Yield (Global-SGY) to assess the effectiveness of various models in delivering safe and correct merges. Results indicate that while the best models achieve an RDS of 66.6%, only a small fraction of runs successfully complete the entire queue, highlighting significant challenges in achieving reliable governance in PR interactions.
Achieving a Relational Delivery Score of 66.6% reveals that even advanced models struggle with the complexities of merging interacting pull requests safely.
Coding-agent benchmarks increasingly cover long-horizon, end-to-end, and interactive development, but typically retain one requested outcome or a fixed change sequence. Sequential policies can process a pull-request (PR) queue one candidate at a time, but when queued PRs interact, maximizing safe delivery can require jointly deciding which changes to merge and in what order. We introduce BulkPR-Bench, an executable benchmark in which an agent must recover consequential PR relations and return a large safe subset in executable order under a rolling-release protocol. The suite contains 581 newly authored candidate PRs on frozen snapshots of 18 real repositories. Registered state-by-state repository execution, including hidden safety checks, validates the gold relation graph; an exact oracle then computes the largest safe subset. Our primary metric, Relational Delivery Score (RDS), scores safe delivery and correct rejection over relation groups from the realized merge trace; Global Safety-Gated Yield (Global-SGY) separately measures strict delivery of the realized whole-queue plan. Under the buffered primary protocol with batch size $K=32$, the three highest RDS estimates among the six models are 66.6%, 62.0%, and 57.9%, compared with 53.1% for the strongest sequential baseline. Only 8 of 324 model runs complete a queue exactly. Critical-relation recall ranges from 35.2% to 57.7%, and diagnostic runs supplied with the gold relations show substantial remaining headroom. Gains on relation groups therefore do not yet translate into dependable whole-queue governance.
