Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the effectiveness of test-time collaboration methods in improving LLM reasoning by framing it as a candidate-selection problem rather than an inherent multi-agent property. The authors introduce a diagnostic framework that quantifies the oracle gap and signal fidelity, revealing that gains from collaboration are limited by these factors. Key findings include significant improvements in verification accuracy on LiveCodeBench and MATH, with a public-test verifier achieving an 8.14 percentage point gain over a baseline, while also highlighting the risks of harm to correct outputs from certain selectors.
Gains from test-time collaboration are constrained by the oracle gap and signal fidelity, with some methods causing harm to already-correct outputs.
Test-time collaboration, including self-consistency, best-of-N selection, critic models, and verifier pipelines, is often credited with broadly improving LLM reasoning, yet its gains are uneven and sometimes negative. We ask when training-free collaboration should be expected to help. For a fixed candidate pool, we decompose a selector or verifier's net gain into measurable factors: recoverable mass, verification-signal coverage, conditional selection quality, and harm to already-correct outputs. This reframes collaboration as a candidate-selection problem rather than as an intrinsic property of a multi-agent topology. Across LiveCodeBench, MATH Level-5 hard subjects, and GPQA-Diamond, gains are bounded first by the oracle gap and then by signal fidelity, which we measure directly as candidate-level agreement between verifier verdicts and official labels. On LiveCodeBench, a public-test verifier (MCC 0.825) gains +8.14 percentage points (pp) over a first-sample baseline; a generated-test verifier (MCC 0.248) improves by +2.70pp and is not statistically distinguishable from an LLM selector, but operates at near-zero harm versus the selector's 4.69% harm rate. On MATH, a symbolic answer-equivalence selector beats self-consistency by +4.67pp, while LLM selectors are negative. On GPQA-Diamond, recoverable mass is only 3.03% and 87.54% of candidate pools are answer-identical; a weaker model's pools shrink both further, suggesting that oracle gap is a joint property of task, model, and sampling configuration. Our framework yields a practical pre-deployment diagnostic: estimate the oracle gap, then measure coverage, signal fidelity, and harm before investing in collaboration.