Search papers, labs, and topics across Lattice.
This paper introduces Reconstruction, a benchmark designed to assess the ability of language models to recover the core research idea of a published paper using only its pre-publication bibliography. By implementing a strict anti-leakage protocol and evaluating seven advanced models across six scientific domains, the authors found that these models achieved modest Match rates of approximately 3-15%. However, a novel multi-agent pipeline that incorporates cross-model review and tournament selection significantly improved performance, yielding Match rates of 23-42%, demonstrating a 2.4x increase over the best single-model approach.
Language models struggle to recover research ideas from bibliographies, achieving only 3-15% accuracy, but a multi-agent approach boosts this to 23-42%.
Can a language model recover the true research idea of a published paper when given only that paper's pre-publication bibliography? We introduce Reconstruction, a blind idea-recovery benchmark that withholds the seed paper and all contemporaneous or future literature, and asks models to propose hypotheses that an independent large language model judge matches against the held-out ground-truth idea. A strict anti-leakage protocol-temporal citation cutoff, anonymous reference IDs, and frozen per-paper bibliographies, which prevents prompt-time leakage of the seed idea. Across six scientific domains and 643 evaluated papers, seven frontier models achieve only modest Match rates (approx. 3-15%). We then evaluate a reference-only multi-agent (top 4) pipeline that combines cross-model review with a Swiss tournament over aligned hypothesis slots, without external web search. Cross-model review plus tournament selection raises Match rates to approx. 23-42% across all six domains, which is an observed approx. 2.4x lift over the best single-model baseline. This draft reports the protocol, anti-leakage design, and current results as an arXiv timestamp.
