Search papers, labs, and topics across Lattice.
This study introduces IdeaForecastBench, a benchmark designed to evaluate the ability of large language models (LLMs) to predict future research ideas based on existing literature. By employing a fixed retrieve-then-judge protocol across 624 episodes and comparing various history-compression strategies in models like GPT-4.1 and Qwen, the authors demonstrate that GPT-4.1-mini outperforms others in Hit@5 and Precision@5 metrics. Notably, while Qwen2.5 excels in producing broader forecasts, the research highlights the complexities in interpreting the relationship between breadth and predictive accuracy.
GPT-4.1 can predict future research ideas with higher accuracy than its competitors, revealing the nuanced interplay between model architecture and forecasting ability.
Large language models increasingly generate research ideas, yet judging their novelty or feasibility at generation time does not establish whether they anticipate subsequent work. We introduce IdeaForecastBench to evaluate research idea forecasting. Given a community's literature up to a cutoff, a system produces up to five ranked ideas, which are evaluated against later papers. The benchmark comprises 624 rolling episodes across 52 topics, with a fixed retrieve-then-judge protocol and separately reported results from two judges. We compare five history-compression strategies across GPT-4.1, Qwen2.5-7B/14B, and Qwen3.5-9B, together with a learned Mode-Decomposition Forecaster (MDF). Under the primary GPT-4.1-mini judge, Summary improves on Direct in Hit@5 and Precision@5 across all four backbones. Qwen2.5 scores above GPT-4.1, whereas Qwen3.5 scores below it. An outcome-blind assessment finds that Qwen2.5 produces broader forecasts, but does not identify how much breadth contributes to its advantage. Threshold and judge diagnostics further clarify the limits of interpreting realization as precise anticipation. IdeaForecastBench provides a common task for studying which research ideas a community subsequently pursues and how reliably this outcome can be measured.
