Search papers, labs, and topics across Lattice.
To evaluate whether LLMs can accurately anticipate scientific trends, the authors construct Research Attention Prediction (RAP), a rolling benchmark across 278 AI/ML fields where agents search temporally restricted arXiv corpora to forecast six-month publication shares across competing research topics. Strikingly, all four diagnostic models underperform a simple exponential moving average (EWMA) baseline, driven by target-conditioned retrieval biases where forecasting-oriented prompts systematically under-sample recent literature relative to state-tracking prompts. Only frontier models with dynamic search marginally surpass EWMA, though targeted fine-tuning on realized historical outcomes improves a 4B model's forecast Spearman correlation by 0.105 on held-out fields and trend-shifting episodes.
LLM research agents consistently lose to a naive moving-average baseline at forecasting scientific trends鈥攏ot due to poor reasoning, but because asking models to predict the future systematically biases their search policies away from the most recent literature.
Large language models (LLMs) increasingly act as research agents, yet their ability to track shifts in research attention is difficult to evaluate because reviews and research ideas lack uniquely verifiable outcomes. We introduce Research Attention Prediction (RAP), a rolling benchmark covering 278 AI/ML fields and 1,390 episodes. At each cut-off, an LLM agent searches a temporally restricted arXiv corpus and predicts the next six months' paper shares across eight frozen research directions. Search generally helps, but all four diagnostic models perform worse than an exact-count exponentially weighted moving average (EWMA) baseline in compositional accuracy. We identify two linked bottlenecks. Under cumulative-history access, State carry-forward outperforms direct Forecast for all four diagnostic models; frozen-evidence replay links a shared component of this reversal to Forecast-oriented policies retrieving a smaller share of recent evidence. Even with exact historical activity, future-specific updating remains limited, with only GPT-5.5 plus reopened Search slightly surpassing EWMA. Fine-tuning on realised outcomes improves Qwen3-4B's forecast Spearman correlation by 0.105 on held-out fields at later origins, with gains also on change-rich episodes.
