Search papers, labs, and topics across Lattice.
This study evaluates AI scientists' capabilities in generating novel ideas through benchmarking in complex, adversarial real-world domains, specifically Formula 1 and Magic: The Gathering. By comparing AI-generated outputs against expert solutions, the research reveals that while models can produce plausible ideas, they struggle with aligning these ideas with real-world innovations. Notably, the findings indicate that the primary challenge lies not in generating ideas but in effectively filtering and prioritizing them to achieve coherent novelty.
AI scientists excel at idea generation but falter in filtering and prioritizing innovations, revealing a critical gap in their capabilities.
Benchmarking the ability of AI scientists to generate novel ideas is notoriously difficult. Existing benchmarks in this field have made progress in evaluating scientific reasoning and research replication, but often rely on synthetic tasks or retrospective targets, which may be confounded by prior exposure. We hypothesize that complex, adversarial, fast-moving real-world domains where expert practitioners independently generate observable outputs can provide a practical solution to fill this gap and evaluate the capabilities needed for AI scientists, including reasoning, novelty, and hypothesis formulation. We instantiate this framework in two structurally different domains, Formula 1 (F1), where models ideate around car design concepts for the 2026 season, and real pre-season innovations provide a ground truth, and Magic: The Gathering (MTG), where models propose decks from a recently updated card pool and are evaluated against 19 Pro Tour (PT) decklists. Across both domains, models produce plausible outputs, but few align with real-world expert solutions. In F1, the best model, GPT-5.2 matched 10 of 40 real innovations with 166 ideas proposed across runs. In MTG, the best deck from Gemini 3 Flash recovered 5 of 7 new-set cards from the third-place PT deck, and across all 108 decks, the cards models selected most often were also the cards most widely adopted by PT decks (Spearman $\rho = 0.74$, $p = 0.0003$). These results suggest that a key capability gap for AI scientists is not idea generation, but filtering, prioritization, and coherent novelty.
