Search papers, labs, and topics across Lattice.
This study evaluates six frontier large language models (LLMs) during the 2026 FIFA World Cup by requiring them to make live predictions on match outcomes and other metrics before each kickoff, ensuring a leakage-free assessment. The models achieved an average accuracy of 63.9% on match outcomes, comparable to betting favorites, but displayed a tendency to agree with each other more often than to provide correct predictions. Notably, the models struggled with close match predictions, indicating that their performance is more influenced by fixture lopsidedness than by the richness of available information.
LLMs may excel at predicting outcomes in sports, but they often converge on incorrect answers, revealing a critical flaw in their forecasting abilities.
Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewhere on the Web, and the evaluation must defend itself against memorisation. We report the opposite design. Over the 39 days of the 2026 FIFA World Cup, six frontier LLMs -- all with extended thinking and native server-side web search -- were asked before every kickoff, one match at a time, to fill in a seven-market prediction card for all 104 matches, plus 12 group winners and a pre-tournament outright pool; no answer existed when the question was asked, so the evaluation is leakage-free by construction rather than by filtering, and the frozen archive holds 4,494 scored predictions. What the tournament establishes is a set of behaviours the six systems share. On match outcome they average 63.9%, level with backing the bookmaker's favourite -- which is in fact what they usually do. They agree with one another far more often than they are right, so a majority vote adds nothing. They under-commit to draws and to goals, and crowd their scoreline picks onto a single prototypical result. Accuracy tracks how lopsided a fixture is rather than how much is known about it: it collapses in the closest ties, where the dossiers are richest, while questions about the tournament as a whole are answered well. On this task the current generation of frontier systems is not sharply differentiated: the standings hold up at the top and the bottom across the run and churn in the middle, and the margins stay narrow throughout. The briefing dossiers, fixtures and official results are released as a benchmark, together with the scoring code.
