Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces WorldReasoner, an evaluation framework designed to assess the reasoning capabilities of language model agents in forecasting real-world events under uncertainty. The framework evaluates agents not only on final-answer accuracy but also on the quality of evidence cited and the reasoning process employed, utilizing a benchmark of 345 tasks derived from over 14,000 articles. Key findings reveal that while temporally valid evidence retrieval significantly enhances outcome accuracy, agents still face challenges in converting grounded evidence into well-calibrated probability forecasts.
Language model agents may appear accurate in event forecasting, but they often rely on memorized facts or flawed reasoning rather than genuine understanding.
Forecasting real-world events requires language-model agents to reason under uncertainty from incomplete, time-bounded information. Yet evaluating whether agents genuinely forecast requires more than final-answer accuracy: a model may be correct by recalling memorized training facts, citing fabricated evidence, or producing an unsupported causal story. We present WorldReasoner, an evaluation framework for temporally valid event forecasting. Each task gives an agent a resolved forecasting question, a simulated forecast date, and access only to evidence available before that date; after resolution, the framework scores the submitted probability, cited evidence, and optional causal event graph. WorldReasoner reports three complementary axes: outcome quality against resolved answers, evidence quality over cited sources, and reasoning quality against post-resolution hindsight graphs. The benchmark is built by an agentic construction pipeline that generates forecasting questions, collects time-stamped evidence, and builds hindsight reference graphs at scale, yielding 345 resolved tasks derived from 14,141 articles with graphs covering 8,087 extracted events. Across six controlled agent settings, temporally valid retrieval is the strongest driver of outcome accuracy; causal graph construction improves key-event recovery; and correct graph-enabled forecasts are more strongly grounded in key events and relevant sources, yet agents still struggle to convert grounded evidence into calibrated probabilities.