Search papers, labs, and topics across Lattice.
The paper introduces a new benchmark, Watson & Holmes, based on a detective tabletop game, to evaluate AI reasoning in naturalistic contexts using narrative evidence and open-ended questions. They developed an automated grading system validated against human assessors for scalable evaluation. Results show AI model performance improved significantly over nine months in 2025, approaching top human performance, with improvements attributed to both model releases and reasoning-oriented architectures.
LLMs are closing in on human-level reasoning in detective games, but still stumble on longer cases, revealing crucial differences in how they process complex narratives.
Existing benchmarks for AI reasoning provide limited insight into how closely these capabilities resemble human reasoning in naturalistic contexts. We present an adaptation of the Watson & Holmes detective tabletop game as a new benchmark designed to evaluate reasoning performance using incrementally presented narrative evidence, open-ended questions and unconstrained language responses. An automated grading system was developed and validated against human assessors to enable scalable and replicable performance evaluation. Results show a clear improvement in AI model performance over time. Over nine months of 2025, model performance rose from the lower quartile of the human comparison group to approximately the top 5%. Around half of this improvement reflects steady advancement across successive model releases, while the remainder corresponds to a marked step change associated with reasoning-oriented model architectures. Systematic differences in the performance of AI models compared to humans, dependent on features of the specific detection puzzle, were mostly absent with the exception of a fall in performance for models when solving longer cases (case lengths being in the range of 1900-4000 words), and an advantage at inductive reasoning for reasoning models at early stages of case solving when evidence was scant.
