Search papers, labs, and topics across Lattice.
OenoBench is a comprehensive benchmark designed to evaluate large language models (LLMs) in the wine domain, comprising 3,266 multiple-choice questions derived from 38,104 verified facts across six thematic pillars. The methodology employs a unique LLM-driven pipeline that reformats and audits facts while ensuring traceability to original sources, with questions generated through diverse strategies and rigorously scored against a human gold standard. Key findings reveal a significant accuracy range of 53%-84% across various model configurations, highlighting the nuanced performance differences among models and the impact of reasoning modes on accuracy.
OenoBench reveals that even leading LLMs struggle with knowledge retention, achieving only 53%-84% accuracy on wine-related questions despite extensive training.
We introduce OenoBench, a wine-domain knowledge benchmark of 3,266 multiple-choice questions across six pillars (regions, grape varieties, viticulture, winemaking, producers, business) and four difficulty tiers. The corpus is built from 38,104 atomic, source-anchored facts extracted by 35 provenance-verified scrapers from government registries (INAO, TTB, OIV), peer-reviewed journals, and Wikipedia/Wikidata. Our methodological contribution is an LLM-driven pipeline in which language models reformat verified facts and audit the result, but never serve as the source of truth: every claim traces to a URL, every question is generated by one of five strategies across five generator families, and every question is scored by a nine-agent audit calibrated against a human gold sheet via Cohen's $\kappa$. Evaluating sixteen frontier configurations, we find: (i) overall accuracy spans 53%-84%, led by o3 at 83.6%; (ii) reasoning-mode lift concentrates in DeepSeek R1 (+6.8pp) and is absent in Claude Opus and Gemini Pro; (iii) Anthropic shows +9pp self preference on its own questions while Google shows -8pp inverse preference; (iv) frontier open-weight models share the cost-vs-accuracy Pareto frontier with proprietary reasoning models; and (v) every config gains around 33pp on closed-book solvable items, revealing a parametric-recall ceiling that only the contextual slice avoids. We release corpus, audit findings, human-review app, and construction code under CC-BY-SA-4.0.
