Search papers, labs, and topics across Lattice.
This paper introduces ComBench, a benchmark specifically designed to assess the combinatorial reasoning capabilities of large language models (LLMs) using Olympiad-level problems. The benchmark comprises 100 human-annotated problems categorized into analysis-centric and construction-centric tasks, allowing for a nuanced evaluation of proof quality and construction validity. Experimental results reveal that even the strongest models exhibit significant gaps in performance, with distinct capabilities in rigorous proof reasoning and constructive realization, highlighting the need for further advancements in mathematical reasoning within AI.
Even the best LLMs struggle with Olympiad-level combinatorics, achieving only 65.4% on a benchmark designed to expose their reasoning limitations.
Combinatorics is central to Olympiad-level mathematical problem solving, requiring deep discrete reasoning, creative constructions, and rigorous structural insight. Recent evidence suggests that even today's strongest frontier models remain uneven on Olympiad combinatorics, revealing a gap in creative mathematical reasoning. We introduce ComBench, an Olympiad-level combinatorics benchmark for evaluating and diagnosing the combinatorial reasoning capabilities of large language models. ComBench contains 100 human-annotated competition-level problems organized around two complementary settings: analysis-centric problems, which primarily require rigorous mathematical arguments, and construction-centric problems, which require explicit constructions in addition to correctness justifications. The evaluation protocol combines rubric-guided proof grading with deterministic construction verification, exposing cases where proof quality and construction validity diverge. Experiments on frontier open- and closed-source models show that ComBench is far from saturated: the strongest model reaches 65.4% overall Avg. and 75.3% overall Best@4. We further find that Rigorous Proof Reasoning and Constructive Realization are distinct capabilities: Kimi-K2.6 trails GPT-5.5 on analysis-centric proof grading but surpasses it on construction-centric Best@4, while Existence and Construction problems remain consistently hardest across representative frontier models.
Tsinghua AI