Search papers, labs, and topics across Lattice.
The paper introduces General365, a benchmark to evaluate general reasoning in LLMs, decoupling it from specialized knowledge by restricting background knowledge to a K-12 level. The benchmark consists of 365 seed problems and 1,095 variant problems across eight categories, designed to be challenging and diverse. Evaluations of 26 LLMs on General365 reveal that even the best model achieves only 62.8% accuracy, highlighting a significant gap in general reasoning compared to domain-specific tasks.
LLMs that ace math and physics still struggle with general reasoning, achieving only 63% accuracy on a new K-12 level benchmark.
Contemporary large language models (LLMs) have demonstrated remarkable reasoning capabilities, particularly in specialized domains like mathematics and physics. However, their ability to generalize these reasoning skills to more general and broader contexts--often termed general reasoning--remains under-explored. Unlike domain-specific reasoning, general reasoning relies less on expert knowledge but still presents formidable reasoning challenges, such as complex constraints, nested logical branches, and semantic interference. To address this gap, we introduce General365, a benchmark specifically designed to assess general reasoning in LLMs. By restricting background knowledge to a K-12 level, General365 explicitly decouples reasoning from specialized expertise. The benchmark comprises 365 seed problems and 1,095 variant problems across eight categories, ensuring both high difficulty and diversity. Evaluations across 26 leading LLMs reveal that even the top-performing model achieves only 62.8% accuracy, in stark contrast to the near-perfect performances of LLMs in math and physics benchmarks. These results suggest that the reasoning abilities of current LLMs are heavily domain-dependent, leaving significant room for improvement in broader applications. We envision General365 as a catalyst for advancing LLM reasoning beyond domain-specific tasks toward robust, general-purpose real-world scenarios. Code, Dataset, and Leaderboard: https://general365.github.io