Search papers, labs, and topics across Lattice.
This paper introduces HOLMES, a novel benchmark designed to evaluate higher-order logical reasoning in large language models (LLMs), addressing the limitations of existing benchmarks that focus primarily on first-order logic. The benchmark comprises 1,379 instances that incorporate natural-language problems paired with formal higher-order logic representations, enabling a more comprehensive assessment of reasoning capabilities in real-world contexts such as law and finance. Experimental results reveal that current LLMs perform poorly on HOLMES, achieving an average accuracy of only 50.64%, highlighting higher-order symbolic reasoning as a significant challenge for developing reliable AI systems.
Current LLMs achieve only 50.64% accuracy on a new benchmark for higher-order logical reasoning, revealing a critical gap in their reasoning capabilities.
Logical reasoning is essential for reliable AI, yet existing benchmarks are largely first-order-logic-centric, focusing on object-level deduction over fixed predicates. This misses many realistic scenarios where models must reason over rules, predicates, functions, constraints, and decision procedures themselves. We introduce HOLMES (Higher-Order Logic Meets real-world Explainable Symbolic reasoning), the first real-world benchmark for higher-order symbolic reasoning in LLMs, containing 1379 instances. Built on higher-order logic, HOLMES pairs natural-language problems with HOL formalizations, ground-truth answers, verifiable reasoning traces, and fine-grained controllable reasoning factors across law and finance. Experiments show that current LLMs still struggle on HOLMES, with an average accuracy of only 50.64% and the best model reaching 59.54%. Our analyses further reveal that high final-answer accuracy can mask shortcut reasoning in conflict-resolution settings, while performance drops sharply under scope-conditioned and compositional reasoning. These findings identify higher-order symbolic reasoning as a key bottleneck for building reliable and verifiable LLMs. The project code and dataset are publicly available at https://github.com/wuyucheng2002/HOLMES.
