Search papers, labs, and topics across Lattice.
ClassicLogic introduces a novel benchmark suite aimed at evaluating compositional generalization in AI through four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. By incorporating a hierarchical knowledge base that formally defines complex solving strategies as compositions of simpler ones, the benchmark enables a detailed assessment of an agent's reasoning capabilities across varying levels of difficulty. This structured approach not only addresses a gap in existing benchmarks but also provides a rigorous testbed for advancing neuro-symbolic reasoning systems in AI.
ClassicLogic reveals that agents can be systematically evaluated on their ability to compose complex problem-solving strategies, a crucial aspect often overlooked in existing benchmarks.
Compositional generalization, the ability to understand and produce novel combinations of known components, remains a fundamental challenge for modern artificial intelligence. While few benchmarks exist, many focus on linguistic tasks and lack complex, explicit compositional structures. We introduce ClassicLogic, a new benchmark suite designed to evaluate an agent's ability to learn and compose problem-solving strategies. The benchmark consists of four classic logic puzzles: Sudoku, KenKen, Kakuro, and Futoshiki. Its core innovation is a hierarchical, explicit knowledge base for each game, where complex solving strategies are formally defined as compositions of simpler, foundational strategies. This structure allows for fine-grained evaluation of an agent's reasoning capabilities, from learning basic rules to applying multi-step compositional strategies to solve puzzles of increasing, mathematically validated difficulty. The open-source benchmark provides a challenging new testbed for advancing neuro-symbolic and other advanced AI reasoning systems.
