Search papers, labs, and topics across Lattice.
This paper addresses the limitations of large language models in reasoning with compound answer options that require logical combinations of atomic judgments. By introducing a framework that decomposes compound options into atomic answers and employs an operator-constrained integer linear program for scoring, the authors ensure that models evaluate each atomic component independently. The proposed method significantly enhances performance, achieving a Macro-F1 score increase from 48.3 to 77.0 on LOGICAL-COMMONSENSEQA and from 47.0 to 75.6 on LOGICAL-SATA, particularly excelling in NEITHER/NOR scenarios.
Large language models can achieve a 28.7-point boost in reasoning accuracy by breaking down complex answer options into simpler atomic judgments.
Large language models often fail when answer options require combining atomic judgments under explicit logical operators, even when they judge the individual atoms correctly. We study compound options connected by AND, OR, and NEITHER/NOR, introducing a framework that decomposes each option into atomic answers and scores contrastive hypotheses about each one, so the model never sees a compound option. An operator-constrained integer linear program then composes the calibrated scores into a single prediction. We evaluate on LOGICAL-COMMONSENSEQA and introduce LOGICAL-SATA, a reading-comprehension benchmark derived from SATA-Bench. Our framework improves Macro-F1 from 48.3 to 77.0 on the human-validated LOGICAL-COMMONSENSEQA split and from 47.0 to 75.6 on LOGICAL-SATA, with the largest gains on NEITHER/NOR.
