Search papers, labs, and topics across Lattice.
This study investigates how well language models adhere to modal logic specifications by constructing paired modal problems that share premises but differ in frame or domain conditions. The results reveal that while four out of five recent models perform poorly under direct prompting, enabling a reasoning mode significantly boosts performance for one model, indicating that adherence to modal semantics is heavily influenced by both inference mode and model identity. The findings suggest that models may align with different familiar logics when frame conditions are omitted, highlighting the complexity of evaluating modal reasoning in AI systems.
Language models struggle with modal logic, often performing below baseline expectations, but a simple switch to reasoning mode can dramatically enhance their accuracy.
Reasoning about necessity and possibility depends on assumptions about accessibility between worlds and about which objects exist at each one. The same inference may therefore hold under one modal system and fail under another. Evaluating language models on such problems requires testing whether their judgments follow the stated semantics rather than a familiar logic. We construct paired modal problems with identical premises and conjecture but different frame or domain conditions; automated reasoning verifies opposite labels. A balanced core prevents the semantic condition alone from revealing the answer. On this core, four of five recent models perform below the condition-only baseline under direct prompting. Yet enabling reasoning mode raises DeepSeek V4 Flash from 4.4% to 88.1% on unchanged prompts. Following stipulated modal semantics thus depends strongly on inference mode as well as model identity. When frame conditions are omitted, models often agree but fit different familiar logics best. We release the formulas, oracle artifacts, countermodels, and responses.
