Search papers, labs, and topics across Lattice.
This paper explores the critical challenge of preference reasoning in large language models (LLMs) under conditions of indeterminacy, which is often overlooked in standard benchmarks. The authors categorize indeterminacy into epistemic and structural types, highlighting how incomplete information and the absence of valid solutions complicate decision-making. Through a series of tasks, they demonstrate that current state-of-the-art LLMs struggle to differentiate between determined and undetermined scenarios, leading to significant miscalibrations in their reasoning capabilities.
LLMs are miscalibrated in their reasoning, failing to distinguish between scenarios where valid solutions exist and where they do not, which could undermine their effectiveness in real-world applications.
As large language models evolve into decision-making agents, the ability to reason over preferences becomes fundamental to alignment, coordination, and collective intelligence. Yet, unlike standard benchmarks, real-world preference reasoning is inherently indeterminate: information may be incomplete, and valid solutions may not exist. We argue that indeterminacy, rather than correctness alone, is a central challenge for AI reasoning. We formalize this challenge along two axes, (i) epistemic indeterminacy, arising from incomplete, partial, or expressive preferences, and (ii) structural indeterminacy, arising from the non-existence of solutions under standard social choice concepts. Across a hierarchy of tasks, we show that state-of-the-art language models systematically fail to distinguish between determined and undetermined instances, exhibiting miscalibrated reasoning even in verification settings.
