Search papers, labs, and topics across Lattice.
This paper introduces the concept of epistemic warrant to evaluate the reliability of individual recommendations made by large language models (LLMs) in the absence of ground truth. By adapting epistemological theories, the authors create a four-tier reliance certificate that categorizes recommendations based on their stability and contextual support. Validation through known-groups tests reveals that stronger warrants correlate with expert consensus, providing a nuanced framework for assessing LLM outputs beyond mere confidence levels.
Epistemic warrant reveals that not all LLM recommendations are created equal, with significant implications for decision-making in organizations.
Large language models are increasingly used to support organizational decisions, yet users often lack a principled basis for assessing whether to rely on a specific recommendation. Existing approaches typically evaluate broad model properties, such as reliability, uncertainty, or robustness, or focus on user trust, rather than the underlying basis for relying on an individual recommendation. Adapting theoretical foundations from epistemology, we introduce epistemic warrant, a decision-level construct that characterizes the stability of a model's preference and the scope over which that preference holds. We operationalize this construct through a four-tier reliance certificate for pairwise recommendations, distinguishing among unstable, context-dependent, locally supported, and broadly supported recommendations. We validate the construct using contemporary methodologies: known-groups tests successfully recover expert-prespecified warrant orderings, and stronger warrants systematically align with independent consensus from crowd workers. Furthermore, we demonstrate that epistemic warrant provides information distinct from verbalized confidence and is not readily explained by decision difficulty. Ultimately, this framework offers a theoretically grounded, implementable approach for characterizing the warrant of individual LLM recommendations when objective ground truth is unavailable.
