Search papers, labs, and topics across Lattice.
This paper investigates the reliability of large language models (LLMs) in ranking tasks critical for resource allocation, such as homelessness services and emergency triage. By employing a classical diagnostic measure, the coefficient of consistency $\zeta$, alongside standard ranking distance measures like Kendall's $\tau$, the authors assess intra-run and inter-run consistency of LLM judgments. The findings reveal significant variability in performance across different LLMs, underscoring the necessity for practitioners to evaluate consistency before relying on these models for high-stakes decisions.
LLMs exhibit substantial variability in ranking consistency, revealing that not all models are equally trustworthy for critical decision-making tasks.
From housing allocation for households experiencing homelessness to triage in emergency departments, LLMs are increasingly being considered as judges of consequential decisions that require ranking people for scarce resources. Ranking large groups simultaneously is cognitively demanding and error-prone. A natural solution, drawing on decades of social choice theory, elicits pairwise comparisons and aggregates them into a total order. However, a fundamental question remains when LLMs serve as the pairwise judge: how can a practitioner tell, before committing to a ranking, whether the LLM's judgments are sufficiently consistent to trust the result? We discuss two different ways of identifying consistency. A classical diagnostic, the coefficient of consistency $\zeta$, originally developed to measure judge reliability by counting circular triads in tournament graphs, provides a cheap, model-free measure of intra-run consistency. Various standard measures of distance between rankings, for example Kendall's $\tau$, can measure inter-run variability. We show, in both theory and practice, that these measures are independently valuable, and advocate for using both to assess reliability of rankings. We demonstrate the practical importance of our results across two high-stakes prioritization tasks: homelessness service allocation and emergency department triage. Three different leading LLMs have considerably different performance profiles across these two axes of consistency. We provide guidelines for how practitioners could think about measuring and assessing consistency before committing to a model for ranking or prioritization.
