Search papers, labs, and topics across Lattice.
This paper introduces the Language Model Utility Taxonomy (LUX), a hierarchical framework for evaluating LLM utility across four domains: performance, interaction, operations, and governance. LUX addresses the gap in unified taxonomies for assessing LLM utility by organizing sociotechnical determinants beyond task-level performance. The authors provide a dynamic web tool linking LUX components to a repository of relevant metrics, enabling quantitative comparison and alignment of model selection with intended use.
LLM evaluation needs a holistic framework beyond just performance metrics, and this paper provides one: LUX, a sociotechnical taxonomy spanning performance, interaction, operations, and governance.
As large language models (LLMs) continue to improve at completing discrete tasks, they are being integrated into increasingly complex and diverse real-world systems. However, task-level success alone does not establish a model's fit for use in practice. In applied, high-stakes settings, LLM effectiveness is driven by a wider array of sociotechnical determinants that extend beyond conventional performance measures. Although a growing set of metrics capture many of these considerations, they are rarely organized in a way that supports consistent evaluation, leaving no unified taxonomy for assessing and comparing LLM utility across use cases. To address this gap, we introduce the Language Model Utility Taxonomy (LUX), a comprehensive framework that structures utility evaluation across four domains: performance, interaction, operations, and governance. Within each domain, LUX is organized hierarchically into thematically aligned dimensions and components, each grounded in metrics that enable quantitative comparison and alignment of model selection with intended use. In addition, an external dynamic web tool is provided to support exploration of the framework by connecting each component to a repository of relevant metrics (factors) for applied evaluation.
