Search papers, labs, and topics across Lattice.
This study systematically evaluates the impact of positional bias in ordinal classification across ten large language models (LLMs), revealing that variations in label order, demonstration order, and placement significantly influence model predictions. The experiments demonstrate that while accuracy and stability are often misaligned, lower scale cardinality consistently enhances both metrics. Additionally, various debiasing methods were tested, with a comparison-based listwise formulation emerging as the most effective, though its performance varied across models and bias sources, highlighting the need for a holistic approach to model selection.
Positional bias in LLMs can drastically alter predictions, with accuracy and stability often at odds, challenging our understanding of model reliability in ordinal classification tasks.
Large language models are increasingly used for ordinal classification, yet semantically equivalent changes to prompt organization can alter their predictions. We conduct systematic experiments to characterize positional bias from label order, demonstration order, and demonstration placement. First, we apply the three probes to ten frontier LLMs on a common ordinal-classification task; every model is sensitive to all three positional sources, showing that the problem is pervasive. Second, we vary eight prompt-, task-, and model-level factors across five datasets; accuracy and stability are often misaligned, and only lower scale cardinality consistently improves both. Third, we compare pointwise, pairwise, and listwise inference, alternative aggregation and debiasing methods, and joint configurations; the tested corrections do not provide a reliable remedy, while a comparison-based listwise formulation offers the best balance but transfers unevenly across models and bias sources. These findings show that positional robustness depends on the full system configuration rather than the model alone. Ordinal-classification systems should therefore be selected jointly for predictive performance and stability.
UW