Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates the impact of input ambiguity on error prediction in Large Language Models (LLMs) by disentangling it from uncertainty quantification (UQ) signals. The authors demonstrate that traditional UQ metrics are less effective in predicting errors for ambiguous inputs, leading to a novel method that incorporates ambiguity information into the error prediction pipeline using Gated Experts and Selective Prediction. Their approach yields significant improvements in error prediction scores, exceeding 10 points in Precision-Recall Rate (PRR) across various datasets and model families, highlighting the importance of considering input ambiguity in UQ tasks.
Disentangling input ambiguity from uncertainty signals can boost error prediction accuracy in LLMs by over 10 points in PRR across diverse datasets.
The task of Error Prediction, namely predicting whether a model output is correct, is commonly tackled with Uncertainty Quantification (UQ). However, while uncertainty metrics capture when models lack knowledge or capacity to make a prediction, they also reflect aleatoric uncertainty, which is inherent in the model input and context. This paper presents a method for improving error prediction for Large Language Models (LLMs), by disentangling input ambiguity from UQ signal. We conduct experiments on the task of Question Answering (QA) with six UQ metrics and show that UQ metrics are more predictive of errors on unambiguous instances than on questions with multiple plausible answers. We use Gated Experts and Selective Prediction to incorporate gold and predicted ambiguity labels into the error prediction pipeline. We find that ambiguity information improves error prediction scores across model families, training and evaluation paradigms, datasets (including allegedly unambiguous ones), and sources of aleatoric uncertainty, yielding improvements of over 10 points of PRR for individual UQ metrics on standard datasets.