Search papers, labs, and topics across Lattice.
This study conducts a large-scale evaluation of uncertainty estimation (UE) methods across 22 languages, revealing critical insights into how language generation impacts model performance. By prompting models to reason in English while handling questions in low-resource languages, the researchers demonstrate significant improvements in UE, indicating that the bottleneck lies in generation rather than comprehension. Additionally, the findings suggest that the choice of UE method should be tailored to model size, with open-box methods favored for smaller models and closed-box methods for larger ones.
Prompting LLMs to reason in English can significantly enhance uncertainty estimation in low-resource languages, revealing a surprising reliance on generation over comprehension.
Uncertainty estimation (UE) enables LLM-powered systems to recognize when to abstain, yet existing research has predominantly focused on English. We present the first large-scale evaluation of UE methods across 22 languages, spanning high-, mid-, and low-resource settings. Using two human-curated Q\&A datasets, we compare open and closed box UE methods (nine in total) across different model sizes and architectures while eliciting long-form reasoning, avoiding LLM-as-a-judge and embedding-based scoring, which can introduce evaluation noise. We report three main actionable findings. First, we find that prompting models to reason in English while keeping questions in low-resource languages substantially improves UE performance, suggesting that comprehension of low-resource languages is largely intact, and that the reliability bottleneck lies in generation rather than understanding. Second, prompting models to reason in English closes the UE performance gap between low and high-resource languages, demonstrating that generation language matters more than the question language. Third, the choice of UE method should depend on model scale: at smaller scales, open-box probability-based methods outperform alternatives; at larger scales, closed-box self-verbalized uncertainty becomes superior. Finally, we provide an analysis of threshold selection for selective prediction, offering guidance on calibrating abstention in multilingual settings.
