Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper addresses the critical need for reliable uncertainty estimation (UE) in code generation by large language models (LLMs), highlighting the unique challenges posed by code compared to natural language. The authors identify three distinct axes of uncertainty—lexical, algorithmic, and functional—and demonstrate that their three-axis ensemble approach significantly enhances predictive performance across five code LLMs. The results show an improvement in average AUROC from 0.696 to 0.776, indicating that code-specific UE methods are essential for ensuring safety and reliability in automated code generation.
Code generation models can achieve a significant boost in reliability by leveraging a tailored uncertainty estimation framework that outperforms traditional natural language methods.
Large language models (LLMs) are increasingly deployed as code generators, where silently wrong programs pose real safety and reliability risks. Reliable uncertainty estimation (UE) is essential for selective prediction, human-in-the-loop review, and downstream agentic decisions. Yet most existing code UE methods are inherited from natural language (NL) generation and ignore properties that make code distinct. We argue that code differs from NL in three ways: a single wrong token can break an entire program (token fragility); algorithmic intent and concrete implementation can disagree independently (intent-code gap); and programs can be executed (executability). We instantiate these properties as three orthogonal uncertainty axes: lexical (Top-K token entropy), algorithmic (pseudo-code consistency), and functional (behavioral consistency). Across five code LLMs, our three-axis ensemble improves average AUROC from 0.696 for the strongest NL-derived baseline to 0.776 (+8.1 points). Notably, on Qwen3-14B, our single-pass Top-K token entropy matches the strongest multi-pass baseline while being over 3x cheaper; across models, it remains a competitive low-cost signal. These results suggest that code UE deserves code-specific design rather than direct NL ports.