Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces Code-MUE, a black-box framework that measures the uncertainty of Code LLMs by utilizing execution-based Semantic Interaction Graphs to bridge the gap between syntax and semantics. By calculating the Von Neumann entropy of the solution space, Code-MUE provides a quantifiable measure of global semantic diversity that correlates strongly with functional correctness. The empirical evaluation across eight state-of-the-art LLMs reveals a Spearman's correlation of up to -0.98, indicating that Code-MUE significantly outperforms existing uncertainty estimation methods and enhances risk detection in software engineering workflows.
Code-MUE reveals a striking -0.98 correlation between uncertainty and functional correctness in Code LLMs, offering a new lens for assessing model reliability.
As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences. Reliable automation, therefore, demands the ability to distinguish between confident, well-supported predictions and stochastic guessing. However, existing uncertainty estimation methods face a critical gap: white and grey-box techniques are often inapplicable to closed-source models, while standard"black-box"text metrics fail to capture the unique fragility of code, where syntactic variation does not always imply semantic divergence. To bridge this syntax-semantics gap, we introduce Code-MUE, a purely black-box framework that measures uncertainty through execution-based Semantic Interaction Graphs. Unlike prior approaches that rely on superficial textual similarity, Code-MUE grounds uncertainty in observable runtime behavior, calculating the Von Neumann entropy of the solution space to quantify global semantic diversity. A large-scale empirical study across eight state-of-the-art LLMs demonstrates that Code-MUE achieves a strong negative correlation with functional correctness (Spearman's correlation up to -0.98), significantly outperforming lexical and embedding-based baselines while enabling robust risk detection and selective prediction in practical workflows.