Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper investigates uncertainty quantification (UQ) methods for detecting functionally incorrect code generated by LLMs across multiple languages and models. It finds that token-probability-based UQ methods generalize well, while NLI-based methods fail due to the inability of NLI models to distinguish functionally different code. To address this, the authors introduce functional equivalence methods, including functional entropy, which leverages LLMs to assess functional equivalence and achieves state-of-the-art performance in predicting code correctness.
LLMs can now better self-assess the correctness of code they generate, thanks to a new functional entropy metric that understands code semantics, outperforming traditional uncertainty methods.
Large language models have shown impressive capabilities in code generation, yet they often produce functionally incorrect code. Uncertainty quantification (UQ) methods have emerged as a promising approach for detecting hallucinations in natural language generation, but their effectiveness for code generation tasks remains underexplored. We systematically evaluate how UQ techniques transfer to code generation across three programming languages, five LLMs, and over 1,700 problems. We find that some token-probability-based methods generalize effectively without modification, while sampling-based methods relying on natural language inference (NLI) fail because NLI models cannot distinguish functionally different code, causing most responses to collapse into a single semantic cluster. To address this, we introduce functional equivalence methods, a family of code-specific methods that replace NLI-based semantic equivalence with an LLM-based functional equivalence assessment, including functional entropy, a code-specific analog of semantic entropy. Functional equivalence methods achieve top AUROC in 11 out of 15 model-benchmark combinations and the best calibration across most settings, consistently outperforming both NLI-based counterparts and all other methods evaluated.