Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
To mitigate logical errors and security vulnerabilities in generated code without the overhead of sandboxed execution, the authors developed CodeTD, a pre-execution verification method that extracts topological features from LLM attention maps to quantify prompt-generation mismatch. Detecting code hallucinations statically is critical for safe real-time deployment where executing untrusted code introduces latency and attack surfaces. Across evaluations spanning 10 code LLMs up to 34B parameters and five programming languages on benchmarks like BigCodeBench and MultiPL-E, CodeTD consistently outperforms existing pre-execution baselines and demonstrates strong cross-benchmark transferability.
The topological geometry of an LLM's attention maps reveals whether code is hallucinated before a single line is executed, outperforming traditional pre-execution heuristics across models up to 34B parameters.
As AI-code assistant tools become widespread, automatic assessment of the correctness of generated code becomes a significant challenge. Code LLMs are prone to hallucinations, which may lead to code that does not solve the required problem, or even to code with severe security vulnerabilities. In this paper, we introduce CodeTD -- the first approach to pre-execution assessment of code correctness based on topological data analysis (TDA) of Code LLMs' attention maps. Our method quantifies prompt-generation mismatch using topological patterns of attention maps. We carry out experiments with common benchmarks (HumanEval, MBPP, BigCodeBench, MultiPL-E), 5 programming languages and 10 Code LLMs of size up to 34B parameters. The experimental results show that the proposed method outperforms recent baselines. Moreover, CodeTD is transferable between coding benchmarks.