Search papers, labs, and topics across Lattice.
This paper investigates the phenomenon of code hallucination in large language models (LLMs), distinguishing it from ordinary code errors by establishing a comprehensive taxonomy based on groundedness, manifestation level, and behavior. The authors create an adversarial benchmark comprising 270 unsatisfiable prompts across multiple programming languages, assessing model responses against a validated human labeling protocol. Key findings reveal that LLMs generate ungrounded code in approximately 60% of cases, with a refusal rate of only 27% for unsatisfiable prompts, highlighting significant limitations in current model reliability.
LLMs generate plausible but ungrounded code 60% of the time, revealing a critical gap in their reliability for programming tasks.
Large language models (LLMs) often produce code that looks plausible but is not grounded in reality. The code may import packages that do not exist or claim to implement algorithms that violate proven theorems, while still compiling and running. We study \emph{code hallucination} as \emph{ungrounded generation} and separate it from ordinary \emph{code error} (bugs in otherwise grounded programs). We propose a taxonomy with three dimensions: \textbf{groundedness} (absolute violations of universal truths vs.\ relative fabrications of contingent or ecosystem-specific facts), \textbf{manifestation level} (syntactic, semantic, or factual), and \textbf{behavior} (from confident fabrication to degenerate output), organized into a severity ordering. We build an \textbf{adversarial} suite of deliberately unsatisfiable tasks where the correct response is to refuse and categorize the responses under our taxonomy. The suite contains \textbf{270 prompts} across six languages and 24 subcategories, paired with \textbf{91 matched solvable controls}, and responses are judged by a two-tier protocol validated against human labels (82\% agreement, $\kappa{=}0.73$). Across twelve open-weight code and reasoning models (4{,}332 judged responses), models produce ungrounded code on about 60\% of unsatisfiable prompts and refuse only 27\%, while wrongly refusing 0\% of the solvable controls.
