Search papers, labs, and topics across Lattice.
This study investigates functional memorization in code language models by comparing a midtrained model, which has been exposed to specific target code, with a pretrained reference model that has not. By prompting both models with Python function signatures and assessing both textual and functional similarity, the authors reveal that functional logic can be extracted even when the generated code is not textually similar to the training data. The findings underscore the necessity for auditing metrics that account for functional equivalence, rather than relying solely on textual overlap.
Functional memorization in code generation reveals that LLMs can replicate logic without textual similarity, challenging traditional auditing methods.
Large language models (LLMs) are increasingly used to generate code at scale. Meanwhile, prior work has investigated whether training data may be recoverable from model outputs, by auditing the textual overlap between training examples and model generations. Code, however, can be functionally equivalent while textually dissimilar. In this work, we study functional memorization: extraction of functional logic beyond what verbatim metrics detect. We construct a counterfactual setup for Olmo-3-32B, comparing a midtrained model (exposed to target code) against a pretrained reference (not exposed). We prompt both models with Python function signatures and measure both textual and functional similarity (i.e., LLM-as-a-judge, execution-based). Our results show clear evidence of functional memorization, highlighting the need for auditing metrics that go beyond textual overlap.
