Search papers, labs, and topics across Lattice.
This paper addresses the transparency issues surrounding AI-generated code by advocating for automated and explainable provenance systems that trace the origins of code back to its prompts, training data, and model components. The authors argue that existing tools fail to provide necessary explanations for problematic outputs, which is crucial for developers and compliance professionals. Their research vision is supported by empirical studies highlighting the demand for provenance information, suggesting that future CodeGenAI tools must integrate these capabilities to enhance accountability and usability.
AI-generated code lacks transparency, but a new framework for automated provenance could revolutionize developer trust and compliance.
Generative AI for code generation has transformed software development, but it has also introduced a critical transparency problem: the origins of AI-generated code are opaque to the developers who use it, the organizations that deploy it, and the compliance professionals responsible for ensuring its legal and quality standards. Existing mitigations flag problematic outputs after the fact without explaining why a model produced them or how future generation could be improved. We present a research vision, grounded in a U.S. NSF-funded research grant, that argues that the next generation of CodeGenAI tools must be built on a foundation of explainable provenance: automated, post-hoc traceability that links generated code back to the prompt components, training data instances, global data features, and internal model components that caused its generation. We grounded this vision in empirical evidence from studies of software developers, model users, and compliance/legal professionals, which show that provenance information is a practical necessity that current tools do not provide. We characterize the problem across four traceability dimensions, outline a research program combining large-scale empirical studies with post-hoc causal and interpretability techniques, and identify the key open challenges that the community must address to realize this vision.
