Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper introduces HybridCodeAuthorship, a benchmark dataset designed for line-level code authorship detection that reflects the hybrid nature of AI and human contributions in industry codebases. By leveraging CodeSearchNet, the authors constructed a dataset of Python code files featuring interleaved human- and AI-authored lines, addressing the limitations of existing benchmarks that focus solely on fully human or AI-generated code. The evaluation of two state-of-the-art detection algorithms revealed that the best-performing model, AIGCode Detector, achieved F1 scores of 0.48 and 0.56 for chunk-level and line-level detection tasks, respectively, highlighting the dataset's challenging nature.
AIGCode Detector achieves surprising performance on a benchmark that mirrors real-world hybrid codebases, revealing the complexities of AI and human collaboration in coding.
Thanks to the rapid adoption of AI code assistants powered by large language models (LLMs), industry codebases are, increasingly, a hybrid of AI- and human-authored code. For risk management and productivity analysis purposes, it is crucial to enable fine-grained location detection of AI-generated code. To develop algorithms for this task, quality benchmarks are needed to assess performance. However, existing benchmarks tend to comprise academic, LeetCode-style problems and presume a code snippet is either completely human-authored or completely AI-authored, which is not reflective of the diverse intents and styles of industry codebases utilizing AI code assistants. To fill these gaps, we introduce HybridCodeAuthorship, a novel benchmark of Python code files with interleaved human- and AI-authored lines of code to simulate authentic utilization of AI code assistants. In this paper, we first present our dataset construction pipeline, which leverages CodeSearchNet, a massive collection of links to open sourced repositories on GitHub. We then benchmark the performance of two state-of-the-art AI-generated code detection algorithms at both the line- and chunk-level. Experimental results demonstrate that HybridCodeAuthorship is a challenging benchmark with a top-scoring algorithm, AIGCode Detector, obtaining a highest F1 score of 0.48 and 0.56 on chunk-level and line-level code detection tasks, respectively.