Search papers, labs, and topics across Lattice.
This paper conducts a systematic empirical study to assess the generalizability of state-of-the-art semantic code clone detectors beyond benchmark datasets. By employing a novel clone operator framework that generates semantically equivalent Type-4 clone instances through transformation operators, the authors evaluate 11 detectors on the BigCloneBench dataset. The findings indicate significant performance degradation across all detectors, revealing a reliance on shortcut learning rather than true semantic understanding, which questions the efficacy of current SOTA methods in real-world applications.
Despite high benchmark scores, SOTA semantic code clone detectors falter in real-world scenarios, revealing a reliance on shortcut learning over genuine semantic equivalence.
Code clone detection has been extensively studied for decades, and recent approaches have begun reporting remarkably high performance for semantic (Type-4) clones on benchmark datasets. However, it remains unclear whether these results reflect a genuine ability to capture semantic equivalence between programs, or simply an ability to exploit dataset-specific patterns. In this paper, we present the first systematic empirical study investigating the generalizability of state-of-the-art (SOTA) semantic code clone detectors beyond benchmark evaluation settings. Inspired by the inherent inclusion relationship among clone types, we propose a clone operator framework consisting of eight transformation operators derived from Type-2 and Type-3 clone variations. Using these operators, we construct distribution-shifted yet semantically equivalent Type-4 clone instances and evaluate 11 representative detectors spanning token-based, tree-based, and graph-based paradigms on the real-world BigCloneBench dataset. Our results reveal substantial performance degradation across all evaluated approaches, despite their strong benchmark performance. Further analyses show that existing detectors heavily rely on shortcut learning based on lexical and structural cues rather than robust semantic understanding. Our findings suggest that current SOTA semantic code clone detectors exhibit limited generalizability in real-world scenarios, highlighting important avenues for future research.
Tsinghua AI