Track papers across labs, topics, and trends.
Search papers, labs, and topics across Lattice.
This paper evaluates the zero-shot generalizability of five state-of-the-art deep learning-based Java code clone detectors, including transformer-based models like CodeBERT and single-task models like FA-AST+GMN. The study trains and fine-tunes models on one dataset and evaluates them on others with different functionalities to assess their ability to generalize to unseen code. Results show that the deep learning models exhibit limited generalizability, with the conventional clone detector NiCad outperforming them in the zero-shot setting.
Despite near-perfect performance claims, deep learning-based code clone detectors fail to generalize to unseen Java code, even falling behind traditional methods.
Deep Learning (DL) is becoming more and more widespread in clone detection, motivated by achieving near-perfect performance for this task. In particular in case of semantic code clones, which share only limited syntax but implement the same or similar functionality, Deep Learning appears to outperform conventional tools. In this paper, we want to investigate the generalizability of DL-based clone detectors for Java. We therefore replicate and evaluate the performance of five state-of-the-art DL-based clone detectors, including Transformers like CodeBERT and single-task models like FA-AST+GMN, in a zero-shot evaluation scenario, where we train/fine-tune and evaluate on different datasets and functionalities. Our experiments demonstrate that the models'generalizability to unseen code is limited. Further analysis reveals that the conventional clone detector NiCad even outperforms the DL-based clone detectors in such a zero-shot evaluation scenario.