[Paper Review] Challenging Machine Learning-based Clone Detectors via Semantic-preserving Code Transformations

Weiwei Zhang, Shengjian Guo+4 more|arXiv (Cornell University)|Nov 21, 2021

Software Engineering Research78 references4 citations

TL;DR

This paper proposes CloneGen, a framework that generates semantic-preserving code transformations to challenge the robustness of machine learning-based clone detectors. By applying lightweight transformations—such as variable renaming and loop restructuring—combined with heuristic and deep reinforcement learning strategies, CloneGen successfully evades detection in state-of-the-art ML-based detectors like ASTNN, TBCCD, and TextLSTM, revealing their vulnerability to simple code variations.

ABSTRACT

Software clone detection identifies similar code snippets. It has been an active research topic that attracts extensive attention over the last two decades. In recent years, machine learning (ML) based detectors, especially deep learning-based ones, have demonstrated impressive capability on clone detection. It seems that this longstanding problem has already been tamed owing to the advances in ML techniques. In this work, we would like to challenge the robustness of the recent ML-based clone detectors through code semantic-preserving transformations. We first utilize fifteen simple code transformation operators combined with commonly-used heuristics (i.e., Random Search, Genetic Algorithm, and Markov Chain Monte Carlo) to perform equivalent program transformation. Furthermore, we propose a deep reinforcement learning-based sequence generation (DRLSG) strategy to effectively guide the search process of generating clones that could escape from the detection. We then evaluate the ML-based detectors with the pairs of original and generated clones. We realize our method in a framework named CloneGen. CloneGen In evaluation, we challenge the two state-of-the-art ML-based detectors and four traditional detectors with the code clones after semantic-preserving transformations via the aid of CloneGen. Surprisingly, our experiments show that, despite the notable successes achieved by existing clone detectors, the ML models inside these detectors still cannot distinguish numerous clones produced by the code transformations in CloneGen. In addition, adversarial training of ML-based clone detectors using clones generated by CloneGen can improve their robustness and accuracy. CloneGen Meanwhile, compared with the commonly-used heuristics, the DRLSG strategy has shown the best effectiveness in generating code clones to decrease the detection accuracy of the ML-based detectors.

Motivation & Objective

• To investigate the robustness of recent machine learning-based clone detectors against semantic-preserving code transformations.
• To develop a lightweight, automated framework that generates code clones capable of evading detection by ML-based detectors.
• To evaluate whether heuristic search strategies and deep reinforcement learning can effectively generate adversarial clones that reduce detector accuracy.
• To demonstrate that current ML-based clone detectors remain vulnerable to simple, semantics-preserving code variations despite high reported accuracy.
• To advocate for improved robustness in clone detection models through adversarial training using generated clones.

Proposed method

• Design and implement 15 lightweight, semantic-preserving code transformation operators (e.g., variable renaming, loop conversion, statement reordering).
• Integrate three heuristic search strategies—Random Search, Genetic Algorithm, and Markov Chain Monte Carlo—to explore effective transformation combinations.
• Develop a deep reinforcement learning-based sequence generation (DRLSG) strategy to guide the search process toward more effective adversarial clone generation.
• Apply the framework to generate transformed code clones from original source code snippets while preserving semantic equivalence.
• Evaluate the generated clones against both ML-based (ASTNN, TBCCD, TextLSTM) and traditional (e.g., FCDetector) clone detectors to measure detection accuracy degradation.
• Use the generated adversarial clones for adversarial training to improve detector robustness and accuracy.

Experimental results

Research questions

• RQ1 Can semantic-preserving code transformations effectively evade detection by state-of-the-art machine learning-based clone detectors?
• RQ2 How do different search strategies (heuristics vs. DRLSG) compare in generating clones that reduce detector accuracy?
• RQ3 To what extent do current ML-based clone detectors fail to detect semantically equivalent code clones after lightweight transformations?
• RQ4 Can adversarial training using generated clones improve the robustness and accuracy of ML-based clone detectors?
• RQ5 What are the limitations of existing clone detectors when faced with simple, yet effective, code transformations?

Key findings

• Despite achieving over 95% accuracy on standard benchmarks, ML-based clone detectors like ASTNN, TBCCD, and TextLSTM failed to detect a significant number of clones generated by CloneGen after semantic-preserving transformations.
• The DRLSG strategy outperformed traditional heuristics in generating clones that evaded detection, demonstrating its effectiveness in guiding the search for adversarial code variants.
• Traditional clone detectors were also impacted by the generated clones, though to a lesser extent than ML-based detectors, indicating broader vulnerability to code transformations.
• Adversarial training using clones generated by CloneGen improved both the robustness and accuracy of the ML-based detectors, suggesting that such data augmentation can enhance model generalization.
• The study reveals a critical robustness gap in current ML-based clone detectors, highlighting the need for more resilient models and better training data that include semantic variants.
• The source code and experimental data are publicly available at https://github.com/CloneGen/CLONEGEN, enabling reproducibility and further research.