[Paper Review] Semantic-Preserving Adversarial Code Comprehension

Yiyang Li, Hongqiu Wu+1 more|arXiv (Cornell University)|Sep 12, 2022

Adversarial Robustness in Machine Learning6 citations

TL;DR

This paper proposes Semantic-Preserving Adversarial Code Embeddings (SPACE), a novel method that enhances pre-trained language models (PrLMs) for code comprehension by generating worst-case adversarial perturbations in the continuous embedding space while preserving code semantics and grammatical correctness. SPACE simultaneously improves model performance and robustness, outperforming strong baselines like CodeBERT and GraphCodeBERT on multiple benchmarks and reducing adversarial success rates by over 10% across state-of-the-art attacks.

ABSTRACT

Based on the tremendous success of pre-trained language models (PrLMs) for source code comprehension tasks, current literature studies either ways to further improve the performance (generalization) of PrLMs, or their robustness against adversarial attacks. However, they have to compromise on the trade-off between the two aspects and none of them consider improving both sides in an effective and practical way. To fill this gap, we propose Semantic-Preserving Adversarial Code Embeddings (SPACE) to find the worst-case semantic-preserving attacks while forcing the model to predict the correct labels under these worst cases. Experiments and analysis demonstrate that SPACE can stay robust against state-of-the-art attacks while boosting the performance of PrLMs for code.

Motivation & Objective

• To address the trade-off between performance and robustness in pre-trained language models (PrLMs) for source code comprehension.
• To explore the feasibility and effectiveness of adversarial training in the continuous embedding space rather than discrete token space for code.
• To develop a practical and differentiable method that maintains semantic meaning and grammatical correctness during adversarial training.
• To simultaneously improve both the generalization performance and robustness of PrLMs on code understanding tasks.

Proposed method

• SPACE generates worst-case adversarial perturbations in the continuous embedding space of code tokens, rather than modifying discrete tokens directly.
• The method uses gradient-based optimization to find the most harmful perturbations that preserve semantic meaning and syntactic correctness.
• Perturbations are constrained to only affect identifier embeddings, leaving keywords and structural tokens unchanged to maintain grammatical integrity.
• The approach is naturally differentiable and integrates seamlessly into standard gradient-based training frameworks with minimal code changes.
• It combines adversarial training with domain-specific characteristics of programming languages, such as semantic preservation and syntactic validity.
• The method is evaluated using both gradient-based and random perturbation baselines, demonstrating superiority in robustness and performance.

Experimental results

Research questions

• RQ1 Can adversarial training in the continuous embedding space improve both performance and robustness of PrLMs for code comprehension without compromising either?
• RQ2 How does semantic-preserving adversarial training compare to standard adversarial training that modifies discrete tokens?
• RQ3 To what extent can embedding-space perturbations maintain grammatical correctness and semantic meaning while still being effective as adversarial examples?
• RQ4 Does the proposed method outperform strong baselines like CodeBERT and GraphCodeBERT in both clean and adversarial settings?
• RQ5 How robust is the model to real-world semantic-preserving code transformations such as identifier renaming and control flow changes?

Key findings

• SPACE significantly improves the performance of CodeBERT and GraphCodeBERT across multiple code comprehension tasks, with average accuracy gains of 2.65% on CodeSearchNet and 2.62% on CodeXGLUE.
• On the CodeQA dataset, SPACE reduces the adversarial success rate (ASR) of GraphCodeBERT by over 10% across all tested attack methods, demonstrating superior robustness.
• SPACE reduces performance drops due to semantic-preserving code transformations by 3.6% and 4.5% on average for CodeBERT and GraphCodeBERT, respectively, compared to baseline models.
• Vanilla adversarial training on all tokens provides minimal robustness gains (less than 0.5%), indicating that it fails to preserve semantic and syntactic integrity.
• Gradient-based perturbations are consistently more effective than random perturbations, and semantic-preserving constraints further enhance robustness.
• The method is practical and efficient, requiring only a few lines of code to integrate into existing training pipelines due to its differentiable nature.