[Paper Review] Gotcha! This Model Uses My Code! Evaluating Membership Leakage Risks in Code Models

Zhou Yang, Zhipeng Zhao+5 more|arXiv (Cornell University)|Oct 2, 2023

Software Engineering ResearchComputer Science3 citations

TL;DR

This paper proposes Gotcha, a novel membership inference attack tailored for code models, leveraging a surrogate model to predict whether a given code snippet was in the training data. The method achieves a high true positive rate of 0.95 and a low false positive rate of 0.10, revealing severe membership leakage risks in code models trained on sensitive code, including private corporate code.

ABSTRACT

Given large-scale source code datasets available in open-source projects and advanced large language models, recent code models have been proposed to address a series of critical software engineering tasks, such as program repair and code completion. The training data of the code models come from various sources, not only the publicly available source code, e.g., open-source projects on GitHub but also the private data such as the confidential source code from companies, which may contain sensitive information (for example, SSH keys and personal information). As a result, the use of these code models may raise new privacy concerns. In this paper, we focus on a critical yet not well-explored question on using code models: what is the risk of membership information leakage in code models? Membership information leakage refers to the risk that an attacker can infer whether a given data point is included in (i.e., a member of) the training data. To answer this question, we propose Gotcha, a novel membership inference attack method specifically for code models. We investigate the membership leakage risk of code models. Our results reveal a worrying fact that the risk of membership leakage is high: although the previous attack methods are close to random guessing, Gotcha can predict the data membership with a high true positive rate of 0.95 and a low false positive rate of 0.10. We also show that the attacker's knowledge of the victim model (e.g., the model architecture and the pre-training data) impacts the success rate of attacks. Further analysis demonstrates that changing the decoding strategy can mitigate the risk of membership leakage. This study calls for more attention to understanding the privacy of code models and developing more effective countermeasures against such attacks.

Motivation & Objective

• To investigate the risk of membership information leakage in code models trained on both public and private source code.
• To address the gap in understanding privacy vulnerabilities specific to code generation models, particularly regarding data membership inference.
• To develop and evaluate a targeted membership inference attack method for code models, given the unique characteristics of code data and model behavior.
• To examine how attacker knowledge of the victim model (e.g., architecture, pre-training data) affects the success of membership inference attacks.
• To explore mitigation strategies, such as decoding strategy modifications, to reduce membership leakage risks in code models.

Proposed method

• The method trains a surrogate model to mimic the victim code model’s confidence scores on input code snippets.
• It uses the confidence scores from the surrogate model to distinguish between training data and non-training data via a binary classification task.
• The attack leverages the model’s prediction confidence as a signal—higher confidence on training examples indicates potential membership.
• The approach is evaluated on CodeGPT, a code completion model, using a diverse set of code snippets from public and private sources.
• The attack is tested under varying levels of attacker knowledge, including full knowledge of model architecture and pre-training data.
• Decoding strategies such as temperature sampling and top-k sampling are modified to assess their impact on reducing membership leakage.

Figure 1 . Splitting the datasets to train and evaluate surrogate models as well as the membership inference classifiers.

Experimental results

Research questions

• RQ1 What is the extent of membership information leakage in code models trained on mixed public and private code?
• RQ2 How effective is a membership inference attack on code models compared to random guessing or existing methods?
• RQ3 How does an attacker’s knowledge of the victim model’s architecture and pre-training data influence the success of membership inference?
• RQ4 Can modifying decoding strategies in code generation models reduce the risk of membership leakage?
• RQ5 What are the implications of membership inference for privacy in code models used in industrial settings?

Key findings

• Gotcha achieves a true positive rate of 0.95 and a false positive rate of 0.10, significantly outperforming random guessing and prior methods.
• The attack's success rate increases substantially when the attacker has detailed knowledge of the victim model’s architecture and pre-training data.
• Membership inference is feasible even when the victim model is a large-scale code generation model like CodeGPT, indicating high privacy risk.
• Changing decoding strategies—such as using higher temperature or top-k sampling—can effectively reduce the model’s confidence leakage and mitigate membership inference risks.
• The study reveals that code models trained on private or sensitive code are vulnerable to membership inference, raising serious privacy concerns for industrial and enterprise use.
• This work is the first to systematically evaluate membership inference attacks on code models, highlighting a critical but previously overlooked privacy threat in code generation systems.

Figure 2 . The impact of different choices of surrogate models on the attack performance.