[Paper Review] Your Code Secret Belongs to Me: Neural Code Completion Tools Can Memorize Hard-Coded Credentials

Yizhan Huang, Yichen Li+3 more|arXiv (Cornell University)|Sep 14, 2023

Ferroelectric and Negative Capacitance Devices46 references4 citations

TL;DR

This paper proposes HCR, a tool that exploits neural code completion tools (NCCTs) like GitHub Copilot and Amazon CodeWhisperer to extract hard-coded credentials from their outputs by generating contextually relevant prompts from real GitHub code. The study demonstrates that NCCTs can memorize and leak real secrets—extracting 2,702 valid credentials from Copilot and 129 from CodeWhisperer, with 3.6% and 5.4% being real, active secrets, respectively.

ABSTRACT

Neural Code Completion Tools (NCCTs) have reshaped the field of software engineering, which are built upon the language modeling technique and can accurately suggest contextually relevant code snippets. However, language models may emit the training data verbatim during inference with appropriate prompts. This memorization property raises privacy concerns of NCCTs about hard-coded credential leakage, leading to unauthorized access to applications, systems, or networks. Therefore, to answer whether NCCTs will emit the hard-coded credential, we propose an evaluation tool called Hard-coded Credential Revealer (HCR). HCR constructs test prompts based on GitHub code files with credentials to reveal the memorization phenomenon of NCCTs. Then, HCR designs four filters to filter out ill-formatted credentials. Finally, HCR directly checks the validity of a set of non-sensitive credentials. We apply HCR to evaluate three representative types of NCCTs: Commercial NCCTs, open-source models, and chatbots with code completion capability. Our experimental results show that NCCTs can not only return the precise piece of their training data but also inadvertently leak additional secret strings. Notably, two valid credentials were identified during our experiments. Therefore, HCR raises a severe privacy concern about the potential leakage of hard-coded credentials in the training data of commercial NCCTs. All artifacts and data are released for future research purposes in https://github.com/HCR-Repo/HCR.

Motivation & Objective

• To investigate whether commercial neural code completion tools (NCCTs) can memorize and leak hard-coded credentials from their training data.
• To develop a systematic method for identifying and extracting such memorized secrets using realistic code contexts.
• To evaluate the privacy risks of NCCTs in real-world deployment, especially concerning sensitive credentials like API keys and tokens.
• To demonstrate that even with black-box access, adversaries can extract real, active secrets from NCCTs using prompt engineering.
• To inform model developers and users about the need for proactive mitigation strategies to prevent credential leakage.

Proposed method

• Identify 18 common secret types (e.g., AWS keys, Stripe tokens) using GitHub’s Secret Scanning list.
• Design regular expressions (regex) to detect secret patterns in code files, enabling automated identification of credential-containing repositories.
• Collect code files with secrets from GitHub, remove the secrets, and use the modified code as prompts to trigger NCCTs to complete the missing credentials.
• Apply four filters—regex pattern matching, entropy thresholding, format validation, and service-specific syntax checks—to distinguish genuine secrets from random strings.
• Use black-box querying to assess NCCT responses and validate extracted secrets via direct access or documentation checks.
• Conduct large-scale evaluation on two commercial NCCTs: GitHub Copilot and Amazon CodeWhisperer.

Experimental results

Research questions

• RQ1 Can commercial NCCTs like GitHub Copilot and Amazon CodeWhisperer memorize and emit hard-coded credentials verbatim during code completion?
• RQ2 How effective is prompt engineering using real code contexts in triggering memorization of secrets in NCCTs?
• RQ3 What proportion of extracted secrets are real, active credentials rather than synthetic or invalid strings?
• RQ4 To what extent do different secret types (e.g., API keys, OAuth tokens) vary in their memorization and leakage risk?
• RQ5 What mitigation strategies are feasible and effective for preventing secret leakage in NCCTs without modifying the model architecture?

Key findings

• HCR successfully extracted 2,702 valid credentials from GitHub Copilot across 900 test prompts, with 3.6% of them being real, active secrets.
• Amazon CodeWhisperer returned 129 valid secrets, of which 5.4% were real and active credentials, indicating a significant memorization risk.
• Two operational credentials were identified—specifically, two Stripe test keys that were verified as valid and active in the Stripe API environment.
• The study confirms that NCCTs trained on public code repositories containing secrets can memorize and leak them verbatim under appropriate prompting.
• The attack is effective even in a black-box setting, requiring no access to model weights or internal parameters.
• The results highlight a critical privacy vulnerability in widely used AI coding assistants, emphasizing the need for improved data curation and output filtering in NCCTs.