[Paper Review] Decoding Secret Memorization in Code LLMs Through Token-Level Characterization

Yuqing Nie, Chong Wang+4 more|arXiv (Cornell University)|Oct 11, 2024

Advanced Malware Detection Techniques4 citations

TL;DR

This paper proposes DeSec, a two-stage method that uses token-level probability analysis to distinguish real from hallucinated secrets in Code LLMs. By identifying four key characteristics of real secret tokens—high and stable probabilities, stronger probability advantages, and early detectability via entropy—it guides decoding to extract more real secrets with higher plausibility than prompt-based baselines.

ABSTRACT

Code Large Language Models (LLMs) have demonstrated remarkable capabilities in generating, understanding, and manipulating programming code. However, their training process inadvertently leads to the memorization of sensitive information, posing severe privacy risks. Existing studies on memorization in LLMs primarily rely on prompt engineering techniques, which suffer from limitations such as widespread hallucination and inefficient extraction of the target sensitive information. In this paper, we present a novel approach to characterize real and fake secrets generated by Code LLMs based on token probabilities. We identify four key characteristics that differentiate genuine secrets from hallucinated ones, providing insights into distinguishing real and fake secrets. To overcome the limitations of existing works, we propose DESEC, a two-stage method that leverages token-level features derived from the identified characteristics to guide the token decoding process. DESEC consists of constructing an offline token scoring model using a proxy Code LLM and employing the scoring model to guide the decoding process by reassigning token likelihoods. Through extensive experiments on four state-of-the-art Code LLMs using a diverse dataset, we demonstrate the superior performance of DESEC in achieving a higher plausible rate and extracting more real secrets compared to existing baselines. Our findings highlight the effectiveness of our token-level approach in enabling an extensive assessment of the privacy leakage risks associated with Code LLMs.

Motivation & Objective

• To address the limitations of prompt engineering in detecting memorized secrets in Code LLMs, which suffer from high hallucination and inefficient extraction.
• To identify intrinsic token-level characteristics that differentiate genuine secrets from hallucinated ones in code generation.
• To develop a method that leverages these characteristics to guide decoding and improve the accuracy and diversity of secret extraction.
• To enable a more comprehensive and reliable assessment of privacy leakage risks in Code LLMs.

Proposed method

• Identify four distinguishing characteristics of real secret tokens: (C1) stabilization at high probabilities, (C2) higher overall probability, (C3) stronger probability advantage, and (C4) early detectability via Shannon entropy.
• Construct an offline token scoring model using a proxy Code LLM to generate training data and train a model that predicts the likelihood a token belongs to a real secret.
• Apply the scoring model during online decoding to reassign token likelihoods by combining it with the original LLM probabilities, guiding the model toward real secret tokens.
• Use a masking enhancement to suppress invalid tokens (e.g., '&') that cause premature generation termination in baseline methods.
• Employ probability re-weighting via the scoring model to increase the selection probability of real secret tokens during decoding.
• Validate the method across five state-of-the-art Code LLMs using a diverse dataset of 1200 code files containing various secret types.

Experimental results

Research questions

• RQ1 What token-level characteristics distinguish real secrets from hallucinated ones in Code LLMs?
• RQ2 Can a token-level scoring model trained on proxy LLMs effectively guide decoding to extract real secrets?
• RQ3 How does DeSec compare to existing prompt-based baselines in terms of plausible rate and real secret extraction?
• RQ4 To what extent is the token scoring model generalizable across different Code LLM architectures?

Key findings

• DeSec achieves an average Plausible Rate (PR) of 44.74% across five Code LLMs, significantly outperforming HCR (10.98%) and BS-5 (23.60%).
• DeSec successfully extracts 1076 real secrets from victim models, surpassing HCR (230 real secrets) and BS-5 (845 real secrets).
• For StarCoder2-15B, the token scoring model achieves precision and F1 scores above 0.90, indicating strong performance on the proxy model.
• The method effectively mitigates premature termination by masking invalid tokens, as demonstrated in case studies where DeSec generated valid secrets while HCR and BS-5 failed or produced implausible outputs.
• The scoring model's performance on DeepSeek-Coder-6.7B is limited by memorization of non-secret examples (e.g., 'AKIDz8krbsJ5yKBZQpn74WFkmLPx3EXAMPLE'), which are incorrectly classified as real secrets.
• Case studies confirm that DeSec generates plausible and real secrets where HCR and BS-5 fail or produce implausible outputs, validating the effectiveness of probability re-weighting and scoring guidance.