[Paper Review] Targeted Attack on GPT-Neo for the SATML Language Model Data Extraction Challenge

Ali Al-Kaswan, Maliheh Izadi+1 more|arXiv (Cornell University)|Feb 13, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper presents a two-stage targeted data extraction attack on GPT-Neo for the SATML2023 Language Model Training Data Extraction Challenge. It first uses contrastive search to maximize recall in generating candidate suffixes from 50-token prefixes, then applies an AutoSklearn-based membership inference attack to select the correct suffix with high precision. The method achieves a recall of 0.405 at a 10% false positive rate, a 34% improvement over the baseline.

ABSTRACT

Previous work has shown that Large Language Models are susceptible to so-called data extraction attacks. This allows an attacker to extract a sample that was contained in the training data, which has massive privacy implications. The construction of data extraction attacks is challenging, current attacks are quite inefficient, and there exists a significant gap in the extraction capabilities of untargeted attacks and memorization. Thus, targeted attacks are proposed, which identify if a given sample from the training data, is extractable from a model. In this work, we apply a targeted data extraction attack to the SATML2023 Language Model Training Data Extraction Challenge. We apply a two-step approach. In the first step, we maximise the recall of the model and are able to extract the suffix for 69% of the samples. In the second step, we use a classifier-based Membership Inference Attack on the generations. Our AutoSklearn classifier achieves a precision of 0.841. The full approach reaches a score of 0.405 recall at a 10% false positive rate, which is an improvement of 34% over the baseline of 0.301.

Motivation & Objective

• To address the gap in data extraction capabilities between untargeted attacks and model memorization by developing a targeted attack.
• To improve the recall of data extraction in black-box settings where shadow models and access to the base model are unavailable.
• To evaluate the effectiveness of different decoding strategies and classifier models in maximizing correct suffix recovery from GPT-Neo.
• To optimize the trade-off between recall and false positive rate in membership inference for language model data extraction.
• To demonstrate a practical, efficient attack pipeline that does not require fine-tuning or prompt-tuning.

Proposed method

• The attack uses a two-stage approach: first, GPT-Neo generates multiple candidate suffixes per 50-token prefix using contrastive search to maximize recall.
• In the second stage, a binary membership inference classifier (AutoSklearn) ranks candidate suffixes by confidence to select the most likely correct one.
• Features for the classifier are derived from sentence embeddings using TF-IDF or Sentence-Transformers, with AutoSklearn automatically selecting the best preprocessing and model pipeline.
• The attack is evaluated using recall at a 10% false positive rate, a metric that balances precision and recall in a relative error-sensitive setting.
• The method avoids fine-tuning or prompt-tuning, relying only on standard decoding and off-the-shelf ML pipelines for efficiency.
• The classifier is trained on generated suffixes and labeled based on membership in the training data, using a black-box security game framework.

Figure 2 : Rank of correct prediction (if exists)

Experimental results

Research questions

• RQ1 What decoding strategy maximizes the recall of correct suffix generation in a black-box data extraction attack on GPT-Neo?
• RQ2 How effective is an AutoSklearn-based membership inference classifier in distinguishing training data suffixes from non-members with minimal false positives?
• RQ3 Can a feature engineering pipeline combining TF-IDF and Sentence-Transformers improve classifier performance in data extraction attacks?
• RQ4 To what extent does the performance of the attack depend on the generation step versus the classification step?
• RQ5 How does the proposed attack compare to the baseline in terms of recall at a 10% false positive rate, and what is the impact of model choice and feature representation?

Key findings

• The attack achieves a recall of 0.405 at a 10% false positive rate, representing a 34% improvement over the baseline score of 0.301.
• Contrastive search was found to be the most effective decoding strategy for maximizing the number of correct candidate suffixes.
• The AutoSklearn classifier achieved a precision of 0.841, significantly outperforming other models like Logistic Regression and SVM.
• The use of Sentence-Transformers embeddings reduced dimensionality from 28,182 (TF-IDF) to 768, enabling faster training without major performance loss.
• The attack pipeline runs in approximately one hour for generation (on an RTX 3080) and a few seconds for the MIA step, making it efficient and scalable.
• The validation set score reached a recall of 0.413 at 10% false positive rate, indicating strong generalization and balanced performance.

Figure 3 : Permutation importance of features