[Paper Review] Thieves on Sesame Street! Model Extraction of BERT-based APIs

Kalpesh Krishna, Gaurav Singh Tomar+3 more|arXiv (Cornell University)|Oct 27, 2019

Adversarial Robustness in Machine Learning50 references73 citations

TL;DR

The paper demonstrates that an attacker can effectively extract a BERT-large based model using only query outputs, even with nonsensical inputs, and analyzes defenses like membership classification and watermarking.

ABSTRACT

We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al. 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks, including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction---membership classification and API watermarking---which while successful against naive adversaries, are ineffective against more sophisticated ones.

Motivation & Objective

• Motivate and formalize the model extraction risk for NLP APIs serving BERT-based models.
• Demonstrate that attackers can replicate strong NLP models without real task data.
• Assess how pretraining and architecture choices impact extraction quality.
• Evaluate simple defenses and discuss their limitations.

Proposed method

• Define victim model g_T as a fine-tuned BERT-based system for NLP tasks.
• Use task-specific query generators (random and wiki-based) plus heuristics to produce input-output pairs without real training data.
• Fine-tune a public BERT baseline on collected (x_i, g_T(x_i)) to obtain extracted model g'_T.
• Evaluate extraction via Accuracy on the original dev sets and Agreement between victim and extracted models.
• Explore effects of output type (probabilities vs. argmax) on extraction.
• Experiment with different attacker architectures (BERT-large/base, XLNet) and training data mix (original/wiki/random).
• Test defenses: membership classification and watermarking, including unanswerable-question settings.

Experimental results

Research questions

• RQ1 Can an adversary reproduce a high-performing NLP model by querying a black-box BERT-based API with nonsensical inputs?
• RQ2 How do query types (random vs. wiki-derived) and task heuristics affect extraction accuracy and model equivalence?
• RQ3 What is the impact of attacker-pretraining and architecture mismatch on extraction success?
• RQ4 Are simple defenses like membership classification and watermarking effective against adaptive attackers?

Key findings

• Extracted models achieve high accuracy on the original development sets across tasks even when trained on nonsensical inputs.
• Wiki-based and random queries produce effective extractions, with high target-task accuracy and variable Agreement relative to the victim.
• Pretraining the attacker on more powerful models (e.g., XLNet-large) can improve extraction performance even with mismatched victim architectures.
• Matching architectures generally improves extraction, but stronger attacker models can surpass mismatches (e.g., XLNet outperforming BERT-based attackers in SQuAD).
• Defenses like membership classification can slow naive attackers but are insufficient against adaptive adversaries; watermarking can reveal theft on watermarked queries but is not a complete solution.
• Extraction remains cost-efficient (often hundreds of dollars) and can deliver near-parity with the victim model on several tasks.