[Paper Review] Thieves of Sesame Street: Model Extraction on BERT-based APIs

Kalpesh Krishna, Gaurav Singh Tomar+3 more|arXiv (Cornell University)|Jan 1, 2020

Adversarial Robustness in Machine LearningComputer Science42 references32 citations

TL;DR

This paper demonstrates that adversaries can extract BERT-based NLP models via query-only access by using random word sequences and task-specific heuristics, achieving near-victim model performance at low cost—highlighting a critical vulnerability in transfer learning. It further evaluates defenses like membership classification and watermarking, showing they can be circumvented by advanced attackers.

ABSTRACT

We study the problem of model extraction in natural language processing, in which an adversary with only query access to a victim model attempts to reconstruct a local copy of that model. Assuming that both the adversary and victim model fine-tune a large pretrained language model such as BERT (Devlin et al., 2019), we show that the adversary does not need any real training data to successfully mount the attack. In fact, the attacker need not even use grammatical or semantically meaningful queries: we show that random sequences of words coupled with task-specific heuristics form effective queries for model extraction on a diverse set of NLP tasks including natural language inference and question answering. Our work thus highlights an exploit only made feasible by the shift towards transfer learning methods within the NLP community: for a query budget of a few hundred dollars, an attacker can extract a model that performs only slightly worse than the victim model. Finally, we study two defense strategies against model extraction—membership classification and API watermarking—which while successful against some adversaries can also be circumvented by more clever ones.

Motivation & Objective

• To investigate the feasibility of extracting BERT-based models using only query access to an API.
• To assess whether real training data is necessary for successful model extraction in NLP.
• To evaluate the effectiveness of defense mechanisms such as membership classification and API watermarking.
• To understand how transfer learning in NLP enables new attack surfaces for model extraction.

Proposed method

• The adversary generates random word sequences as inputs to the victim API, bypassing the need for semantically meaningful or grammatically correct queries.
• Task-specific heuristics are applied to guide query selection based on expected model outputs, improving extraction efficiency.
• The adversary collects input-output pairs from the victim model to train a local surrogate model that mimics its behavior.
• The extracted model is evaluated on downstream NLP tasks such as natural language inference and question answering.
• Two defense strategies are tested: membership classification to detect extraction attempts and API watermarking to trace model copies.
• The attack is evaluated under a budget constraint of a few hundred dollars, simulating realistic resource limits.

Experimental results

Research questions

• RQ1 Can a query-only adversary extract a BERT-based model without access to real training data or meaningful input sequences?
• RQ2 How effective are random word sequences as queries in extracting high-performing NLP models?
• RQ3 To what extent do membership classification and watermarking defenses prevent model extraction attacks?
• RQ4 Can advanced adversaries circumvent existing defense mechanisms in model extraction attacks?

Key findings

• Random word sequences, when combined with task-specific heuristics, enable highly effective model extraction on diverse NLP tasks.
• The extracted models achieve performance within a few percentage points of the victim model, even without real training data.
• Model extraction is feasible for under a few hundred dollars in query costs, making it a practical threat.
• Membership classification and watermarking defenses can be circumvented by more sophisticated adversaries.
• The vulnerability arises primarily from the widespread use of transfer learning and fine-tuned BERT models in production APIs.
• The results underscore a critical security gap in NLP systems relying on query-based access to powerful language models.