[Paper Review] The Secret Sharer: Measuring Unintended Neural Network Memorization & Extracting Secrets

Nicholas Carlini, Chang Liu+3 more|arXiv (Cornell University)|Feb 22, 2018

Privacy-Preserving Technologies in DataComputer Science33 references123 citations

TL;DR

This paper introduces 'exposure,' a simple metric to measure unintended memorization of secrets in deep learning models. Using black-box API access, the authors demonstrate efficient secret extraction and show memorization occurs early, persists across models, and is not due to overfitting—highlighting the need for stronger defenses, which they validate can be achieved via differentially private training.

ABSTRACT

Machine learning models based on neural networks and deep learning are being rapidly adopted for many purposes. What those models learn, and what they may share, is a significant concern when the training data may contain secrets and the models are public -- e.g., when a model helps users compose text messages using models trained on all users' messages. This paper presents exposure: a simple-to-compute metric that can be applied to any deep learning model for measuring the memorization of secrets. Using this metric, we show how to extract those secrets efficiently using black-box API access. Further, we show that unintended memorization occurs early, is not due to over-fitting, and is a persistent issue across different types of models, hyperparameters, and training strategies. We experiment with both real-world models (e.g., a state-of-the-art translation model) and datasets (e.g., the Enron email dataset, which contains users' credit card numbers) to demonstrate both the utility of measuring exposure and the ability to extract secrets. Finally, we consider many defenses, finding some ineffective (like regularization), and others to lack guarantees. However, by instantiating our own differentially-private recurrent model, we validate that by appropriately investing in the use of state-of-the-art techniques, the problem can be resolved, with high utility.

Motivation & Objective

• To quantify unintended memorization of sensitive data in deep learning models trained on private or secret-laden datasets.
• To develop a practical, universally applicable metric—'exposure'—for measuring memorization in any deep learning model.
• To demonstrate that secrets can be efficiently extracted using only black-box API access, even in production models.
• To investigate the root causes of memorization, including its timing, persistence, and independence from overfitting.
• To evaluate existing defenses and validate that differentially private training can effectively mitigate memorization while preserving model utility.

Proposed method

• Propose 'exposure' as a metric based on the model's output probability distribution when prompted with memorized data instances.
• Use black-box query access to a model's API to iteratively probe for memorized secrets by measuring output likelihoods.
• Train models on datasets containing secrets (e.g., Enron emails with credit card numbers) to evaluate memorization across diverse architectures and training strategies.
• Apply various defenses such as weight decay, dropout, and adversarial training to assess their effectiveness in reducing exposure.
• Implement and evaluate a differentially private recurrent model to validate that privacy-preserving training can eliminate memorization with high utility.

Experimental results

Research questions

• RQ1 To what extent do deep learning models memorize sensitive information from their training data, even when the data is not intended to be learned?
• RQ2 Can secrets be extracted from a model using only black-box API access, and how efficiently?
• RQ3 Does memorization occur early in training and persist across different model architectures and hyperparameters?
• RQ4 Are common regularization techniques effective at reducing memorization, or do they fail to address the root cause?
• RQ5 Can differentially private training effectively eliminate memorization while maintaining model utility?

Key findings

• Memorization of secrets occurs early in training and persists even in models not overfit, indicating it is not a byproduct of overfitting.
• The 'exposure' metric successfully quantifies memorization across diverse models and datasets, including state-of-the-art translation models.
• Secrets such as credit card numbers from the Enron dataset can be extracted with high success rates using only black-box API queries.
• Common defenses like weight decay and dropout fail to reduce exposure, indicating they are ineffective against memorization.
• Differentially private training effectively eliminates memorization while maintaining high model utility, validating it as a viable defense.