[Paper Review] Towards Demystifying Membership Inference Attacks

Stacey Truex, Ling Liu+3 more|arXiv (Cornell University)|Jun 28, 2018

Adversarial Robustness in Machine LearningComputer Science35 references86 citations

TL;DR

The paper formalizes black-box membership inference attacks, builds a general attack framework using shadow datasets and shadow models, and empirically shows data-driven and transferable vulnerabilities across models and datasets, including insider risks in federated learning.

ABSTRACT

Membership inference attacks seek to infer membership of individual training instances of a model to which an adversary has black-box access through a machine learning-as-a-service API. In providing an in-depth characterization of membership privacy risks against machine learning models, this paper presents a comprehensive study towards demystifying membership inference attacks from two complimentary perspectives. First, we provide a generalized formulation of the development of a black-box membership inference attack model. Second, we characterize the importance of model choice on model vulnerability through a systematic evaluation of a variety of machine learning models and model combinations using multiple datasets. Through formal analysis and empirical evidence from extensive experimentation, we characterize under what conditions a model may be vulnerable to such black-box membership inference attacks. We show that membership inference vulnerability is data-driven and corresponding attack models are largely transferable. Though different model types display different vulnerabilities to membership inference, so do different datasets. Our empirical results additionally show that (1) using the type of target model under attack within the attack model may not increase attack effectiveness and (2) collaborative learning exposes vulnerabilities to membership inference risks when the adversary is a participant. We also discuss countermeasure and mitigation strategies.

Motivation & Objective

• Characterize membership privacy risks in machine learning services under black-box access.
• Develop a generalized attack model framework with shadow datasets and shadow models.
• Evaluate how target model type and training data affect vulnerability to membership inference attacks.
• Explore insider membership inference risks in federated learning scenarios.
• Discuss countermeasures and mitigation strategies.

Proposed method

• Formalize a general attack model for black-box membership inference as a binary classifier task.
• Introduce shadow dataset generation to mimic the target training data structure using API probing.
• Create attack model training data from shadow models to train a binary membership classifier.
• Explore ensemble approaches to shadow model generation to increase attack generality and robustness.
• Demonstrate data-driven vulnerability and transferability of attack models across target models and datasets.
• Examine insider threats in federated learning as a membership inference risk.

Experimental results

Research questions

• RQ1 Under what conditions is a model vulnerable to black-box membership inference attacks?
• RQ2 How does the target model type, training data, and attack data generation influence attack effectiveness and transferability?
• RQ3 Can shadow datasets and shadow models accurately reflect the target model behavior for effective attacks?
• RQ4 What are the risks of membership inference in federated (federated learning) settings, including insider threats?

Key findings

• Membership inference vulnerability is data-driven and attack models are largely transferable across settings.
• Different datasets and different target models produce varying vulnerability, indicating no single pattern of weakness for all cases.
• Using the type of target model within the attack model does not necessarily increase attack effectiveness.
• Collaborative or federated learning environments expose vulnerabilities to membership inference when insiders participate.
• Attack construction via shadow datasets and shadow models can be effective even with black-box access.
• Countermeasures and mitigation strategies are discussed to address these privacy risks.