[Paper Review] Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learning

Ahmed Salem, Apratim Bhattacharya+3 more|arXiv (Cornell University)|Apr 1, 2019

Adversarial Robustness in Machine LearningComputer Science89 citations

TL;DR

The paper shows that differences in black-box model outputs before and after updates can leak information about the updating data, introducing four encoder-decoder attacks to infer or reconstruct the updating set.

ABSTRACT

Machine learning (ML) has progressed rapidly during the past decade and the major factor that drives such development is the unprecedented large-scale data. As data generation is a continuous process, this leads to ML model owners updating their models frequently with newly-collected data in an online learning scenario. In consequence, if an ML model is queried with the same set of data samples at two different points in time, it will provide different results. In this paper, we investigate whether the change in the output of a black-box ML model before and after being updated can leak information of the dataset used to perform the update, namely the updating set. This constitutes a new attack surface against black-box ML models and such information leakage may compromise the intellectual property and data privacy of the ML model owner. We propose four attacks following an encoder-decoder formulation, which allows inferring diverse information of the updating set. Our new attacks are facilitated by state-of-the-art deep learning techniques. In particular, we propose a hybrid generative model (CBM-GAN) that is based on generative adversarial networks (GANs) but includes a reconstructive loss that allows reconstructing accurate samples. Our experiments show that the proposed attacks achieve strong performance.

Motivation & Objective

• Motivate and formalize the updating set leakage risk in online learning under black-box access.
• Propose four attacks that infer properties or reconstruct the updating data from posterior differences.
• Develop encoder-decoder architectures to exploit posterior differences for various updating-set information.

Proposed method

• Formulate a general encoder-decoder attack pipeline using posterior difference as input.
• Use a shadow-model approach to generate ground-truth data for training attacks.
• Single-sample attacks include label inference and sample reconstruction.
• Multi-sample attacks include label distribution estimation and reconstruction of the updating set.
• Introduce CBM-GAN, a conditional best-of-many GAN, to reconstruct multiple updating samples.
• Evaluate attacks on MNIST, CIFAR-10, and Insta-NY with probing set of 100 samples.

Experimental results

Research questions

• RQ1 Can differences in a target model’s outputs after an update leak information about the updating set?
• RQ2 How effectively can black-box attackers infer labels or reconstruct data from the updating set using an encoder-decoder setup?
• RQ3 What is the extent of leakage for single-sample versus multi-sample updating sets?
• RQ4 Can shadow models enable realistic training of attack models under black-box constraints?
• RQ5 How well can advanced generative models reconstruct updating sets from posterior differences?

Key findings

• Single-sample label inference attack achieves accuracies of 0.97 on Insta-NY, 0.96 on CIFAR-10, and 0.68 on MNIST.
• Single-sample reconstruction attack outperforms random baselines and approaches autoencoder performance on MNIST/CIFAR-10.
• Multi-sample label distribution estimation attack reduces KL-divergence and improves accuracy over random baselines across datasets.
• CBM-GAN enables generating multiple samples for the updating set conditioned on posterior difference, outperforming baselines on MNIST, CIFAR-10, and Insta-NY.
• Attacks remain effective under shadow-model training and probing with 100-sample updating sets; some transfer relaxations are explored.
• The framework demonstrates that model output differences can leak substantial updating-set information.