[Paper Review] Leave-one-out Distinguishability in Machine Learning

Jiayuan Ye, Anastasia Borovykh+2 more|arXiv (Cornell University)|Sep 29, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper introduces Leave-One-Out Distinguishability (LOOD), a novel analytical framework that quantifies how much a single training data point influences a machine learning model's output distribution using Gaussian Processes and Neural Network Gaussian Processes (NNGPs). The method enables efficient, accurate estimation of data memorization and privacy risks—validated by strong correlation with membership inference attack performance—while also allowing optimal query identification for reconstructing training data with high fidelity.

ABSTRACT

We introduce an analytical framework to quantify the changes in a machine learning algorithm's output distribution following the inclusion of a few data points in its training set, a notion we define as leave-one-out distinguishability (LOOD). This is key to measuring data **memorization** and information **leakage** as well as the **influence** of training data points in machine learning. We illustrate how our method broadens and refines existing empirical measures of memorization and privacy risks associated with training data. We use Gaussian processes to model the randomness of machine learning algorithms, and validate LOOD with extensive empirical analysis of leakage using membership inference attacks. Our analytical framework enables us to investigate the causes of leakage and where the leakage is high. For example, we analyze the influence of activation functions, on data memorization. Additionally, our method allows us to identify queries that disclose the most information about the training data in the leave-one-out setting. We illustrate how optimal queries can be used for accurate **reconstruction** of training data.

Motivation & Objective

• To develop an efficient, analytical method for measuring the influence of individual training data points on model predictions, especially in the context of data memorization and information leakage.
• To address the computational infeasibility of empirical leave-one-out retraining by providing a closed-form analytical solution using Gaussian processes.
• To identify which queries maximize information leakage about a specific training point, enabling targeted reconstruction of training data.
• To investigate how model architecture choices, such as activation functions, affect information leakage through the lens of LOOD.
• To validate the framework’s accuracy by correlating LOOD with real-world membership inference attack performance.

Proposed method

• The paper models the output distribution of machine learning models using Gaussian Processes (GPs), particularly Neural Network Gaussian Processes (NNGPs), to analytically compute the statistical divergence between models trained with and without a single data point.
• LOOD is defined as the Kullback-Leibler divergence between the predictive distributions of models trained on full and leave-one-out datasets, capturing the influence of a single data point.
• The method leverages closed-form expressions for GP posterior means and variances to compute LOOD efficiently without retraining, reducing computation by over two orders of magnitude.
• The framework uses kernel matrix approximations and matrix perturbation theory to derive bounds on LOOD, particularly under RBF and spherical NNGP kernels.
• Optimal queries that maximize LOOD are identified via gradient-based optimization, enabling high-fidelity reconstruction of training data from model outputs.
• Theoretical analysis proves that the differing data point itself is a stationary point of LOOD, and that low-rank kernel matrices imply low LOOD, indicating reduced information leakage.

Experimental results

Research questions

• RQ1 How can we analytically quantify the influence of a single training data point on a model’s output distribution without retraining?
• RQ2 Which queries maximize information leakage about a specific training data point in a leave-one-out setting?
• RQ3 How does the choice of activation function affect information leakage, as measured by LOOD?
• RQ4 To what extent can training data be reconstructed using only black-box access to model predictions and optimal queries?
• RQ5 How well does LOOD correlate with real-world privacy attacks like membership inference?

Key findings

• LOOD computed under NNGP models shows strong correlation with membership inference attack success rates on benchmark datasets, validating its accuracy in measuring privacy risks.
• The optimal query for maximizing LOOD is the differing data point itself, and this holds even when the query is slightly perturbed, indicating high sensitivity around the training point.
• LOOD remains largely unchanged even as the number of queries increases, implying that all relevant information about the differing point is concentrated in the model’s prediction at that point.
• Reconstructed images from optimized queries using LOOD show significant resemblance to original training data, even when the optimization lands on sub-optimal queries, demonstrating effective data reconstruction.
• Theoretical analysis proves that a low-rank kernel matrix implies low LOOD, suggesting that architectural choices affecting kernel rank can be used to reduce information leakage.
• Mean distance LOOD under NNGP closely matches actual prediction differences from leave-one-out retraining of deep neural networks, confirming its fidelity in measuring self-influence and memorization.