[Paper Review] A Bayesian Approach To Analysing Training Data Attribution In Deep Learning

Elisa Nguyen, Minjoon Seo+1 more|arXiv (Cornell University)|May 31, 2023

Adversarial Robustness in Machine Learning4 citations

TL;DR

This paper introduces a Bayesian framework for training data attribution (TDA) in deep learning, treating model parameters and TDA scores as random variables to quantify uncertainty from stochastic training. It finds that ground-truth TDA values are often dominated by noise from model initialization and batch composition, recommending TDA use only for low-variance, high-signal pairs where estimates are reliable.

ABSTRACT

Training data attribution (TDA) techniques find influential training data for the model's prediction on the test data of interest. They approximate the impact of down- or up-weighting a particular training sample. While conceptually useful, they are hardly applicable to deep models in practice, particularly because of their sensitivity to different model initialisation. In this paper, we introduce a Bayesian perspective on the TDA task, where the learned model is treated as a Bayesian posterior and the TDA estimates as random variables. From this novel viewpoint, we observe that the influence of an individual training sample is often overshadowed by the noise stemming from model initialisation and SGD batch composition. Based on this observation, we argue that TDA can only be reliably used for explaining deep model predictions that are consistently influenced by certain training data, independent of other noise factors. Our experiments demonstrate the rarity of such noise-independent training-test data pairs but confirm their existence. We recommend that future researchers and practitioners trust TDA estimates only in such cases. Further, we find a disagreement between ground truth and estimated TDA distributions and encourage future work to study this gap. Code is provided at https://github.com/ElisaNguyen/bayesian-tda.

Motivation & Objective

• To address the unreliability of training data attribution (TDA) in deep learning due to stochastic training processes.
• To model TDA scores as random variables using Bayesian inference, capturing uncertainty from model initialization and batch composition.
• To evaluate the reliability of TDA methods by analyzing signal-to-noise ratios in ground-truth TDA values.
• To guide practitioners on when TDA can be trusted, based on variance and signal strength in attribution scores.
• To highlight the gap between estimated TDA and ground-truth distributions, urging improved evaluation standards.

Proposed method

• Treat the learned model as a Bayesian posterior p(θ|𝒟) over parameters, acknowledging that training outcomes depend on initialization and batch composition.
• Model TDA scores τ(zⱼ,z) as random variables derived from two posteriors: p(θ|𝒟) and p(θ\j|𝒟\j), representing full and reduced training sets.
• Quantify the reliability of TDA by computing the expected value 𝔼|τ| and variance Var(τ), identifying cases where signal dominates noise.
• Use subsampled datasets and gradient-based TDA methods (e.g., influence functions, TracIn) to estimate TDA scores across multiple random seeds and batches.
• Compare estimated TDA values τ′ to ground-truth TDA values τ obtained via full retraining, analyzing discrepancies.
• Analyze contributing factors to variance in TDA, including model size, dataset size, and batch composition.

Figure 1 : A Bayesian interpretation of training data attribution (TDA).

Experimental results

Research questions

• RQ1 To what extent is the ground-truth training data attribution (TDA) value τ(zⱼ,z) dominated by noise from model initialization and SGD batch composition?
• RQ2 In how many test-train pairs is the signal-to-noise ratio of TDA values sufficiently high to trust TDA estimates?
• RQ3 How do TDA estimation methods like influence functions compare to the true global change from leave-one-out retraining?
• RQ4 Why do discrepancies exist between estimated TDA distributions and ground-truth TDA distributions?
• RQ5 Under what conditions can TDA be reliably used for interpreting deep model decisions?

Key findings

• The ground-truth TDA value τ(zⱼ,z) is often dominated by noise, with √Var(τ) > 𝔼|τ| in many cases, rendering it unreliable.
• TDA methods such as influence functions model only local changes in the model, while LOO retraining induces global changes, explaining estimation gaps.
• Low-variance, high-signal TDA pairs exist but are rare, and TDA should only be trusted in such cases.
• The signal-to-noise ratio of TDA values is significantly reduced in larger models and larger datasets due to increased stochasticity.
• There is a consistent disagreement between estimated TDA distributions and ground-truth TDA distributions, indicating a need for improved evaluation metrics.
• Variance in TDA scores is primarily driven by model initialization and batch composition, not just model complexity or dataset size.

Figure 2 : Sources of randomness for Bayesian posteriors. In each case, the training starts from initialisation $\theta_{0}$ . Depending on whether $z_{j}$ is included in the training data, one has either samples from the original posterior $p(\theta|\mathcal{D})$ or from the counterfactual posterio