[Paper Review] On the Accuracy of Influence Functions for Measuring Group Effects

Pang Wei Koh, Kai-Siang Ang+2 more|arXiv (Cornell University)|May 30, 2019

Advanced Statistical Modeling Techniques29 references52 citations

TL;DR

The paper empirically analyzes influence functions for group data removal, showing strong rank correlation with actual effects and consistent underestimation, supported by a one-step Newton approximation and two case studies (CDR and MultiNLI).

ABSTRACT

Influence functions estimate the effect of removing a training point on a model without the need to retrain. They are based on a first-order Taylor approximation that is guaranteed to be accurate for sufficiently small changes to the model, and so are commonly used to study the effect of individual points in large datasets. However, we often want to study the effects of large groups of training points, e.g., to diagnose batch effects or apportion credit between different data sources. Removing such large groups can result in significant changes to the model. Are influence functions still accurate in this setting? In this paper, we find that across many different types of groups and for a range of real-world datasets, the predicted effect (using influence functions) of a group correlates surprisingly well with its actual effect, even if the absolute and relative errors are large. Our theoretical analysis shows that such strong correlation arises only under certain settings and need not hold in general, indicating that real-world datasets have particular properties that allow the influence approximation to be accurate.

Motivation & Objective

• Motivate measuring the effect of removing large, coherent groups of training points beyond individuals.
• Assess whether influence functions remain informative for group-level perturbations in real datasets.
• Develop a theoretical framework (one-step Newton) to relate predicted and actual group effects.
• Explore practical implications of group influence on labeling functions and crowdworkers in real tasks.

Proposed method

• Define actual group effect I*_f(w) and predicted effect I_f(w) via influence functions.
• Use a first-order Taylor (influence) approximation with Hessian H and gradient g to compute I_f(w).
• Introduce the one-step Newton approximation Δθ_Nt(w) to approximate parameter change after removing w.
• Decompose error into Newton-actual and Newton-influence terms and bound them under Lipschitz assumptions (Propositions 1–2).
• Specialize analysis to self-loss and to removing copies of a single point to derive cone-like bounds (Propositions 3–4).
• Empirically evaluate on five datasets and discuss implications for data sourcing questions (CDR, MultiNLI).

Experimental results

Research questions

• RQ1 Do influence functions maintain useful correlations with the actual group effects when removing large, coherent data groups?
• RQ2 Under what conditions do influence-based predictions underestimate or fail to correlate with actual group effects?
• RQ3 Can a one-step Newton approximation explain the observed correlation and underestimation between predicted and actual group effects?
• RQ4 How do group influences relate to real-world data collection scenarios such as labeling functions and crowdworkers?

Key findings

• Influence functions correlate highly with actual group effects on test predictions, test loss, and self-loss across diverse coherent groups (Spearman 0.89–0.99).
• Predicted influence tends to underestimate the actual effect for most group removals, except when test-loss effects are negative.
• The one-step Newton approximation explains much of the relationship between predicted and actual effects and provides a bounded error view (cone constraints) under certain settings.
• Increasing regularization (larger λ) tightens the cone and improves correlation between predicted and actual effects.
• Case studies show practical insights: labeling function coverage in CDR correlates with influence on test loss, while crowdworker size in MultiNLI does not predict influence.