[Paper Review] Dataset Pruning: Reducing Training Data by Examining Generalization Influence

Shuo Yang, Zeke Xie+4 more|arXiv (Cornell University)|May 19, 2022

Machine Learning and Data Classification21 citations

TL;DR

Proposes an optimization-based dataset pruning method that selects a largest subset of training data whose removal keeps the generalization gap within a predefined bound, improving training efficiency.

ABSTRACT

The great success of deep learning heavily relies on increasingly larger training data, which comes at a price of huge computational and infrastructural costs. This poses crucial questions that, do all training data contribute to model's performance? How much does each individual training sample or a sub-training-set affect the model's generalization, and how to construct the smallest subset from the entire training data as a proxy training set without significantly sacrificing the model's performance? To answer these, we propose dataset pruning, an optimization-based sample selection method that can (1) examine the influence of removing a particular set of training samples on model's generalization ability with theoretical guarantee, and (2) construct the smallest subset of training data that yields strictly constrained generalization gap. The empirically observed generalization gap of dataset pruning is substantially consistent with our theoretical expectations. Furthermore, the proposed method prunes 40% training examples on the CIFAR-10 dataset, halves the convergence time with only 1.3% test accuracy decrease, which is superior to previous score-based sample selection methods.

Motivation & Objective

• Motivate and define the dataset pruning problem as reducing training data with a guaranteed generalization bound.
• Leverage influence functions to approximate parameter changes caused by removing data points.
• Formulate and solve a discrete optimization to maximize pruned data while constraining parameter change.
• Provide theoretical generalization guarantees and empirical validation across datasets and architectures.

Proposed method

• Define epsilon-redundant subset based on parameter-change norm between full- and pruned-models.
• Estimate per-sample parameter influence using Influence Functions and the Hessian inverse: I_param(z) = -H_theta^{-1} grad_theta L(z, theta).
• Approximate subset impact by aggregating influences: sum_{z in D_hat} I_param(z) and constrain its L2 norm to be <= epsilon.
• Formulate two discrete optimization problems: (a) generalization-guaranteed pruning to maximize subset size under the epsilon constraint; (b) cardinality-guaranteed pruning to minimize parameter-change given a fixed m.

Experimental results

Research questions

• RQ1 How large a subset can be removed from the training data without exceeding a predefined generalization gap?
• RQ2 Can influence-function-based approximations reliably bound the impact on generalization when removing groups of samples?
• RQ3 Do pruned datasets preserve cross-architecture performance and support neural architecture search efficiency?
• RQ4 What is the empirical trade-off between pruning ratio, generalization, and training efficiency?

Key findings

• The pruning method achieves a 40% reduction on CIFAR-10 with only 1.3% test accuracy loss.
• The observed generalization gap aligns with theoretical bounds based on epsilon, showing tight empirical consistency.
• Compared to baselines, the optimization-based pruning outperforms random, Herding, Forgetting, GraNd, EL2N, and simple influence-score methods, especially at higher pruning ratios.
• Pruned data generalizes well to unseen architectures (e.g., pruning with a small network transfers to ResNet18/50).
• Pruned datasets significantly reduce training time (e.g., nearly halved convergence time on CIFAR-10) with minimal performance loss.
• In NAS-like experiments, pruned proxy datasets yield architectures close in performance to those found using the full dataset while dramatically reducing search time and storage.