[Paper Review] Beyond neural scaling laws: beating power law scaling via data pruning

Ben Sorscher, Robert Geirhos+3 more|arXiv (Cornell University)|Jun 29, 2022

Anomaly Detection Techniques and Applications85 citations

TL;DR

The paper presents theory and experiments showing that data pruning can beat traditional power-law scaling of error with dataset size, achieving exponential-like gains under Pareto-optimal pruning, and introduces a self-supervised pruning metric that rivals supervised ones at scale.

ABSTRACT

Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how in theory we can break beyond power law scaling and potentially even reduce it to exponential scaling instead if we have access to a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this improved scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling in practice on ResNets trained on CIFAR-10, SVHN, and ImageNet. Next, given the importance of finding high-quality pruning metrics, we perform the first large-scale benchmarking study of ten different data pruning metrics on ImageNet. We find most existing high performing metrics scale poorly to ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metrics. Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.

Motivation & Objective

• Motivate and analyze whether pruning training data can outperform standard power-law scaling of error with dataset size.
• Develop a theoretical framework for data pruning in the student–teacher perceptron setting to predict scaling behavior.
• Empirically validate exponential-like scaling with pruned dataset size on real models and datasets.
• Benchmark existing data-pruning metrics at ImageNet scale to identify strengths/limitations.
• Propose a self-supervised pruning metric that approaches the performance of supervised metrics without labels.

Proposed method

• Formulate a margin-based pruning algorithm where a probe model ranks examples by margin and a pruned subset is trained to completion.
• Use replica-method–based analysis in the high-dimensional limit to derive ε(α_tot,f,θ) and predict optimal pruning regimes.
• Derive predictions: (a) optimal pruning strategy depends on initial data amount; (b) Pareto-optimal pruning can yield exponential scaling with pruned data size.
• Conduct large-scale experiments with ResNets and ViTs on CIFAR-10, SVHN, ImageNet, and CIFAR-10 transfer tasks to test theory.
• Benchmark eight supervised pruning metrics on ImageNet and introduce self-supervised prototypes based on clustering in SSL embeddings.
• Propose a self-supervised pruning metric (prototype distance to cluster centroids) and compare its performance to supervised metrics.

Experimental results

Research questions

• RQ1 Can data pruning beat power-law scaling of test error with respect to dataset size?
• RQ2 What pruning strategies (which examples to keep) maximize test performance for varying initial data amounts?
• RQ3 Do Pareto-optimal pruning strategies enable exponential-like scaling with pruned dataset size in practice?
• RQ4 Which pruning metrics scale to ImageNet, and how can self-supervised methods perform without labels?
• RQ5 Can pruning pre-training or fine-tuning data on foundation models retain or improve performance on downstream tasks?

Key findings

• Analytic theory predicts two regimes: with abundant data, keeping hard examples; with scarce data, keeping easy examples; and Pareto-optimal pruning can yield exponential-like scaling with pruned size.
• Empirically observed exponential-like scaling with pruned dataset size in ResNets on SVHN, CIFAR-10, and ImageNet, and ViTs on CIFAR-10.
• Most existing pruning metrics scale poorly to ImageNet, while the best require heavy compute or labels; a new self-supervised prototype metric matches supervised metrics in many settings.
• Self-supervised pruning using clustering in SSL embeddings achieves comparable performance to strong supervised metrics while remaining label-free.
• Pruning pre-training data (upstream) can match or exceed performance obtained with full pre-training when fine-tuning on downstream tasks, suggesting efficient transfer learning via data pruning.
• A large-scale ImageNet benchmark shows substantial pruning benefits and informs metric performance across methods.