[Paper Review] DeltaGrad: Rapid retraining of machine learning models

Yinjun Wu, Edgar Dobriban+1 more|arXiv (Cornell University)|Jun 26, 2020

Computational Physics and Python Applications36 citations

TL;DR

DeltaGrad provides a fast retraining method for models when small sets of training samples are added or removed, by caching training information and using an approximate Hessian via L-BFGS to update the model efficiently.

ABSTRACT

Machine learning models are not static and may need to be retrained on slightly changed datasets, for instance, with the addition or deletion of a set of data points. This has many applications, including privacy, robustness, bias reduction, and uncertainty quantifcation. However, it is expensive to retrain models from scratch. To address this problem, we propose the DeltaGrad algorithm for rapid retraining machine learning models based on information cached during the training phase. We provide both theoretical and empirical support for the effectiveness of DeltaGrad, and show that it compares favorably to the state of the art.

Motivation & Objective

• Motivate the need for rapid model retraining when datasets change (e.g., privacy, robustness, bias correction, uncertainty quantification).
• Develop a general retraining method applicable to empirical risk minimization solved via SGD/ GD without extra randomization.
• Provide theoretical guarantees on the accuracy of incremental updates and demonstrate empirical speedups on standard datasets.

Proposed method

• Introduce DeltaGrad that updates model parameters after small data changes using a leave-r-out gradient formulation.
• Cache gradients and parameter vectors from full-data training to approximate the new gradient via a quasi-Newton step (L-BFGS) on the increment.
• Compute exact gradients periodically (burn-in and every T0 iterations) and use historical deltas to form the approximate Hessian B_t.
• Leverage the Cauchy mean-value theorem to relate the gradient at the updated point to the gradient at the original point, enabling efficient approximation.
• Extend the approach to stochastic gradient descent (SGD) with minibatches and provide convergence guarantees under standard assumptions (strong convexity, smoothness, bounded gradients, Lipschitz Hessian).
• Provide complexity analysis showing when DeltaGrad is faster than retraining from scratch, especially for small r relative to n.

Experimental results

Research questions

• RQ1 Can DeltaGrad efficiently update models when a small number of training samples are added or removed?
• RQ2 What are the theoretical guarantees on the accuracy of DeltaGrad compared to retraining on the updated dataset?
• RQ3 How does DeltaGrad perform empirically in terms of speed and accuracy across standard datasets and model types (logistic regression, neural networks)?
• RQ4 How can DeltaGrad be extended to SGD and minibatch settings, and what are the associated convergence properties?
• RQ5 What practical applications arise from rapid retraining (privacy, robustness, debiasing, uncertainty quantification)?

Key findings

• DeltaGrad achieves substantial speed-ups (up to 6.5x in reported experiments) with negligible accuracy loss when small fractions of data are added or removed.
• The incremental updates produced by DeltaGrad closely track the true retrained parameters, with the distance to the exact solution typically an order of magnitude smaller than the baseline deviation (w^U* − w^*).
• DeltaGrad is effective across multiple datasets (MNIST, covtype, HIGGS, RCV1) and also adapts to simple neural networks and two-layer networks.
• Theoretical results show that the error ||w^U_t − w^I_t|| is o(r/n) for strongly convex objectives, implying accurate approximations as the fraction of data changed shrinks.
• In SGD settings, DeltaGrad achieves error bounds that shrink with increasing mini-batch size B, indicating accurate approximations when B is large and r/n is small.
• Experimental results include both batch and online addition/deletion, demonstrating robust speedups and consistent predictive performance.