[Paper Review] Coresets for Data-efficient Training of Machine Learning Models

Baharan Mirzasoleiman, Jeff Bilmes+1 more|arXiv (Cornell University)|Jun 5, 2019

Stochastic Gradient Optimization Techniques37 citations

TL;DR

CRAIG selects a weighted data subset (coreset) that closely approximates the full gradient, enabling incremental gradient methods to converge with the same speed as using the full data, and yielding substantial speedups in practice.

ABSTRACT

Incremental gradient (IG) methods, such as stochastic gradient descent and its variants are commonly used for large scale optimization in machine learning. Despite the sustained effort to make IG methods more data-efficient, it remains an open question how to select a training data subset that can theoretically and practically perform on par with the full dataset. Here we develop CRAIG, a method to select a weighted subset (or coreset) of training data that closely estimates the full gradient by maximizing a submodular function. We prove that applying IG to this subset is guaranteed to converge to the (near)optimal solution with the same convergence rate as that of IG for convex optimization. As a result, CRAIG achieves a speedup that is inversely proportional to the size of the subset. To our knowledge, this is the first rigorous method for data-efficient training of general machine learning models. Our extensive set of experiments show that CRAIG, while achieving practically the same solution, speeds up various IG methods by up to 6x for logistic regression and 3x for training deep neural networks.

Motivation & Objective

• Motivate data-efficient training to reduce computational and energy costs in large-scale ML.
• Develop a principled subset selection method to approximate the full gradient with a small, weighted core set.
• Provide theoretical convergence guarantees showing IG on the subset matches IG on the full data.
• Demonstrate practical speedups and applicability to both convex and non-convex models.

Proposed method

• Define an objective L(S) that bounds the gradient estimation error of subset S against the full dataset V.
• Transform the gradient-approximation objective into a monotone submodular facility-location function F and solve via a greedy algorithm.
• Compute subset weights gamma_j as the counts of components closest to each subset element in gradient space.
• Prove convergence of any IG method applied to S to the same number of epochs as full data, with an error term tied to epsilon.
• Provide practical guidelines for applying CRAIG to deep networks, including gradient-bound approximations that do not require full backpropagation.

Experimental results

Research questions

• RQ1 Can a small subset of data (with weights) approximate the full gradient well enough to preserve IG convergence behavior?
• RQ2 What is the impact of using CRAIG-selected subsets on convergence rate and final solution for convex problems?
• RQ3 Do the CRAIG subsets provide substantial speedups for SGD, SAGA, SVRG, and deep network training without sacrificing accuracy?
• RQ4 How can CRAIG be extended to deep networks where gradient bounds are harder to compute?

Key findings

• CRAIG enables IG on the subset to converge to the same solution as IG on the full data, with speedups proportional to |V|/|S|.
• For strongly convex problems, IG on the CRAIG subset converges with an error term O(epsilon) and matches the full-data rate up to constants.
• Experiments show speedups up to 6x for convex problems and up to 3x for non-convex deep networks while achieving similar loss and accuracy.
• CRAIG subsets (as small as 10% of data in some cases) can closely approximate the full gradient and outperform random subsets.
• CRAIG is compatible with SGD, SAGA, and SVRG, and yields practical performance gains on large-scale datasets like Covtype and Ijcnn1.
• In neural networks, CRAIG reduces training time with maintained or improved generalization in experiments like MNIST with a 2-layer network.