[Paper Review] Un-regularizing: approximate proximal point and faster stochastic algorithms for empirical risk minimization

Roy Frostig, Rong Ge+2 more|arXiv (Cornell University)|Jun 24, 2015

Sparse and Compressive Sensing TechniquesEngineering17 references68 citations

TL;DR

This paper introduces a novel framework that accelerates stochastic algorithms for empirical risk minimization (ERM) by using approximate proximal point methods to reduce the original problem to solving regularized subproblems with large strong convexity. By applying fast stochastic solvers to these subproblems and leveraging a black-box reduction, the method achieves accelerated convergence rates without incurring the bias typically introduced by regularization, significantly improving running time across diverse problem settings.

ABSTRACT

We develop a family of accelerated stochastic algorithms that minimize sums of convex functions. Our algorithms improve upon the fastest running time for empirical risk minimization (ERM), and in particular linear least-squares regression, across a wide range of problem settings. To achieve this, we establish a framework based on the classical proximal point algorithm. Namely, we provide several algorithms that reduce the minimization of a strongly convex function to approximate minimizations of regularizations of the function. Using these results, we accelerate recent fast stochastic algorithms in a black-box fashion. Empirically, we demonstrate that the resulting algorithms exhibit notions of stability that are advantageous in practice. Both in theory and in practice, the provided algorithms reap the computational benefits of adding a large strongly convex regularization term, without incurring a corresponding bias to the original problem.

Motivation & Objective

• To address the suboptimal convergence rates of existing stochastic algorithms for ERM, particularly their dependence on the condition number.
• To bridge the gap between algorithms that solve regularized ERM with small regularization (for acceleration) and those that solve unregularized ERM (for unbiased solutions).
• To develop a black-box reduction that enables acceleration for unregularized ERM by solving only regularized subproblems with large regularization.
• To improve both theoretical running time guarantees and practical stability in high-dimensional, poorly conditioned problems.

Proposed method

• The framework uses an approximate variant of the classical proximal point algorithm (PPA) to iteratively minimize the original objective by solving regularized subproblems with large strong convexity.
• It reduces the minimization of a μ-strongly convex function f to approximate minimizations of f(x) + (λ/2)‖x−x₀‖², where λ ≥ 2μ.
• The method provides a theoretical bound on the required accuracy of the inner minimization procedure, ensuring that the outer loop converges linearly with only a polylogarithmic overhead in the condition number.
• It enables acceleration by combining fast stochastic solvers (e.g., SVRG, SDCA) as inner minimizers with a novel outer loop that re-centers the proximal term over iterations.
• The approach allows for larger regularization in the subproblems, which enhances numerical stability and convergence speed without introducing significant bias to the original solution.
• Dual ascent methods are used to further improve practical performance when the inner minimizer is implemented via duality.

Experimental results

Research questions

• RQ1 Can we achieve accelerated convergence for unregularized ERM without incurring the bias of explicit regularization?
• RQ2 How can we reduce the dependence of stochastic ERM algorithms on the problem’s condition number?
• RQ3 What approximation accuracy is required in the inner minimization to ensure linear convergence of the outer loop?
• RQ4 Can we use large regularization in subproblems to stabilize and accelerate convergence while still recovering an accurate solution to the original problem?
• RQ5 How does the proposed framework compare in practice to standard algorithms like SGD, SVRG, and SDCA in terms of stability and convergence speed?

Key findings

• The proposed accelerated approximate proximal point algorithm achieves a convergence rate of O(1/c) for the original ERM problem, with a runtime overhead of O(√(⌈λ/μ⌉)polylog(λ/μ)) compared to solving the regularized subproblem.
• The method enables acceleration for ERM under strong convexity and smoothness assumptions, improving upon prior algorithms that either had suboptimal condition number dependence or required small regularization.
• Empirical results show that Dual APPA (the proposed algorithm) exhibits superior stability and convergence behavior across different datasets (MNIST, CIFAR, Protein), especially under poor hyperparameter choices.
• SDCA and APPA degrade more gracefully with increasing λ compared to SGD and SVRG, which exhibit sharp divergence when λ is overestimated.
• The final objective values achieved by APPA are consistently lower than those of SDCA and SGD, even when SDCA is run to convergence, indicating that re-centering the proximal term improves solution quality.
• The framework achieves faster convergence than state-of-the-art methods across a wide range of problem settings, including linear least-squares and logistic regression, by effectively leveraging large regularization in subproblems.