[Paper Review] Stochastic Majorization-Minimization Algorithms for Large-Scale Optimization

Julien Mairal|arXiv (Cornell University)|Jun 19, 2013

Sparse and Compressive Sensing Techniques32 references89 citations

TL;DR

This paper introduces a stochastic majorization-minimization (SMM) algorithm for large-scale optimization, where a surrogate function is updated using single data points and minimized iteratively. It achieves an O(1/√n) convergence rate for convex problems and almost sure convergence to stationary points in non-convex settings, enabling scalable solutions for large or infinite datasets in machine learning and signal processing.

ABSTRACT

Majorization-minimization algorithms consist of iteratively minimizing a majorizing surrogate of an objective function. Because of its simplicity and its wide applicability, this principle has been very popular in statistics and in signal processing. In this paper, we intend to make this principle scalable. We introduce a stochastic majorization-minimization scheme which is able to deal with large-scale or possibly infinite data sets. When applied to convex optimization problems under suitable assumptions, we show that it achieves an expected convergence rate of $O(1/\sqrt{n})$ after $n$ iterations, and of $O(1/n)$ for strongly convex functions. Equally important, our scheme almost surely converges to stationary points for a large class of non-convex problems. We develop several efficient algorithms based on our framework. First, we propose a new stochastic proximal gradient method, which experimentally matches state-of-the-art solvers for large-scale $\ell_1$-logistic regression. Second, we develop an online DC programming algorithm for non-convex sparse estimation. Finally, we demonstrate the effectiveness of our approach for solving large-scale structured matrix factorization problems.

Motivation & Objective

• Address the scalability of majorization-minimization (MM) algorithms for large-scale or infinite datasets in machine learning and signal processing.
• Develop a stochastic variant of MM that updates surrogates using single data points, reducing memory dependence on dataset size.
• Establish theoretical convergence guarantees for both convex and non-convex optimization problems under mild assumptions.
• Design efficient algorithms based on the framework for practical applications such as ℓ1-logistic regression and structured matrix factorization.

Proposed method

• Formulate a stochastic majorization-minimization scheme that constructs a surrogate function based on a single observed data point at each iteration.
• Use first-order surrogate functions that are ρ-strongly convex, majorize the objective, and have L-Lipschitz continuous gradients of the approximation error.
• Update the surrogate function incrementally using online data points, ensuring memory complexity independent of training set size.
• Apply the scheme to composite and constrained problems via a new stochastic proximal gradient method.
• Extend the framework to non-convex problems using online DC programming for sparse estimation.
• Apply the algorithm to structured matrix factorization with flexible loss and regularization functions.

Experimental results

Research questions

• RQ1 Can majorization-minimization be made scalable to large or infinite datasets while preserving convergence guarantees?
• RQ2 What convergence rates can be achieved by a stochastic MM algorithm for convex and strongly convex problems?
• RQ3 Does the stochastic MM scheme converge almost surely to stationary points in non-convex optimization problems?
• RQ4 How does the proposed stochastic proximal gradient method compare to state-of-the-art solvers for large-scale ℓ1-logistic regression?
• RQ5 Can the framework effectively handle structured matrix factorization with complex regularization and loss functions in an online setting?

Key findings

• The stochastic MM algorithm achieves an expected convergence rate of O(1/√n) for convex problems and O(1/n) for strongly convex problems after n iterations.
• For non-convex problems, the algorithm almost surely converges to a set of stationary points under suitable assumptions.
• The proposed stochastic proximal gradient method performs competitively with state-of-the-art solvers on large-scale ℓ1-logistic regression tasks.
• The online DC programming algorithm outperforms batch alternatives in large-scale non-convex sparse estimation problems.
• The framework enables efficient online structured matrix factorization with flexible loss and regularization functions, extending prior work.
• Theoretical analysis confirms convergence under mild conditions, including uniform convergence of surrogate functions and bounded gradients of approximation errors.