[Paper Review] Accelerated Stochastic Greedy Coordinate Descent by Soft Thresholding Projection onto Simplex

Chaobing Song, Shaobo Cui+2 more|arXiv (Cornell University)|Feb 25, 2017

Sparse and Compressive Sensing Techniques3 citations

TL;DR

This paper proposes Accelerated Stochastic Greedy Coordinate Descent (ASGCD), which combines Nesterov's acceleration and stochastic optimization with a novel greedy selection rule based on ℓ1-norm square approximation. It introduces the SOTOPO algorithm for exact solution of the resulting convex subproblem, achieving the optimal O(√(1/ϵ)) convergence rate while reducing greedy selection complexity by a factor of the sample size, significantly improving performance on high-dimensional, dense, sparse-solution problems.

ABSTRACT

In this paper we study the well-known greedy coordinate descent (GCD) algorithm to solve $\ell_1$-regularized problems and improve GCD by the two popular strategies: Nesterov's acceleration and stochastic optimization. Firstly, we propose a new rule for greedy selection based on an $\ell_1$-norm square approximation which is nontrivial to solve but convex, then an efficient algorithm called "SOft ThreshOlding PrOjection (SOTOPO)" is proposed to exactly solve the $\ell_1$-regularized $\ell_1$-norm square approximation problem, which is induced by the new rule. Based on the new rule and the SOTOPO algorithm, the Nesterov's acceleration and stochastic optimization strategies are then successfully applied to the GCD algorithm. The resulted algorithm called accelerated stochastic greedy coordinate descent (ASGCD) has the optimal convergence rate $O(\sqrt{1/ε})$, meanwhile, it reduces the iteration complexity of greedy selection up to a factor of sample size. Both theoretically and empirically, we show that ASGCD has better performance for high-dimensional and dense problems with sparse solution.

Motivation & Objective

• To address the limitation of greedy coordinate descent (GCD) in achieving optimal convergence rates due to non-convexity in existing Gauss-Southwell rules.
• To develop a convex reformulation of the GCD subproblem using ℓ1-norm square approximation to enable Nesterov’s acceleration.
• To reduce the iteration complexity of greedy selection by introducing stochastic optimization, avoiding full gradient computation.
• To design an efficient algorithm that combines acceleration, stochasticity, and exact solution of the new subproblem for ℓ1-regularized problems.

Proposed method

• Proposes a new greedy selection rule based on ℓ1-norm square approximation of the smooth function f(x), replacing the traditional quadratic approximation.
• Introduces the SOft ThreshOlding PrOjection (SOTOPO) algorithm to exactly solve the resulting ℓ1-regularized ℓ1-norm square problem in O(d + |Q| log |Q|) time, where |Q| ≪ d.
• Applies Nesterov’s acceleration via a carefully designed mirror descent step to achieve the optimal O(√(1/ϵ)) convergence rate.
• Employs stochastic optimization by sampling one training example per iteration instead of computing the full gradient, reducing greedy selection cost by a factor of the sample size n.
• Combines the new selection rule, SOTOPO, and stochastic acceleration into a unified algorithm: Accelerated Stochastic Greedy Coordinate Descent (ASGCD).
• Uses a coupling argument with Bregman divergence and telescoping inequalities to prove convergence, with parameters tuned to ensure optimal rate.

Experimental results

Research questions

• RQ1 Can Nesterov’s acceleration be successfully applied to greedy coordinate descent when the subproblem is non-convex due to cardinality constraints?
• RQ2 Can a convex reformulation of the GCD subproblem via ℓ1-norm square approximation enable acceleration while preserving sparsity and efficiency?
• RQ3 Can stochastic optimization be integrated into greedy selection without full gradient computation, maintaining convergence guarantees?
• RQ4 Does the proposed ASGCD algorithm achieve the optimal O(√(1/ϵ)) convergence rate for ℓ1-regularized problems?
• RQ5 How does ASGCD compare in iteration complexity and convergence speed to state-of-the-art methods on high-dimensional, dense, sparse-solution problems?

Key findings

• ASGCD achieves the optimal convergence rate of O(√(1/ϵ)) for ℓ1-regularized problems, matching the best-known rate for first-order methods.
• The iteration complexity of greedy selection is reduced by a factor of the sample size n through stochastic sampling, avoiding full gradient computation.
• The SOTOPO algorithm solves the ℓ1-regularized ℓ1-norm square problem in O(d + |Q| log |Q|) time, outperforming prior Euclidean projection methods like SOPOPO that require O(d log d).
• For high-dimensional and dense problems with sparse solutions, ASGCD demonstrates superior performance compared to state-of-the-art methods, both theoretically and empirically.
• Theoretical analysis shows that ASGCD requires O(√(C L1 ||x*||1 / √ϵ)) iterations to achieve F(x) − F(x*) ≤ ϵ, where C grows slowly with dimension d and is upper bounded by log²(d).
• The final convergence bound is E[F(˜xS) − F(x*)] ≤ 4/(S + 3)² × (1 + (1 + 2β(b))/(2m) × C) × L||x*||₁², confirming the optimal rate and dependence on problem parameters.