[Paper Review] Efficiency of Coordinate Descent Methods For Structured Nonconvex Optimization

Qi Deng, Chenghao Lan|arXiv (Cornell University)|Sep 3, 2019

Sparse and Compressive Sensing Techniques30 references4 citations

TL;DR

This paper proposes novel coordinate descent (CD) methods for structured nonconvex optimization problems with a nonconvex objective composed of a smooth, a convex, and a concave term. It introduces randomized coordinate subgradient, randomly permuted CD, accelerated CD-based proximal DC, and proximal point methods, all with asymptotic convergence to critical points and sublinear convergence rates, achieving superior complexity for ill-conditioned problems compared to gradient methods.

ABSTRACT

Novel coordinate descent (CD) methods are proposed for minimizing nonconvex functions consisting of three terms: (i) a continuously differentiable term, (ii) a simple convex term, and (iii) a concave and continuous term. First, by extending randomized CD to nonsmooth nonconvex settings, we develop a coordinate subgradient method that randomly updates block-coordinate variables by using block composite subgradient mapping. This method converges asymptotically to critical points with proven sublinear convergence rate for certain optimality measures. Second, we develop a randomly permuted CD method with two alternating steps: linearizing the concave part and cycling through variables. We prove asymptotic convergence to critical points and sublinear complexity rate for objectives with both smooth and concave parts. Third, we extend accelerated coordinate descent (ACD) to nonsmooth and nonconvex optimization to develop a novel randomized proximal DC algorithm whereby we solve the subproblem inexactly by ACD. Convergence is guaranteed with at most a few number of ACD iterations for each DC subproblem, and convergence complexity is established for identification of some approximate critical points. Fourth, we further develop the third method to minimize certain ill-conditioned nonconvex functions: weakly convex functions with high Lipschitz constant to negative curvature ratios. We show that, under specific criteria, the ACD-based randomized method has superior complexity compared to conventional gradient methods. Finally, an empirical study on sparsity-inducing learning models demonstrates that CD methods are superior to gradient-based methods for certain large-scale problems.

Motivation & Objective

• To develop efficient coordinate descent methods for large-scale structured nonconvex optimization problems with nonconvex objectives composed of smooth, convex, and concave terms.
• To extend randomized and accelerated CD methods to nonsmooth and nonconvex settings with convergence guarantees.
• To improve iteration complexity and running time by using accelerated CD to solve subproblems inexactly in a proximal DC framework.
• To establish complexity advantages for ill-conditioned problems with high Lipschitz-to-negative-curvature ratios.
• To empirically validate the superiority of CD methods over gradient-based approaches in large-scale sparse learning models.

Proposed method

• Proposes a randomized coordinate subgradient descent (RCSD) method that updates block coordinates using composite subgradient mappings, with convergence to critical points and sublinear rate for a generalized optimality measure.
• Introduces a randomly permuted CD (RPCD) method that alternates between linearizing the concave term and cycling through block updates, ensuring asymptotic convergence and sublinear rate under certain conditions.
• Develops an accelerated proximal DC (ACPDC) algorithm by applying accelerated CD to solve subproblems in a difference-of-convex (DC) framework, requiring only a few ACD iterations per subproblem.
• Proposes an ACD-based proximal point method (ACPP) for weakly convex functions, achieving faster convergence rates when the ratio of Lipschitz constant to negative curvature is high.
• Uses strong convexity and norm-based inequalities to derive convergence bounds, leveraging expected decrease in objective function and recursive error decay.
• Employs randomized coordinate selection with non-uniform sampling and establishes convergence via Lyapunov function analysis and recursive inequalities involving step-sizes and curvature parameters.

Experimental results

Research questions

• RQ1 Can randomized coordinate descent be extended to nonsmooth and nonconvex problems with a composite structure including a concave term?
• RQ2 Can a randomly permuted CD method with linearization of the concave part achieve sublinear convergence for nonconvex problems?
• RQ3 Can accelerated coordinate descent be effectively applied to inexact subproblem solving in a DC algorithm framework for nonconvex optimization?
• RQ4 Does the ACD-based method achieve better complexity than standard gradient methods for ill-conditioned nonconvex problems?
• RQ5 Can the proposed CD methods outperform gradient-based methods in large-scale sparse learning applications?

Key findings

• The randomized coordinate subgradient method (RCSD) converges asymptotically to critical points and achieves a sublinear convergence rate for a generalized optimality measure in nonsmooth nonconvex settings.
• The randomly permuted CD (RPCD) method ensures asymptotic convergence to critical points and sublinear rate for the subgradient norm when the convex regularizer is zero.
• The ACPDC algorithm guarantees convergence with only a few accelerated CD iterations per subproblem, significantly improving efficiency over exact DC and proximal DC methods.
• For weakly convex functions with high Lipschitz-to-negative-curvature ratios, the ACPP method achieves better complexity than classic CD methods, with a complexity bound of O(√μ₀ + ∑√Li / √μ₀) log(1/ε).
• Empirical results show CD methods outperform gradient-based methods in large-scale sparsity-inducing learning models, demonstrating superior scalability and convergence speed.
• Theoretical analysis establishes that the expected objective gap decays exponentially with rate (1 - √μ₀ / (√μ₀ + T₁/₂))ᵏ, leading to O(log(1/ε)) iteration complexity for ε-accuracy.