[Paper Review] Stochastic DCA for minimizing a large sum of DC functions with application to Multi-class Logistic Regression

Hoai An Le Thi, Hoai Minh Le+2 more|arXiv (Cornell University)|Nov 10, 2019

Stochastic Gradient Optimization Techniques51 references32 citations

TL;DR

The paper introduces stochastic and inexact stochastic DC algorithms (SDCA and ISDCA) for minimizing large sums of DC functions, proves their convergence to critical points, and applies them to group-variable selection in multi-class logistic regression with strong experimental results.

ABSTRACT

We consider the large sum of DC (Difference of Convex) functions minimization problem which appear in several different areas, especially in stochastic optimization and machine learning. Two DCA (DC Algorithm) based algorithms are proposed: stochastic DCA and inexact stochastic DCA. We prove that the convergence of both algorithms to a critical point is guaranteed with probability one. Furthermore, we develop our stochastic DCA for solving an important problem in multi-task learning, namely group variables selection in multi class logistic regression. The corresponding stochastic DCA is very inexpensive, all computations are explicit. Numerical experiments on several benchmark datasets and synthetic datasets illustrate the efficiency of our algorithms and their superiority over existing methods, with respect to classification accuracy, sparsity of solution as well as running time.

Motivation & Objective

• Address the large-scale problem of minimizing a large sum of DC functions F(x) = (1/n) sum_i F_i(x).
• Develop SDCA and ISDCA to reduce computation by updating only a subset of DC components per iteration.
• Establish convergence guarantees (almost sure convergence to a critical point) for both SDCA and ISDCA.
• Apply the proposed methods to group variable selection in multi-class logistic regression and demonstrate efficiency on real and synthetic datasets.

Proposed method

• Formulate each F_i as a DC function F_i = g_i - h_i and aggregate to F = G - H with G = (1/n) sum g_i and H = (1/n) sum h_i.
• Propose SDCA by updating the minorants of only a random subset of h_i at each iteration and solving the convex subproblem (8).
• Define the convex subproblem as min_x { G(x) - <v^l, x> } where v^l ∈ ∂H(x^l).
• Prove almost sure convergence of SDCA to a critical point under mild conditions; establish ∑ ||x^{l}-x^{l-1}||^2 < ∞ and ||x^{l}-x^{l-1}|| → 0 a.s. when ρ(h_i) > 0.
• Introduce ISDCA by allowing ε-subgradients and ε-solution computations with ∑ ε^l < ∞, preserving convergence properties.
• Demonstrate SDCA/ISDCA in the context of group-variable selection in multi-class logistic regression with nonconvex penalties approximating the ℓ_{q,0}-norm via η_α, and solve the resulting DC-programs.

Experimental results

Research questions

• RQ1 Can SDCA efficiently minimize a large sum of DC functions by updating only a subset of the components per iteration?
• RQ2 Do SDCA and its inexact variant ISDCA converge almost surely to a critical point of the large-sum DC objective?
• RQ3 How can SDCA be applied to group-variable selection in multi-class logistic regression with nonconvex ℓ_{q,0} type regularization?
• RQ4 How does the proposed approach compare to existing methods in terms of accuracy, sparsity, and running time on large-scale datasets?

Key findings

• Both SDCA and ISDCA converge almost surely to a critical point of the large-sum DC objective.
• SDCA retains convergence guarantees when only a subset of h_i components are updated and even under inexact computations (ISDCA).
• The stochastic approaches enable efficient, scalable optimization for group-variable selection in multi-class logistic regression with nonconvex penalties.
• Numerical experiments on large real and synthetic datasets show competitive classification accuracy, higher sparsity, and reduced running time compared to related methods.