[Paper Review] Newton Method for Sparse Logistic Regression: Quadratic Convergence and Extensive Simulations

Rui Wang, Naihua Xiu+1 more|arXiv (Cornell University)|Jan 9, 2019

Sparse and Compressive Sensing Techniques43 references4 citations

TL;DR

This paper proposes a Newton method for sparsity-constrained logistic regression, establishing a first-order optimality condition via a τ-stationary point and solving it efficiently through Newton iteration. The method achieves global and quadratic convergence with low computational complexity, outperforming seven state-of-the-art solvers in numerical experiments on synthetic and real-world data.

ABSTRACT

Sparse logistic regression, {as an effective tool of classification,} has been developed tremendously in recent two decades, from its origination the $\ell_1$-regularized version to the sparsity constrained models. This paper is carried out on the sparsity constrained logistic regression by the Newton method. We begin with establishing its first-order optimality condition associated with a $ au$-stationary point. This point can be equivalently interpreted as an equation system which is then efficiently solved by the Newton method. The method has a considerably low computational complexity and enjoys global and quadratic convergence properties. Numerical experiments on random and real data demonstrate its superior performance when against seven state-of-the-art solvers.

Motivation & Objective

• To develop an efficient optimization method for sparsity-constrained logistic regression, a key tool in high-dimensional classification.
• To establish a first-order optimality condition for the sparsity-constrained problem using a τ-stationary point.
• To design a Newton-based algorithm that solves the optimality condition efficiently with low computational complexity.
• To demonstrate the method's superiority over existing state-of-the-art solvers in both synthetic and real-world datasets.
• To prove global and quadratic convergence of the proposed algorithm under the given formulation.

Proposed method

• The method formulates the sparsity-constrained logistic regression problem using a τ-stationary condition, which acts as a first-order optimality criterion.
• The τ-stationary condition is reformulated as a system of equations that can be solved using Newton's method.
• The Newton iteration is applied to this equation system, leveraging second-order information for fast convergence.
• The algorithm is designed to maintain low computational complexity by exploiting sparsity in the Hessian and gradient computations.
• Global convergence is ensured through a line search strategy, while quadratic convergence is achieved near the solution.
• The method is implemented and tested on both random and real-world datasets to evaluate performance.

Experimental results

Research questions

• RQ1 Can a Newton method be effectively applied to sparsity-constrained logistic regression with guaranteed convergence properties?
• RQ2 How does the proposed method compare in performance to seven state-of-the-art solvers on sparse classification tasks?
• RQ3 What is the computational complexity of the Newton-based approach compared to existing methods?
• RQ4 Does the method achieve global and quadratic convergence in practice for this class of problems?
• RQ5 Can the τ-stationary condition serve as a reliable and solvable optimality condition for sparse logistic regression?

Key findings

• The proposed Newton method achieves global and quadratic convergence for sparsity-constrained logistic regression.
• The method exhibits significantly lower computational complexity compared to standard approaches due to sparsity exploitation.
• On both random and real-world datasets, the method outperforms seven state-of-the-art solvers in terms of solution quality and convergence speed.
• The τ-stationary condition provides an equivalent and solvable reformulation of the first-order optimality condition for the problem.
• Numerical experiments confirm the method's robustness and efficiency across diverse data settings.
• The algorithm maintains high accuracy while achieving fast convergence, especially in high-dimensional sparse settings.