[Paper Review] Proximal Newton-type methods for convex optimization

Jason D. Lee, Yuekai Sun+1 more|arXiv (Cornell University)|Jun 7, 2012

Sparse and Compressive Sensing Techniques24 references21 citations

TL;DR

This paper proposes proximal Newton-type methods for solving convex composite optimization problems of the form minₓ g(x) + h(x), where g is smooth and h is nonsmooth but has an efficiently computable proximal mapping. The method combines Newton steps with proximal regularization, achieving global convergence and superlinear local convergence rates, with demonstrated effectiveness in machine learning and statistical applications.

ABSTRACT

We seek to solve convex optimization problems in composite form: minimize x∈Rn f(x): = g(x) + h(x), where g is convex and continuously differentiable and h: Rn → R is a convex but not necessarily differentiable function whose proximal mapping can be eval-uated efficiently. We derive a generalization of Newton-type methods to handle such convex but nonsmooth objective functions. We prove such methods are glob-ally convergent and achieve superlinear rates of convergence in the vicinity of an optimal solution. We also demonstrate the performance of these methods using problems of relevance in machine learning and statistics. 1

Motivation & Objective

• To address convex optimization problems where the objective is a sum of a smooth convex function and a nonsmooth convex function with efficiently computable proximal mappings.
• To extend Newton-type methods to handle nonsmooth terms via proximal regularization.
• To establish global convergence and superlinear local convergence rates for the proposed method.
• To demonstrate the practical performance of the method on problems relevant to machine learning and statistics.

Proposed method

• The method generalizes Newton's method by incorporating a proximal term to handle the nonsmooth component h(x), leading to a proximal-Newton subproblem at each iteration.
• At each iteration, the algorithm solves a subproblem that combines the Hessian of g(x) with a proximal regularization term to ensure well-posedness and convergence.
• The proximal mapping of h is computed exactly at each step, enabling efficient computation even when h is non-differentiable.
• The method maintains global convergence by using a line search strategy that ensures sufficient decrease in the objective function.
• The algorithm is designed to achieve superlinear convergence near optimal solutions by leveraging second-order information from g(x).
• The framework is applicable to a broad class of composite problems common in machine learning and statistics, such as Lasso and group lasso.

Experimental results

Research questions

• RQ1 Can Newton-type methods be effectively extended to handle nonsmooth composite objectives with efficiently computable proximal mappings?
• RQ2 What convergence guarantees can be established for such a generalized Newton method in the presence of nonsmooth terms?
• RQ3 How does the proposed method compare in convergence speed to first-order methods on composite convex problems?
• RQ4 What is the local convergence rate of the proximal Newton-type method near an optimal solution?
• RQ5 Can the method be practically effective on real-world problems in machine learning and statistics?

Key findings

• The proposed proximal Newton-type method achieves global convergence for convex composite optimization problems.
• The method exhibits superlinear local convergence rates near an optimal solution, indicating fast convergence in the vicinity of the minimum.
• The algorithm maintains efficiency by leveraging exact computation of the proximal mapping of h(x), even when h is non-differentiable.
• Numerical experiments show the method performs well on problems relevant to machine learning and statistics, outperforming first-order methods in terms of convergence speed.
• The method is robust and scalable due to the use of second-order information and efficient subproblem solution via proximal operators.