[Paper Review] Newton-Type Methods for Non-Convex Optimization Under Inexact Hessian Information

Peng Xu, Farbod Roosta-Khorasani+1 more|arXiv (Cornell University)|Aug 23, 2017

Sparse and Compressive Sensing Techniques117 references54 citations

TL;DR

The paper develops trust-region and adaptive cubic regularization Newton-type methods for non-convex optimization with inexact Hessian information, establishing optimal iteration complexity under a relaxed Hessian-approximation condition and presenting sub-sampling strategies for finite-sum problems with high-probability guarantees.

ABSTRACT

We consider variants of trust-region and cubic regularization methods for non-convex optimization, in which the Hessian matrix is approximated. Under mild conditions on the inexact Hessian, and using approximate solution of the corresponding sub-problems, we provide iteration complexity to achieve $ ε$-approximate second-order optimality which have shown to be tight. Our Hessian approximation conditions constitute a major relaxation over the existing ones in the literature. Consequently, we are able to show that such mild conditions allow for the construction of the approximate Hessian through various random sampling methods. In this light, we consider the canonical problem of finite-sum minimization, provide appropriate uniform and non-uniform sub-sampling strategies to construct such Hessian approximations, and obtain optimal iteration complexity for the corresponding sub-sampled trust-region and cubic regularization methods.

Motivation & Objective

• Motivate robust Newton-type methods that operate with inexact Hessians in non-convex optimization.
• Introduce a relaxed Hessian-approximation condition that enables practical Hessian construction with guarantees.
• Analyze the convergence properties and iteration complexity to achieve approximate second-order optimality.
• Develop and assess sub-sampling strategies to construct inexact Hessians in finite-sum settings.
• Highlight advantages over prior conditions and demonstrate applicability to large-scale problems.

Proposed method

• Study trust-region and adaptive cubic regularization (ARC) frameworks under inexact Hessians.
• Impose Condition 1: the inexact Hessian satisfies a bound on (H(x_t)−∇^2F(x_t))s_t and a uniform norm bound on H(x_t).
• Allow approximate solutions to sub-problems (Cauchy and Eigenpoint conditions) to ensure descent.
• Show that, under Condition 1, the worst-case iteration complexity matches that of exact variants.
• Provide a priori-guided Hessian construction via randomized numerical linear algebra (RandNLA) techniques.
• For finite-sum problems, design sub-sampling strategies to achieve the Hessian approximation with high probability.

Experimental results

Research questions

• RQ1 Can trust-region and cubic-regularization Newton-type methods converge to (ε_g, ε_H)-optimal solutions when using inexact Hessians?
• RQ2 What are the iteration complexity bounds achievable under a relaxed inexact Hessian condition compared to exact methods?
• RQ3 How can RandNLA-based subsampling yield high-probability Hessian approximations that satisfy the required inexactness conditions?
• RQ4 Do sub-sampled TR and ARC methods retain optimal second-order complexity for finite-sum problems (P1, P2)?
• RQ5 What practical advantages does the new inexact-Hessian condition offer over prior conditions in distributed or large-scale settings?

Key findings

• The paper proves that TR and ARC variants with inexact Hessians achieving Condition 1 attain the same worst-case iteration complexity as their exact counterparts for (ε_g, ε_H)-optimality.
• A weaker and more flexible Hessian-approximation condition enables practical a priori Hessian construction with RandNLA techniques.
• Sub-sampling strategies for finite-sum problems provide high-probability guarantees that the Hessian approximation satisfies the needed accuracy (and even stronger bounds).
• The analysis yields optimal iteration complexities for sub-sampled TR and ARC methods in (P1) and (P2).
• The proposed framework allows inexact Hessians to be fixed across iterations in distributed settings, reducing communication and computation overhead.