[Paper Review] Complexity Analysis of a Stochastic Cubic Regularisation Method under Inexact Gradient Evaluations and Dynamic Hessian Accuracy.

Stefania Bellavia, Gianmarco Gurioli|arXiv (Cornell University)|Jan 29, 2020

Sparse and Compressive Sensing Techniques27 references4 citations

TL;DR

This paper proposes a stochastic cubic regularisation method with adaptive, inexact Hessian and gradient approximations for nonconvex optimisation. It extends deterministic complexity analysis to the stochastic setting, proving convergence to a first-order stationary point in O(ε^(-3/2)) expected iterations, matching the known worst-case optimal complexity under inexactness and bounded error variance.

ABSTRACT

We here adapt an extended version of the adaptive cubic regularisation method with dynamic inexact Hessian information for nonconvex optimisation in [2] to the stochastic optimisation setting. While exact function evaluations are still considered, this novel variant inherits the innovative use of adaptive accuracy requirements for Hessian approximations introduced in [2] and additionally employs inexact computations of the gradient. Without restrictions on the variance of the errors, we assume that these approximations are available within a sufficiently large, but fixed, probability and we extend, in the spirit of [13], the deterministic analysis of the framework to its stochastic counterpart, showing that the expected number of iterations to reach a first-order stationary point matches the well known worst-case optimal complexity. This is, in fact, still given by O(epsilon^(-3/2)), with respect to the first-order epsilon tolerance.

Motivation & Objective

• To extend the adaptive cubic regularisation framework with dynamic Hessian accuracy to the stochastic optimisation setting.
• To address the challenge of inexact gradient evaluations in nonconvex optimisation without restricting error variance.
• To maintain optimal worst-case iteration complexity under inexactness and probabilistic approximation guarantees.
• To generalise the deterministic complexity analysis of [2] to a stochastic setting with inexact gradients and Hessian approximations.
• To establish that the expected number of iterations to reach a first-order stationary point matches the known optimal complexity bound.

Proposed method

• Adapts the adaptive cubic regularisation method from [2] to stochastic optimisation by incorporating inexact gradient evaluations.
• Introduces dynamic accuracy requirements for Hessian approximations, adjusting precision based on progress toward optimality.
• Assumes gradient approximations are available with a fixed, sufficiently large probability, without variance restrictions.
• Extends the deterministic complexity framework from [13] to the stochastic case, maintaining theoretical guarantees.
• Uses probabilistic bounds on approximation errors to ensure convergence under inexactness.
• Employs a trust-region-like mechanism with cubic regularisation to control step size and ensure sufficient decrease in the objective.

Experimental results

Research questions

• RQ1 Can the adaptive cubic regularisation method with dynamic Hessian accuracy be extended to the stochastic setting with inexact gradients?
• RQ2 What is the expected iteration complexity of the proposed stochastic method under inexact gradient and Hessian approximations?
• RQ3 Does the method preserve the optimal O(ε^(-3/2)) complexity bound in expectation despite inexactness?
• RQ4 How do probabilistic guarantees on approximation quality affect convergence and complexity?
• RQ5 Can the deterministic complexity analysis be rigorously extended to the stochastic case with inexact gradients?

Key findings

• The proposed stochastic cubic regularisation method achieves an expected iteration complexity of O(ε^(-3/2)) to reach a first-order stationary point.
• The complexity bound matches the well-known worst-case optimal complexity for nonconvex optimisation, even under inexact gradient and Hessian evaluations.
• The method maintains adaptive accuracy requirements for Hessian approximations, which are dynamically adjusted based on progress.
• The analysis holds without restrictions on the variance of gradient approximation errors, relying only on a fixed, sufficiently large probability of approximation availability.
• The framework successfully generalises the deterministic complexity analysis of [2] and [13] to the stochastic setting with inexactness.
• The expected number of iterations scales optimally with respect to the first-order tolerance ε, confirming robustness under inexactness.