[Paper Review] Adaptive Regularization for Nonconvex Optimization Using Inexact Function Values and Randomly Perturbed Derivatives

Stefania Bellavia, Gianmarco Gurioli+2 more|arXiv (Cornell University)|May 10, 2020

Sparse and Compressive Sensing Techniques28 references4 citations

TL;DR

This paper proposes an adaptive regularization algorithm for nonconvex optimization that handles inexact function values and randomly perturbed derivatives, achieving optimal evaluation complexity bounds in expectation. It proves that the algorithm computes approximate $q$-th order critical points in $O(\epsilon^{-(p+1)/(p-q+1)})$ evaluations for $q \leq 2$, and $O(\epsilon^{-q(p+1)/p})$ for $q > 2$, with bounds that are sharp in the order of accuracy tolerances.

ABSTRACT

A regularization algorithm allowing random noise in derivatives and inexact function values is proposed for computing approximate local critical points of any order for smooth unconstrained optimization problems. For an objective function with Lipschitz continuous $p$-th derivative and given an arbitrary optimality order $q \leq p$, it is shown that this algorithm will, in expectation, compute such a point in at most $O\left(\left(\min_{j\in\{1,\ldots,q\}}ε_j ight)^{-\frac{p+1}{p-q+1}} ight)$ inexact evaluations of $f$ and its derivatives whenever $q\in\{1,2\}$, where $ε_j$ is the tolerance for $j$th order accuracy. This bound becomes at most $O\left(\left(\min_{j\in\{1,\ldots,q\}}ε_j ight)^{-\frac{q(p+1)}{p}} ight)$ inexact evaluations if $q>2$ and all derivatives are Lipschitz continuous. Moreover these bounds are sharp in the order of the accuracy tolerances. An extension to convexly constrained problems is also outlined.

Motivation & Objective

• To develop a regularization algorithm that computes approximate local critical points of arbitrary order for smooth nonconvex optimization problems under inexact function and derivative evaluations.
• To analyze the evaluation complexity of such an algorithm when function values and derivatives are subject to random noise and inexactness.
• To establish tight worst-case complexity bounds in expectation that match the known bounds for exact evaluations, under mild assumptions on the noise and derivative accuracy.
• To extend the framework to convexly constrained optimization problems while preserving complexity guarantees.

Proposed method

• The algorithm uses adaptive regularization with a trust-region-like framework, where model accuracy is adjusted based on inexact function and derivative information.
• It defines a stochastic stopping criterion based on randomized derivative approximations, ensuring feasibility and convergence under bounded noise.
• The method relies on a new definition of the model improvement function $\phi_{f,j}^{\delta_j}(x)$ that accounts for inexactness and bounds the worst-case decrease in the model.
• It employs a line search or trust-region step selection strategy that maintains feasibility and ensures sufficient decrease in the model, even with noisy derivatives.
• Theoretical analysis uses a modified version of the inexact adaptive regularization (IAR) framework, with assumptions on the Lipschitz continuity of the $p$-th derivative and bounded noise in derivatives and function values.
• For convexly constrained problems, the algorithm enforces feasibility at each iterate and modifies the stopping criterion to reflect the constrained nature of the iterates.

Experimental results

Research questions

• RQ1 What is the evaluation complexity of an adaptive regularization algorithm when function values and derivatives are inexact and subject to random noise?
• RQ2 Can the algorithm achieve complexity bounds that match those of exact algorithms in expectation, despite noisy derivative and function evaluations?
• RQ3 How does the complexity depend on the order $q$ of the critical point sought and the smoothness $p$ of the objective function?
• RQ4 Can the theoretical complexity bounds be extended to convexly constrained optimization problems with inexact data?
• RQ5 Are the derived complexity bounds sharp in the order of the accuracy tolerance $\epsilon$?

Key findings

• The algorithm computes an $(\epsilon,\delta)$-approximate $q$-th order minimizer in at most $O(\epsilon^{-(p+1)/(p-q+1)})$ inexact evaluations in expectation when $q \in \{1,2\}$, matching the known complexity for exact evaluations.
• For $q > 2$, the expected number of evaluations is bounded by $O(\epsilon^{-q(p+1)/p})$, which is also sharp in the order of $\epsilon$.
• The complexity bounds are optimal in the sense that they match the best-known bounds for exact derivative and function evaluations, even under random perturbations.
• The analysis shows that the algorithm maintains the same order of complexity as the exact case, provided the probability of accurate derivative approximations is sufficiently high.
• The extension to convexly constrained problems preserves the same complexity bounds, with modifications to the stopping criterion and feasibility enforcement.
• The results hold under mild assumptions: Lipschitz continuity of the $p$-th derivative and bounded, independent noise in function and derivative evaluations.