[Paper Review] Deterministic Nonsmooth Nonconvex Optimization

Michael I. Jordan, Guy Kornowski+3 more|arXiv (Cornell University)|Feb 16, 2023

Stochastic Gradient Optimization Techniques4 citations

TL;DR

This paper resolves the open question of whether deterministic algorithms can achieve dimension-free complexity in nonsmooth nonconvex optimization by proving a lower bound of $Ω(d)$ for any deterministic first-order algorithm, showing randomization is necessary. It further establishes that zeroth-order oracle access is essential for finite-time convergence in deterministic settings, and proposes a novel white-box deterministic smoothing for ReLU networks that enables dimension-free optimization, circumventing the lower bound in practice.

ABSTRACT

We study the complexity of optimizing nonsmooth nonconvex Lipschitz functions by producing $(δ,ε)$-stationary points. Several recent works have presented randomized algorithms that produce such points using $ ilde O(δ^{-1}ε^{-3})$ first-order oracle calls, independent of the dimension $d$. It has been an open problem as to whether a similar result can be obtained via a deterministic algorithm. We resolve this open problem, showing that randomization is necessary to obtain a dimension-free rate. In particular, we prove a lower bound of $Ω(d)$ for any deterministic algorithm. Moreover, we show that unlike smooth or convex optimization, access to function values is required for any deterministic algorithm to halt within any finite time. On the other hand, we prove that if the function is even slightly smooth, then the dimension-free rate of $ ilde O(δ^{-1}ε^{-3})$ can be obtained by a deterministic algorithm with merely a logarithmic dependence on the smoothness parameter. Motivated by these findings, we turn to study the complexity of deterministically smoothing Lipschitz functions. Though there are efficient black-box randomized smoothings, we start by showing that no such deterministic procedure can smooth functions in a meaningful manner, resolving an open question. We then bypass this impossibility result for the structured case of ReLU neural networks. To that end, in a practical white-box setting in which the optimizer is granted access to the network's architecture, we propose a simple, dimension-free, deterministic smoothing that provably preserves $(δ,ε)$-stationary points. Our method applies to a variety of architectures of arbitrary depth, including ResNets and ConvNets. Combined with our algorithm, this yields the first deterministic dimension-free algorithm for optimizing ReLU networks, circumventing our lower bound.

Motivation & Objective

• To determine whether deterministic algorithms can achieve dimension-free complexity in optimizing nonsmooth nonconvex Lipschitz functions.
• To investigate the necessity of randomization and the role of zeroth-order oracle access in deterministic algorithms for this class of problems.
• To develop a deterministic smoothing method for ReLU neural networks that preserves $(\delta,\epsilon)$-Goldstein stationary points and enables dimension-free optimization.
• To resolve the open question of whether efficient deterministic black-box smoothings exist for Lipschitz functions.

Proposed method

• Proves a dimension-dependent lower bound of $\Omega(d)$ for any deterministic first-order algorithm, demonstrating that randomization is necessary to achieve dimension-free rates.
• Establishes that deterministic algorithms using only gradient oracles cannot guarantee finite-time convergence to $(\delta,\epsilon)$-Goldstein stationary points, requiring function value access (zeroth-order oracle).
• Introduces a deterministic smoothing procedure for ReLU networks in a white-box setting, leveraging architectural knowledge to preserve $(\delta,\epsilon)$-Goldstein stationarity.
• Analyzes the complexity of smoothing by proving that no efficient black-box deterministic smoothing can provide meaningful guarantees for general Lipschitz functions.
• Combines the proposed deterministic smoothing with a dimension-free algorithm for slightly-smooth functions, achieving $\widetilde{O}(\delta^{-1}\epsilon^{-3})$ oracle complexity with logarithmic smoothness dependence.
• Uses inductive analysis and recursive bounds on subgradient norms and neighborhood radii to establish convergence guarantees under the proposed smoothing.

Experimental results

Research questions

• RQ1 Is it possible to achieve a dimension-free rate in deterministic nonsmooth nonconvex optimization?
• RQ2 Does the use of a zeroth-order oracle enable finite-time convergence in deterministic algorithms for this problem?
• RQ3 Can a deterministic black-box smoothing procedure meaningfully smooth any Lipschitz function?
• RQ4 Can a deterministic, dimension-free algorithm be constructed for ReLU neural networks using architectural knowledge?
• RQ5 What is the trade-off between smoothness and dimension dependence in deterministic optimization of nonsmooth nonconvex functions?

Key findings

• Any deterministic first-order algorithm for finding $(\delta,\epsilon)$-Goldstein stationary points must incur a lower bound of $\Omega(d)$ oracle calls, proving that randomization is necessary for dimension-free complexity.
• Deterministic algorithms cannot guarantee finite-time convergence to $(\delta,\epsilon)$-Goldstein stationary points without access to function values, establishing the necessity of a zeroth-order oracle.
• For slightly-smooth functions, a deterministic algorithm achieves the optimal dimension-free rate of $\widetilde{O}(\delta^{-1}\epsilon^{-3})$ with only logarithmic dependence on the smoothness parameter.
• No efficient deterministic black-box smoothing procedure can meaningfully smooth general Lipschitz functions, resolving an open question in the literature.
• A novel white-box deterministic smoothing is proposed for ReLU networks of arbitrary depth, including ResNets and ConvNets, that preserves $(\delta,\epsilon)$-Goldstein stationarity.
• Combining the proposed white-box smoothing with the slightly-smooth algorithm yields the first deterministic, dimension-free algorithm for optimizing ReLU networks, circumventing the general lower bound.