[Paper Review] Methods of Nonconvex Optimization

V. S. Mikhalevich, A. M. Gupal+1 more|arXiv (Cornell University)|Jun 14, 2024

Advanced Optimization Algorithms Research40 citations

TL;DR

A comprehensive overview of finite-dimensional nonconvex nonsmooth optimization, introducing generalized differentiable functions and numerical methods that extend subgradient techniques to nonconvex settings, including stochastic and randomized approaches.

ABSTRACT

This book is devoted to finite-dimensional problems of non-convex non-smooth optimization and numerical methods for their solution. The problem of nonconvexity is studied in the book on two main models of nonconvex dependencies: these are the so-called generalized differentiable functions and locally Lipschitz functions. Non-smooth functions naturally arise in various applications. In addition, they often appear in the theory of extremal problems itself due to the operations of taking the maximum and minimum, decomposition techniques, exact non-smooth penalties, and duality. The considered models of nonconvexity are quite general and cover the majority of practically important optimization problems; they clearly show all the difficulties of non-convex optimization. The method of studying the generalized differentiable functions is that for these functions a generalization of the concept of gradient is introduced, a calculus is constructed, and various properties of nonconvex problems are studied in terms of generalized gradients. As for numerical methods, it is possible to extend the theory and algorithms of subgradient descent of convex optimization to problems with generalized differentiable functions. Methods for solving Lipschitz problems are characterized by the fact that the original functions are approximated by smoothed ones and iterative minimization procedures are applied to them. With this approach, it is possible to approximate the gradients of smoothed functions by stochastic finite differences and thus to construct methods without calculating gradients. A similar approach can be justified in generalized differentiable and Lipschitz stochastic programming. In these cases, various generalizations of the classical stochastic approximation and stochastic quasi-gradient method are obtained for solving constrained nonconvex nonsmooth stochastic programming problems.

Motivation & Objective

• Motivate and formalize the study of finite-dimensional nonconvex nonsmooth optimization.
• Introduce generalized differentiable functions as a broad framework for nonconvex analysis and calculus.
• Develop and survey numerical methods that extend gradient-based techniques to nonconvex nonsmooth problems.

Proposed method

• Define generalized differentiable functions and generalized gradients (pseudogradients) with an expansion f(y)=f(x)+<g,y−x>+o(x,y,g).
• Show local Lipschitz continuity and calculus rules for generalized gradients, including chain rule and max/min operations.
• Develop finite-difference and stochastic gradient methods that do not require explicit gradients, via random directions and averaging strategies.
• Extend gradient-descent-type methods to nonconvex nonsmooth problems with constraints and relaxation schemes.
• Introduce smoothing techniques and relaxation concepts to construct and analyze nonconvex optimization algorithms.
• Explore stochastic extensions, including stochastic generalized gradients and averaging procedures, for Lipschitz and generalized differentiable functions.

Experimental results

Research questions

• RQ1 How can nonconvex nonsmooth optimization be studied within a unified framework that generalizes gradients?
• RQ2 What numerical methods can minimize generalized differentiable functions without requiring exact gradients?
• RQ3 How do stochastic and randomized techniques extend nonconvex optimization algorithms to uncertain settings?
• RQ4 Under what conditions do generalized gradient methods converge in nonconvex contexts with constraints?
• RQ5 How can global optimization aspects be integrated with local generalized gradient methods?

Key findings

• Generalized differentiable functions provide a broad, locally Lipschitz framework that includes continuously differentiable, convex, and semismooth functions.
• A generalized gradient (pseudogradient) calculus enables expansion and convergence analysis similar to classical gradients for nonconvex nonsmooth problems.
• Randomized finite-difference directions can approximate generalized gradients and lead to stationary points without explicit gradient computation.
• Averaging and nonmonotone descent directions (heavy-ball and gully-step) yield anti-gullies and improved convergence behavior in nonconvex settings.
• Relaxation, smoothing, and penalty-based methods extend nonconvex nonsmooth optimization to constrained problems.
• Stochastic generalized gradients and averaging techniques provide convergence frameworks for stochastic and Lipschitz optimization problems.