[论文解读] Methods of Nonconvex Optimization

V. S. Mikhalevich, A. M. Gupal+1 more|arXiv (Cornell University)|Jun 14, 2024

Advanced Optimization Algorithms Research被引用 40

一句话总结

对有限维非凸非光滑优化的全面综述，引入广义可微函数和将子梯度技术扩展到非凸情形的数值方法，包括随机和随机化方法。

ABSTRACT

This book is devoted to finite-dimensional problems of non-convex non-smooth optimization and numerical methods for their solution. The problem of nonconvexity is studied in the book on two main models of nonconvex dependencies: these are the so-called generalized differentiable functions and locally Lipschitz functions. Non-smooth functions naturally arise in various applications. In addition, they often appear in the theory of extremal problems itself due to the operations of taking the maximum and minimum, decomposition techniques, exact non-smooth penalties, and duality. The considered models of nonconvexity are quite general and cover the majority of practically important optimization problems; they clearly show all the difficulties of non-convex optimization. The method of studying the generalized differentiable functions is that for these functions a generalization of the concept of gradient is introduced, a calculus is constructed, and various properties of nonconvex problems are studied in terms of generalized gradients. As for numerical methods, it is possible to extend the theory and algorithms of subgradient descent of convex optimization to problems with generalized differentiable functions. Methods for solving Lipschitz problems are characterized by the fact that the original functions are approximated by smoothed ones and iterative minimization procedures are applied to them. With this approach, it is possible to approximate the gradients of smoothed functions by stochastic finite differences and thus to construct methods without calculating gradients. A similar approach can be justified in generalized differentiable and Lipschitz stochastic programming. In these cases, various generalizations of the classical stochastic approximation and stochastic quasi-gradient method are obtained for solving constrained nonconvex nonsmooth stochastic programming problems.

研究动机与目标

• 激励并形式化研究有限维非凸非光滑优化。
• 将 generalized differentiable functions 作为非凸分析与微积分的广泛框架引入。
• 发展并综述将基于梯度的方法扩展到非凸非光滑问题的数值方法。

提出的方法

• 定义 generalized differentiable functions 与 generalized gradients (pseudogradients) 及其展开式 f(y)=f(x)+<g,y−x>+o(x,y,g)。
• 给出 generalized gradients 的局部 Lipschitz 连续性及微积分规则，包括链式法则以及对 max/min 运算的处理。
• 通过随机方向和平均策略，开发不需要显式梯度的有限差分与随机梯度方法。
• 将梯度下降型方法推广到带约束和松弛方案的非凸非光滑问题。
• 引入平滑技术和松弛概念来构建和分析非凸优化算法。
• 探索随机扩展，包括随机 generalized gradients 与对 Lipschitz 与 generalized differentiable functions 的平均化过程。

实验结果

研究问题

• RQ1 如何在一个统一框架中研究非凸非光滑优化，该框架推广梯度？
• RQ2 哪些数值方法可以在不需要精确梯度的情况下最小化 generalized differentiable functions？
• RQ3 随机和随机化技术如何将非凸优化算法推广到不确定情形？
• RQ4 在带约束的非凸情境下，generalized gradient 方法在何种条件下收敛？
• RQ5 如何将全局优化方面与局部 generalized gradient 方法结合？

主要发现

• Generalized differentiable functions 提供一个广义的局部 Lipschitz 框架，包含连续可微、凸和 semismooth 函数。
• generalized gradient (pseudogradient) 微积分使扩展和收敛分析在非凸非光滑问题上类似于经典梯度的方法。
• 随机有限差分方向可以近似 generalized gradients 并在不进行显式梯度计算的情况下导致驻点。
• 平均化和非单调下降方向（heavy-ball 和 gully-step）在非凸情形下产生 anti-gullies 并改善收敛行为。
• 松弛、平滑和基于惩罚的方法将非凸非光滑优化推广到带约束的问题。
• 随机 generalized gradients 与平均化技术为随机和 Lipschitz 优化问题提供收敛框架。