[Paper Review] Convex Synthesis of Accelerated Gradient Algorithms for Optimization and Saddle Point Problems using Lyapunov functions

Dennis Gramlich, Christian Ebenbauer+1 more|arXiv (Cornell University)|Jun 17, 2020

Sparse and Compressive Sensing Techniques14 references4 citations

TL;DR

This paper proposes a convex synthesis framework for accelerated gradient algorithms using Lyapunov functions and semi-definite programming, enabling the design of optimal algorithms for both convex optimization and saddle-point problems. The method generalizes beyond strongly convex functions to include non-convex cases, achieving performance matching or exceeding state-of-the-art methods like the Triple Momentum Method with non-conservative, convex design conditions.

ABSTRACT

This paper considers the problem of designing accelerated gradient-based algorithms for optimization and saddle-point problems. The class of objective functions is defined by a generalized sector condition. This class of functions contains strongly convex functions with Lipschitz gradients but also non-convex functions, which allows not only to address optimization problems but also saddle-point problems. The proposed design procedure relies on a suitable class of Lyapunov functions and on convex semi-definite programming. The proposed synthesis allows the design of algorithms that reach the performance of state-of-the-art accelerated gradient methods and beyond.

Motivation & Objective

• To develop a convex synthesis procedure for accelerated gradient algorithms that is non-conservative and applicable to both convex and non-convex objective functions.
• To extend algorithm design beyond standard strongly convex functions to include non-convex functions that allow for saddle-point search, useful in constrained optimization.
• To ensure the synthesis conditions are as tight as the corresponding analysis conditions, avoiding conservatism common in prior methods.
• To incorporate structural properties of the objective function into the algorithm design for faster convergence rates.
• To unify the design of algorithms for optimization and saddle-point problems under a single framework using generalized sector conditions.

Proposed method

• The method employs a generalized sector condition to define a broad class of objective functions, including strongly convex and certain non-convex functions.
• A general class of Lyapunov functions is used to derive stability and convergence conditions for the algorithm dynamics.
• The design problem is formulated as a convex semi-definite program (SDP) by expressing the Lyapunov conditions as linear matrix inequalities (LMIs).
• The synthesis conditions are derived such that feasibility of the analysis LMIs implies feasibility of the synthesis LMIs, ensuring non-conservativeness.
• The framework allows for the incorporation of structural properties of the objective function into the algorithm parameters via the SDP formulation.
• The method leverages congruence transformations and spectral properties to ensure robustness and convergence guarantees under the generalized sector condition.

Experimental results

Research questions

• RQ1 Can a convex synthesis framework be developed for accelerated gradient methods that is non-conservative and applicable to both convex and non-convex functions?
• RQ2 How can Lyapunov functions be used to design algorithms that converge to saddle points rather than minima?
• RQ3 To what extent can the design procedure incorporate structural properties of the objective function to improve convergence rates?
• RQ4 Can the synthesis conditions be made as tight as the corresponding analysis conditions, avoiding conservatism?
• RQ5 How does the proposed method compare in performance to existing accelerated methods like the Triple Momentum Method?

Key findings

• The proposed synthesis framework achieves the same convergence rates as the Triple Momentum Method for strongly convex functions with Lipschitz gradients.
• The method allows for the design of algorithms that converge to saddle points by including non-convex functions in the generalized sector condition.
• The synthesis conditions are non-conservative: feasibility of the analysis LMIs is equivalent to feasibility of the synthesis LMIs, ensuring no loss of performance in design.
• The framework enables the incorporation of structural properties of the objective function into the algorithm design, leading to potentially faster convergence rates than generic methods.
• The method is formulated as a convex semi-definite program, making it computationally tractable and suitable for automated algorithm generation.
• The approach generalizes existing IQC and robust control-based methods by allowing broader function classes and tighter design conditions.