[Paper Review] A unified framework for Bregman proximal methods: subgradient, gradient, and accelerated gradient schemes

David H. Gutman, Javier Peña|arXiv (Cornell University)|Dec 26, 2018

Sparse and Compressive Sensing Techniques10 citations

TL;DR

This paper introduces a unified framework for analyzing Bregman proximal first-order methods in convex optimization, leveraging convex conjugate properties to prove convergence rates for subgradient, gradient, and accelerated gradient schemes. It proposes a new accelerated Bregman proximal gradient algorithm that achieves the best-known convergence rate under mild relative smoothness and triangle scaling assumptions, without requiring prior knowledge of smoothness or scaling constants.

ABSTRACT

We provide a unified framework for analyzing the convergence of Bregman proximal first-order algorithms for convex minimization. Our framework hinges on properties of the convex conjugate and gives novel proofs of the convergence rates of the Bregman proximal subgradient, Bregman proximal gradient, and a new accelerated Bregman proximal gradient algorithm under fairly general and mild assumptions. Our accelerated Bregman proximal gradient algorithm attains the best-known accelerated rate of convergence when suitable relative smoothness and triangle scaling assumptions hold. However, the algorithm requires no prior knowledge of any related smoothness or triangle scaling constants.

Motivation & Objective

• To unify the analysis of Bregman proximal first-order algorithms for convex minimization using convex conjugate properties.
• To establish convergence rates for Bregman proximal subgradient, gradient, and accelerated gradient methods under general and mild assumptions.
• To develop a new accelerated Bregman proximal gradient algorithm that achieves optimal convergence rates without prior knowledge of smoothness or triangle scaling constants.
• To provide novel convergence proofs that are simpler and more general than existing approaches.

Proposed method

• The framework is built on properties of the convex conjugate to analyze and unify convergence behavior across different Bregman proximal schemes.
• It introduces a new accelerated Bregman proximal gradient algorithm that adapts dynamically to problem structure without requiring knowledge of smoothness or triangle scaling constants.
• Convergence rates are derived using relative smoothness and triangle scaling assumptions, which are general and mild.
• The method leverages Bregman divergences to define proximal operators that generalize standard Euclidean projections.
• The analysis establishes convergence rates by exploiting duality and convex conjugate identities to bound iterates' progress.
• The framework allows for a unified treatment of subgradient, gradient, and accelerated schemes under a single theoretical umbrella.

Experimental results

Research questions

• RQ1 How can Bregman proximal subgradient, gradient, and accelerated gradient methods be analyzed under a single theoretical framework?
• RQ2 What are the minimal assumptions required to achieve convergence for Bregman proximal first-order methods?
• RQ3 Can an accelerated Bregman proximal gradient algorithm be designed that attains the best-known convergence rate without prior knowledge of smoothness or scaling constants?
• RQ4 How do convex conjugate properties enable simpler and more general convergence proofs for these methods?

Key findings

• The proposed framework provides novel, simplified proofs of convergence rates for Bregman proximal subgradient and gradient methods using convex conjugate analysis.
• The new accelerated Bregman proximal gradient algorithm achieves the best-known convergence rate under relative smoothness and triangle scaling assumptions.
• The algorithm does not require prior knowledge of smoothness or triangle scaling constants, enhancing its practical applicability.
• The convergence analysis holds under fairly general and mild assumptions, broadening the scope of applicable optimization problems.