[Paper Review] Fast Stochastic Bregman Gradient Methods: Sharp Analysis and Variance Reduction

Radu-Alexandru Dragomir, Mathieu Even+1 more|arXiv (Cornell University)|Apr 20, 2021

Stochastic Gradient Optimization Techniques40 references4 citations

TL;DR

This paper introduces fast stochastic Bregman gradient methods with sharp convergence analysis and variance reduction, showing that Bregman Stochastic Gradient Descent (BSGD) converges to the exact minimizer in the interpolation setting and achieves fast convergence via variance reduction in finite-sum problems. The proposed BSAGA algorithm matches state-of-the-art performance in tomographic reconstruction and distributed optimization, demonstrating practical efficiency despite theoretical assumptions on the reference function's regularity.

ABSTRACT

We study the problem of minimizing a relatively-smooth convex function using stochastic Bregman gradient methods. We first prove the convergence of Bregman Stochastic Gradient Descent (BSGD) to a region that depends on the noise (magnitude of the gradients) at the optimum. In particular, BSGD with a constant step-size converges to the exact minimizer when this noise is zero (\\emph{interpolation} setting, in which the data is fit perfectly). Otherwise, when the objective has a finite sum structure, we show that variance reduction can be used to counter the effect of noise. In particular, fast convergence to the exact minimizer can be obtained under additional regularity assumptions on the Bregman reference function. We illustrate the effectiveness of our approach on two key applications of relative smoothness: tomographic reconstruction with Poisson noise and statistical preconditioning for distributed optimization.

Motivation & Objective

• To provide tight convergence guarantees for Bregman Stochastic Gradient Descent (BSGD) in the context of relative smoothness and strong convexity.
• To analyze the behavior of BSGD in the interpolation setting, where noise at the optimum is zero.
• To develop and analyze a Bregman variant of the SAGA algorithm (BSAGA) with variance reduction for finite-sum problems.
• To demonstrate the practical effectiveness of variance-reduced Bregman methods in key applications such as Poisson inverse problems and distributed optimization.
• To investigate the role of higher-order regularity in the reference function h for achieving fast convergence rates.

Proposed method

• The paper uses Bregman divergences defined by a strictly convex, twice continuously differentiable reference function h to generalize standard gradient descent.
• It establishes convergence of BSGD to a neighborhood of the minimizer whose size depends on the noise magnitude at the optimum, with exact convergence in the interpolation setting.
• For finite-sum problems, the paper introduces BSAGA, a Bregman analog of SAGA, which maintains and updates gradient estimates to reduce variance.
• Theoretical analysis shows that fast convergence to the exact minimizer is achievable under additional regularity assumptions on h, such as quadratic or Lipschitz Hessian.
• The method leverages mirror descent updates via the dual variable update: ∇h(x_{t+1}) = ∇h(x_t) - η_t g_t.
• The paper analyzes the transient regime where gradient noise G_t may be large, and shows that practical performance remains strong despite conservative theoretical bounds.

Experimental results

Research questions

• RQ1 Under what conditions does BSGD converge to the exact minimizer rather than a neighborhood of it?
• RQ2 Can variance reduction techniques be effectively adapted to Bregman gradient methods to achieve fast convergence in finite-sum problems?
• RQ3 How do higher-order regularity properties of the reference function h affect convergence rates in Bregman methods?
• RQ4 What is the practical performance of variance-reduced Bregman methods in real-world applications like tomographic reconstruction and distributed optimization?
• RQ5 To what extent does the choice of reference function h impact convergence speed and stability in non-Euclidean settings?

Key findings

• BSGD converges to the exact minimizer when the noise at the optimum is zero, which corresponds to the interpolation setting.
• In the finite-sum case, variance reduction via BSAGA enables fast convergence to the exact minimizer under additional regularity assumptions on the reference function h.
• BSAGA achieves performance on par with the state-of-the-art MU algorithm for tomographic reconstruction with Poisson noise.
• In distributed optimization, BSAGA outperforms BGD and BSGD, especially after the initial phase where BSGD saturates due to variance.
• Despite theoretical conservatism, constant step-sizes are sufficient for convergence in practice, suggesting that gradient noise G_t does not impede progress as severely as theory might imply.
• The transient regime with high G_t does not significantly hinder performance in the tested applications, indicating robustness in real-world settings.