[Paper Review] Analysis of Generalized Bregman Surrogate Algorithms for Nonsmooth Nonconvex Statistical Learning

Yiyuan She, Zhifeng Wang+1 more|arXiv (Cornell University)|Dec 16, 2021

Sparse and Compressive Sensing Techniques4 citations

TL;DR

This paper proposes a generalized Bregman surrogate framework for nonsmooth, nonconvex statistical learning, enabling global convergence with geometric rates under regularity conditions. It establishes provable statistical guarantees for fixed points of the algorithm and develops adaptive momentum-based accelerations without requiring convexity or smoothness.

ABSTRACT

Modern statistical applications often involve minimizing an objective function that may be nonsmooth and/or nonconvex. This paper focuses on a broad Bregman-surrogate algorithm framework including the local linear approximation, mirror descent, iterative thresholding, DC programming and many others as particular instances. The recharacterization via generalized Bregman functions enables us to construct suitable error measures and establish global convergence rates for nonconvex and nonsmooth objectives in possibly high dimensions. For sparse learning problems with a composite objective, under some regularity conditions, the obtained estimators as the surrogate's fixed points, though not necessarily local minimizers, enjoy provable statistical guarantees, and the sequence of iterates can be shown to approach the statistical truth within the desired accuracy geometrically fast. The paper also studies how to design adaptive momentum based accelerations without assuming convexity or smoothness by carefully controlling stepsize and relaxation parameters.

Motivation & Objective

• Address the lack of universal convergence rate analysis for nonconvex, nonsmooth optimization in high-dimensional statistical learning.
• Provide global convergence guarantees for a broad class of algorithms—including LLA, iterative thresholding, and mirror descent—under a unified Bregman surrogate framework.
• Establish statistical accuracy of fixed-point estimators in composite objectives, even when not local minimizers.
• Design adaptive momentum-based acceleration schemes that do not rely on convexity or smoothness assumptions.
• Demonstrate fast, geometric convergence of iterates toward the statistical truth in sparse high-dimensional models.

Proposed method

• Reformulate optimization via generalized Bregman surrogates $ g(\boldsymbol{\beta}; \boldsymbol{\beta}^{(t)}) = f(\boldsymbol{\beta}) + \Delta_{\psi}(\boldsymbol{\beta}, \boldsymbol{\beta}^{(t)}) $, where $ \psi $ is not restricted to be smooth or convex.
• Use generalized Bregman functions to construct problem-specific error measures that enable rigorous convergence analysis.
• Apply the MM principle with higher-order matching at $ \boldsymbol{\beta} = \boldsymbol{\beta}^{(t)} $, avoiding strict majorization constraints.
• Introduce adaptive stepsize and relaxation parameters to enable momentum-based acceleration in nonconvex, nonsmooth settings.
• Leverage calculus of generalized Bregman functions to derive tight regularity conditions and simplify theoretical proofs.
• Validate the framework on $ P_H $-penalized regression with both quadratic and Tukey’s biweight loss functions under high-dimensional settings.

Figure C.1: Log-log plot of optimization error v.s. number of iterations: mirror descent for IS divergence minimization with 50 random starting points. All error curves are bounded above by the dashed line which has slope $-1$ .

Experimental results

Research questions

• RQ1 Can a unified framework be developed to analyze global convergence rates for nonsmooth, nonconvex statistical learning problems?
• RQ2 Do fixed points of Bregman surrogate algorithms in composite objectives enjoy provable statistical accuracy, even without being local minimizers?
• RQ3 Can momentum-based acceleration be successfully generalized to nonconvex, nonsmooth problems without assuming smoothness or convexity?
• RQ4 How fast do iterates converge to the statistical truth in high-dimensional sparse models under this framework?
• RQ5 What is the impact of adaptive stepsize and relaxation on computational efficiency and statistical accuracy in practice?

Key findings

• The generalized Bregman surrogate framework unifies and reinterprets existing algorithms such as LLA, iterative thresholding, and mirror descent under a common theoretical umbrella.
• Under regularity conditions, the sequence of iterates $ \boldsymbol{\beta}^{(t)} $ converges geometrically fast to the statistical truth $ \boldsymbol{\beta}^* $, with statistical error decreasing exponentially.
• Fixed points of the algorithm achieve minimax-optimal statistical accuracy in a composite objective setting, even when not local minimizers.
• The convergence rate is governed by a Bregman discrepancy measure $ \Delta_{\psi}(\boldsymbol{\beta}^*, \boldsymbol{\beta}^{(t)}) $, which exhibits exponential decay in both early and late stages of iteration.
• Adaptive momentum acceleration reduces the number of iterations by up to 90% in IS divergence minimization and cuts overall running time by over 30% in robust regression, with improved statistical precision.
• Simulations confirm that all initial points converge to the same order of statistical accuracy, validating the global convergence and robustness of the framework.

Figure C.2: Log-log plot of optimization error v.s. number of iterations: DC programming for capped- $\ell_{1}$ SVM (50 different initial points).