[Paper Review] Relatively-Smooth Convex Optimization by First-Order Methods, and Applications

Haihao Lu, Robert M. Freund+1 more|arXiv (Cornell University)|Oct 18, 2016

Sparse and Compressive Sensing Techniques17 references4 citations

TL;DR

This paper introduces a new framework for first-order convex optimization by generalizing smoothness to 'relative smoothness' with respect to a user-defined reference function $ h(\cdot) $, enabling efficient algorithms for problems where traditional uniform smoothness does not hold. The key contribution is a new convergence analysis and algorithmic framework that achieves $ O(1/k) $ sublinear convergence for non-uniformly smooth functions, demonstrated on $ D $-optimal design and other non-smoothable problems.

ABSTRACT

The usual approach to developing and analyzing first-order methods for smooth convex optimization assumes that the gradient of the objective function is uniformly smooth with some Lipschitz constant $L$. However, in many settings the differentiable convex function $f(\cdot)$ is not uniformly smooth -- for example in $D$-optimal design where $f(x):=-\ln \det(HXH^T)$, or even the univariate setting with $f(x) := -\ln(x) + x^2$. Herein we develop a notion of "relative smoothness" and relative strong convexity that is determined relative to a user-specified "reference function" $h(\cdot)$ (that should be computationally tractable for algorithms), and we show that many differentiable convex functions are relatively smooth with respect to a correspondingly fairly-simple reference function $h(\cdot)$. We extend two standard algorithms -- the primal gradient scheme and the dual averaging scheme -- to our new setting, with associated computational guarantees. We apply our new approach to develop a new first-order method for the $D$-optimal design problem, with associated computational complexity analysis. Some of our results have a certain overlap with the recent work \cite{bbt}.

Motivation & Objective

• Address the limitation of standard first-order methods that require uniform Lipschitz continuity of the gradient, which fails in many practical convex optimization problems.
• Develop a generalized smoothness condition—relative smoothness—relative to a computationally tractable reference function $ h(\cdot) $, allowing analysis of non-uniformly smooth functions.
• Extend the primal gradient and dual averaging schemes to the relative smoothness setting, providing new convergence guarantees.
• Apply the framework to $ D $-optimal design and other non-smoothable problems where traditional methods fail due to unbounded Lipschitz constants.
• Balance computational tractability and convergence rate by selecting $ h(\cdot) $ to minimize the Hessian of the gap function $ Lh - f $, while keeping subproblem solutions efficient.

Proposed method

• Define relative smoothness: a differentiable convex function $ f $ is $ L $-smooth relative to a $ 1 $-strongly convex reference function $ h $ if $ \nabla f(x) - \nabla f(y) \leq L \cdot \|x - y\|_h $, where $ \|\cdot\|_h $ is the norm induced by $ h $.
• Use Bregman distances $ D_h(x,y) = h(x) - h(y) - \langle \nabla h(y), x - y \rangle $ to define the new optimization subproblems.
• Modify the primal gradient scheme to use $ L D_h(x, x^i) $ in the update, replacing the standard $ \|x - x^i\|_2^2 $ term.
• Prove a generalized three-point property under relative smoothness, enabling convergence analysis via Bregman distances.
• Establish convergence rate $ f(x^k) - f(x^*) \leq \frac{L D_h(x^*, x^0)}{k} $, generalizing the classical $ O(1/k) $ rate.
• Propose a strategy to balance subproblem tractability and convergence by choosing $ h(\cdot) $ such that the Hessian of $ Lh - f $ is small, avoiding both poor conditioning and hard subproblems.

Experimental results

Research questions

• RQ1 Can first-order methods be extended to convex functions that are not uniformly smooth, such as $ f(x) = -\ln(x) + x^2 $ or $ f(x) = -\ln \det(HXH^T) $?
• RQ2 What is a suitable generalization of smoothness that allows convergence guarantees for non-uniformly smooth functions?
• RQ3 How can the Bregman distance framework be adapted to maintain $ O(1/k) $ convergence rates under relative smoothness?
• RQ4 What criteria should guide the selection of the reference function $ h(\cdot) $ to balance convergence rate and subproblem complexity?
• RQ5 Can the new framework be successfully applied to $ D $-optimal design and other problems with unbounded gradient Lipschitz constants?

Key findings

• The paper establishes that many non-uniformly smooth convex functions—such as $ f(x) = -\ln(x) + x^2 $ and $ f(x) = -\ln \det(HXH^T) $—are relatively smooth with respect to simple reference functions like $ h(x) = \frac{1}{2}\|x\|_2^2 $ or $ h(x) = \sum x_i \ln x_i $.
• The relative smoothness framework allows the primal gradient scheme to achieve $ O(1/k) $ convergence rate: $ f(x^k) - f(x^*) \leq \frac{L D_h(x^*, x^0)}{k} $, even when $ f $ is not uniformly smooth.
• For $ D $-optimal design, the method achieves a computational complexity bound that avoids the exponential growth of $ L_f $ seen in classical approaches, enabling practical implementation.
• The Hessian of the gap function $ \nabla^2(Lh - f) $ is non-negative definite under relative smoothness, ensuring the convergence bound is well-defined and non-increasing.
• The choice of $ h(\cdot) $ involves a trade-off: $ h(\cdot) = f(\cdot) $ gives zero Hessian gap but hard subproblems; $ h(\cdot) = \frac{1}{2}\|\cdot\|_2^2 $ gives easy subproblems but poor Hessian control.
• The framework generalizes standard first-order methods and provides a principled way to select $ h(\cdot) $, such as using preconditioning matrices $ B $, to improve convergence without sacrificing computational feasibility.