[Paper Review] Conditional gradient type methods for composite nonlinear and stochastic optimization

Saeed Ghadimi|arXiv (Cornell University)|Feb 2, 2016

Sparse and Compressive Sensing Techniques23 references4 citations

TL;DR

This paper introduces a conditional gradient type (CGT) method for composite nonlinear and stochastic optimization, combining weakly smooth and strongly convex terms. It achieves optimal iteration complexity for both nonconvex and convex problems, with variants that avoid parameter estimation and maintain low per-iteration cost, while extending to stochastic settings with improved complexity bounds for weakly smooth, nonconvex, and strongly convex problems.

ABSTRACT

In this paper, we present a conditional gradient type (CGT) method for solving a class of composite optimization problems where the objective function consists of a (weakly) smooth term and a (strongly) convex regularization term. While including a strongly convex term in the subproblems of the classical conditional gradient (CG) method improves its rate of convergence, it does not cost per iteration as much as general proximal type algorithms. More specifically, we present a unified analysis for the CGT method in the sense that it achieves the best-known rate of convergence when the weakly smooth term is nonconvex and possesses (nearly) optimal complexity if it turns out to be convex. While implementation of the CGT method requires explicitly estimating problem parameters like the level of smoothness of the first term in the objective function, we also present a few variants of this method which relax such estimation. Unlike general proximal type parameter free methods, these variants of the CGT method do not require any additional effort for computing (sub)gradients of the objective function and/or solving extra subproblems at each iteration. We then generalize these methods under stochastic setting and present a few new complexity results. To the best of our knowledge, this is the first time that such complexity results are presented for solving stochastic weakly smooth nonconvex and (strongly) convex optimization problems.

Motivation & Objective

• To develop a unified conditional gradient type (CGT) method for composite optimization problems with weakly smooth and strongly convex components.
• To achieve the best-known iteration complexity for nonconvex weakly smooth problems and nearly optimal rates when the problem is convex.
• To design parameter-free variants that avoid explicit estimation of smoothness parameters without extra computational cost.
• To extend the CGT framework to stochastic settings where only noisy gradients are available.
• To establish new complexity bounds for stochastic weakly smooth nonconvex and strongly convex optimization problems.

Proposed method

• The method solves subproblems by minimizing a linear approximation of the smooth term plus the regularization term over a convex set, avoiding proximal-like subproblems.
• It uses a backtracking line search or fixed stepsizes based on problem parameters, with convergence analyzed via descent inequalities involving gradient mapping and Bregman distance.
• Key components include the gradient mapping $ g_{X,k} $, the sequence $ y_k $, and the update rule $ x_k = (1 - \alpha_k) y_{k-1} + \alpha_k s_k $, where $ s_k $ solves a linear subproblem.
• For stochastic settings, the method uses stochastic oracle access with batched gradients, and analyzes expected suboptimality and stationarity.
• Two-phase variants are proposed to improve large-deviation bounds using multiple runs and solution selection.
• Theoretical analysis relies on Hölder continuity of the gradient and strong convexity of the regularization term with parameter $ \mu \geq 0 $.

Experimental results

Research questions

• RQ1 Can a conditional gradient type method achieve optimal iteration complexity for weakly smooth nonconvex composite problems?
• RQ2 Can such a method maintain low per-iteration cost while avoiding explicit knowledge of smoothness parameters?
• RQ3 What is the iteration complexity of CGT methods under stochastic settings with noisy gradient information?
• RQ4 Can the method be unified to automatically adapt to convexity, achieving optimal rates in both nonconvex and convex regimes?
• RQ5 How do the stochastic variants compare in complexity to existing first-order methods for weakly smooth stochastic optimization?

Key findings

• The CGT method achieves $ \mathcal{O}(1/\epsilon) $ iteration complexity for finding an $ \epsilon $-stationary point in nonconvex weakly smooth problems, matching the best-known bound.
• For convex weakly smooth problems, the method achieves $ \mathcal{O}(1/\epsilon^{\frac{1-\nu}{1+3\nu}}) $ complexity, matching the optimal bound up to logarithmic factors.
• In the smooth case ($ \nu = 1 $), the stochastic variant achieves $ \mathcal{O}(1/\epsilon^2) $ oracle complexity for nonconvex problems, which is optimal up to logarithmic factors.
• When $ f $ is convex and smooth, the stochastic method achieves $ \mathcal{O}(1/\epsilon \log(1/\epsilon)) $ complexity, which is optimal up to a logarithmic factor.
• The parameter-free variants avoid estimating $ L_\nu $ without additional gradient or subproblem computation, maintaining low per-iteration cost.
• The first complexity bounds for stochastic weakly smooth nonconvex and strongly convex optimization are established, filling a gap in the literature.