[Paper Review] The Complexity of Large-scale Convex Programming under a Linear Optimization Oracle

Guanghui Lan|arXiv (Cornell University)|Sep 22, 2013

Sparse and Compressive Sensing TechniquesEngineering46 references86 citations

TL;DR

This paper establishes lower complexity bounds for linear-optimization-based convex programming (LCP) methods, proving the conditional gradient (CndG) method is optimal for smooth convex problems under an LO oracle. It introduces new accelerated LCP variants—PA-CndG and PDA-CndG—derived from Nesterov’s method, demonstrating superior convergence and numerical performance, especially on box-constrained problems.

ABSTRACT

This paper considers a general class of iterative optimization algorithms, referred to as linear-optimization-based convex programming (LCP) methods, for solving large-scale convex programming (CP) problems. The LCP methods, covering the classic conditional gradient (CG) method (a.k.a., Frank-Wolfe method) as a special case, can only solve a linear optimization subproblem at each iteration. In this paper, we first establish a series of lower complexity bounds for the LCP methods to solve different classes of CP problems, including smooth, nonsmooth and certain saddle-point problems. We then formally establish the theoretical optimality or nearly optimality, in the large-scale case, for the CG method and its variants to solve different classes of CP problems. We also introduce several new optimal LCP methods, obtained by properly modifying Nesterov's accelerated gradient method, and demonstrate their possible advantages over the classic CG for solving certain classes of large-scale CP problems.

Motivation & Objective

• To establish tight lower complexity bounds for LCP methods solving smooth, nonsmooth, and saddle-point convex problems under a linear optimization oracle.
• To formally prove the theoretical optimality of the conditional gradient (CndG) method and its variants in the large-scale setting.
• To develop new optimal or near-optimal LCP methods by adapting Nesterov’s accelerated gradient method to the LO oracle framework.
• To demonstrate through numerical experiments that PDA-CndG significantly outperforms classic CndG and PA-CndG on certain classes of large-scale problems, particularly those with box-type constraints.

Proposed method

• Derives lower complexity bounds for LCP methods using worst-case analysis, generalizing existing results in the literature.
• Applies Nesterov’s optimal first-order method framework to construct new LCP variants—PA-CndG and PDA-CndG—by modifying the update rules to suit the LO oracle constraint.
• Introduces the PDA-CndG method via a primal-dual acceleration technique, enhancing convergence for problems with structured constraints.
• Employs a linear optimization oracle that solves subproblems of the form $\arg\min_{x\in X} \langle p, x \rangle$ at each iteration, avoiding costly proximal or projection steps.
• Analyzes convergence rates under different problem classes: smooth, nonsmooth, and saddle-point problems, under the LO oracle model.
• Validates performance via extensive numerical experiments on QP problems over simplices, spectrahedra, hypercubes, and hypercube-simplex intersections.

Experimental results

Research questions

• RQ1 What are the fundamental lower complexity bounds for LCP methods solving smooth, nonsmooth, and saddle-point convex problems under a linear optimization oracle?
• RQ2 Is the classic conditional gradient (CndG) method optimal among LCP methods for smooth convex problems under the LO oracle?
• RQ3 Can Nesterov’s acceleration technique be effectively adapted to the LCP framework to yield faster-converging algorithms?
• RQ4 How do the new LCP variants—PA-CndG and PDA-CndG—compare numerically to CndG and PA-CndG on large-scale problems with box-type constraints?
• RQ5 Under what problem structures does PDA-CndG significantly outperform standard CndG in practice, despite similar worst-case complexity bounds?

Key findings

• The CndG method achieves the optimal convergence rate of $\mathcal{O}(1/\epsilon)$ for smooth convex problems under the LO oracle, matching the established lower bound.
• For nonsmooth and saddle-point problems, the paper establishes lower complexity bounds and shows that certain LCP variants achieve nearly optimal performance.
• The PDA-CndG method outperforms both CndG and PA-CndG by up to two orders of magnitude on QP problems over hypercubes and hypercube-simplex intersections.
• In numerical experiments, PDA-CndG achieved objective values at 100 iterations comparable to CndG at 1,000 iterations, with better accuracy by 1–3 digits.
• The PDA-CndG method maintains optimal convergence rates while demonstrating superior practical performance on box-constrained problems, especially as dimensionality increases.
• The PA-CndG and PDA-CndG methods inherit the same worst-case complexity as CndG but exhibit significantly faster convergence in practice due to improved step-size and momentum strategies.