[Paper Review] Block-Coordinate Frank-Wolfe Optimization for Structural SVMs

Simon Lacoste-Julien, Martin Jaggi+2 more|arXiv (Cornell University)|Jul 19, 2012

Stochastic Gradient Optimization TechniquesComputer Science40 references151 citations

TL;DR

This paper proposes a randomized block-coordinate Frank-Wolfe algorithm for solving the dual structural SVM optimization problem, which achieves the same O(1/ε) convergence rate as full Frank-Wolfe but with only a single maximization oracle call per iteration—matching the efficiency of stochastic subgradient methods. Unlike stochastic methods, it enables exact line search and provides a computable duality gap for stopping, leading to superior empirical performance on structured prediction tasks.

ABSTRACT

We propose a randomized block-coordinate variant of the classic Frank-Wolfe algorithm for convex optimization with block-separable constraints. Despite its lower iteration cost, we show that it achieves a similar convergence rate in duality gap as the full Frank-Wolfe algorithm. We also show that, when applied to the dual structural support vector machine (SVM) objective, this yields an online algorithm that has the same low iteration complexity as primal stochastic subgradient methods. However, unlike stochastic subgradient methods, the block-coordinate Frank-Wolfe algorithm allows us to compute the optimal step-size and yields a computable duality gap guarantee. Our experiments indicate that this simple algorithm outperforms competing structural SVM solvers.

Motivation & Objective

• Address the inefficiency and poor convergence control of stochastic subgradient methods in training structural SVMs, which require manual step-size tuning and lack stopping criteria.
• Overcome the high per-iteration cost of batch Frank-Wolfe methods, which require a full oracle call per training example.
• Develop a block-coordinate variant of Frank-Wolfe that maintains theoretical convergence while reducing iteration cost to a single oracle call per iteration.
• Enable exact line search and duality gap computation in structural SVM training, providing practical advantages over stochastic methods.
• Demonstrate that the proposed method outperforms existing solvers in both convergence speed and generalization performance on benchmark structured prediction tasks.

Proposed method

• Propose a randomized block-coordinate Frank-Wolfe algorithm for convex optimization with block-separable constraints, where each block corresponds to a training example.
• At each iteration, select a random block (training example) and solve a linear subproblem over that block’s variable space using the maximization oracle.
• Compute the optimal step-size in closed-form via exact line search, eliminating the need for heuristic or adaptive step-size rules.
• Maintain a duality gap estimate that allows for a reliable stopping criterion, a key practical advantage over stochastic subgradient methods.
• Leverage the sparsity of Frank-Wolfe iterates to efficiently handle the exponential number of dual variables in structural SVMs.
• Prove that the block-coordinate variant retains the O(1/ε) convergence rate in duality gap, even under approximate maximization oracles.

Experimental results

Research questions

• RQ1 Can a block-coordinate variant of the Frank-Wolfe algorithm achieve the same convergence rate as the full Frank-Wolfe method while reducing per-iteration cost?
• RQ2 Does the proposed method enable exact line search and duality gap computation in structural SVM training, unlike stochastic subgradient methods?
• RQ3 Can the block-coordinate Frank-Wolfe algorithm outperform existing stochastic and batch solvers in convergence speed and test accuracy on structured prediction tasks?
• RQ4 How does the algorithm perform under approximate maximization oracles, and does it maintain theoretical convergence guarantees?
• RQ5 What is the impact of using optimal step-sizes versus fixed or adaptive step-sizes in practice for structural SVM training?

Key findings

• The block-coordinate Frank-Wolfe algorithm achieves an O(1/ε) convergence rate in duality gap, matching the theoretical rate of the full Frank-Wolfe method.
• The algorithm requires only a single call to the maximization oracle per iteration, matching the iteration cost of stochastic subgradient methods.
• The use of exact line search leads to significantly faster convergence in early iterations compared to fixed or adaptive step-sizes.
• The algorithm provides a computable duality gap, enabling a reliable stopping criterion, which is not available in standard stochastic subgradient methods.
• Empirical results show that the block-coordinate Frank-Wolfe method outperforms competing solvers in terms of objective value and generalization, especially in early training passes.
• The method is robust to approximate maximization oracles, maintaining convergence guarantees even when oracles are not solved to full precision.