[Paper Review] Dual coordinate solvers for large-scale structural SVMs

Deva Ramanan|arXiv (Cornell University)|Dec 6, 2013

Human Pose and Action Recognition22 references14 citations

TL;DR

This paper presents a dual coordinate descent solver for large-scale structural SVMs that efficiently handles out-of-core datasets by combining batch optimization with online data processing. By leveraging hot-starting from previous solutions and dynamically balancing data access and optimization, the method achieves batch-like stability and convergence speed while scaling to datasets too large to fit in memory.

ABSTRACT

This manuscript describes a method for training linear SVMs (including binary SVMs, SVM regression, and structural SVMs) from large, out-of-core training datasets. Current strategies for large-scale learning fall into one of two camps; batch algorithms which solve the learning problem given a finite datasets, and online algorithms which can process out-of-core datasets. The former typically requires datasets small enough to fit in memory. The latter is often phrased as a stochastic optimization problem; such algorithms enjoy strong theoretical properties but often require manual tuned annealing schedules, and may converge slowly for problems with large output spaces (e.g., structural SVMs). We discuss an algorithm for an "intermediate" regime in which the data is too large to fit in memory, but the active constraints (support vectors) are small enough to remain in memory. In this case, one can design rather efficient learning algorithms that are as stable as batch algorithms, but capable of processing out-of-core datasets. We have developed such a MATLAB-based solver and used it to train a collection of recognition systems for articulated pose estimation, facial analysis, 3D object recognition, and action classification, all with publicly-available code. This writeup describes the solver in detail.

Motivation & Objective

• Address the challenge of training structural SVMs on datasets too large to fit in memory, bridging the gap between batch and online learning.
• Develop a solver that maintains the stability and convergence properties of batch methods while processing data in a streaming fashion.
• Enable efficient training of linear SVMs, including structural and latent SVMs, on out-of-core datasets with minimal memory overhead.
• Introduce adaptive scheduling between data access and optimization to improve convergence speed and model quality.
• Support flexible regularization and non-negativity constraints in the optimization framework for broader applicability.

Proposed method

• Adapt the fast dual coordinate descent algorithm from liblinear to handle generalized SVM formulations, including structural and latent SVMs.
• Modify the solver to operate in a partially online regime, processing data in small in-memory batches while maintaining global convergence.
• Use previously computed solutions as hot-starts for each new optimization batch, significantly reducing per-iteration cost.
• Maintain upper and lower bounds on the duality gap to guide an optimal trade-off between exploring new data and optimizing over existing data.
• Generalize the QP formulation to support shared slack variables across multiple constraints per example, enabling efficient handling of large output spaces.
• Map various SVM variants—binary, multiclass, regression, structural, and latent SVMs—into a unified optimization framework using a common dual QP formulation.

Experimental results

Research questions

• RQ1 Can a dual coordinate descent method be adapted to train structural SVMs on out-of-core datasets while preserving the convergence and stability of batch methods?
• RQ2 How can hot-starting from prior solutions improve the efficiency of iterative optimization in large-scale learning?
• RQ3 What is the optimal strategy for balancing data access and optimization in an online setting to accelerate convergence?
• RQ4 How can shared slack variables be effectively used to control loss growth in problems with exponentially large output spaces?
• RQ5 Can the proposed method be generalized to support non-negativity constraints and flexible regularization in SVM training?

Key findings

• The proposed solver achieves stable and fast convergence on large-scale structural SVM problems by combining batch optimization with online data processing.
• Hot-starting optimizations using previous solutions reduces the computational cost of each new batch, enabling efficient incremental learning.
• The adaptive scheduling strategy based on duality gap bounds leads to faster convergence compared to fixed or random scheduling in online settings.
• The method successfully trains a variety of recognition systems—including articulated pose estimation, facial analysis, 3D object recognition, and action classification—on large out-of-core datasets.
• The solver supports multiple SVM variants, including binary, multiclass, regression, structural, and latent structural SVMs, through a unified QP formulation.
• Empirical results show the method outperforms standard online algorithms in convergence speed and model quality, particularly on problems with large output spaces.