[Paper Review] Fast Epigraphical Projection-based Incremental Algorithms for Wasserstein Distributionally Robust Support Vector Machine

Jiajin Li, Caihua Chen+1 more|arXiv (Cornell University)|Oct 24, 2020

Sparse and Compressive Sensing Techniques26 references6 citations

TL;DR

This paper proposes two novel epigraphical projection-based incremental algorithms for solving Wasserstein distributionally robust support vector machines (DRSVM), leveraging efficient closed-form updates and convergence guarantees under a Hölderian growth condition. The methods achieve orders-of-magnitude speedup over state-of-the-art solvers, especially on large-scale problems, with empirical results showing significant performance gains as dataset size increases.

ABSTRACT

Wasserstein extbf{D}istributionally extbf{R}obust extbf{O}ptimization (DRO) is concerned with finding decisions that perform well on data that are drawn from the worst-case probability distribution within a Wasserstein ball centered at a certain nominal distribution. In recent years, it has been shown that various DRO formulations of learning models admit tractable convex reformulations. However, most existing works propose to solve these convex reformulations by general-purpose solvers, which are not well-suited for tackling large-scale problems. In this paper, we focus on a family of Wasserstein distributionally robust support vector machine (DRSVM) problems and propose two novel epigraphical projection-based incremental algorithms to solve them. The updates in each iteration of these algorithms can be computed in a highly efficient manner. Moreover, we show that the DRSVM problems considered in this paper satisfy a Hölderian growth condition with explicitly determined growth exponents. Consequently, we are able to establish the convergence rates of the proposed incremental algorithms. Our numerical results indicate that the proposed methods are orders of magnitude faster than the state-of-the-art, and the performance gap grows considerably as the problem size increases.

Motivation & Objective

• To address the scalability limitations of general-purpose solvers in solving large-scale Wasserstein distributionally robust SVM problems.
• To develop efficient, incremental optimization algorithms tailored for the convex reformulation of DRSVM under Wasserstein ambiguity sets.
• To establish convergence rates by proving the DRSVM problem satisfies a Hölderian growth condition with explicitly determined exponents.
• To achieve significant speedups over existing methods, particularly as problem size increases.
• To provide a scalable alternative to off-the-shelf solvers like YALMIP and CPLEX for DRO-based learning models.

Proposed method

• Proposes two epigraphical projection-based incremental algorithms that exploit the structure of the DRSVM problem for efficient per-iteration updates.
• Uses operator splitting and ADMM to decompose the $w$-subproblem into subproblems with closed-form solutions.
• Applies a single-sample proximal point method with adaptive penalty parameter update via $\rho_{k+1} = \gamma \rho_k$ for acceleration.
• Reformulates the $w$-subproblem using auxiliary variables $y$ and $g$ to enable efficient projection onto the $\ell_q$-norm constraint.
• Employs a standard ADMM framework with a shrinking penalty strategy to enhance convergence and stability.
• Derives a transformed problem (28) via variable scaling to maintain the same structure as the original, enabling efficient computation.

Experimental results

Research questions

• RQ1 Can epigraphical projection-based incremental algorithms achieve faster convergence and scalability for Wasserstein DRSVM compared to general-purpose solvers?
• RQ2 Does the DRSVM problem satisfy a Hölderian growth condition, and can this be used to establish convergence rates?
• RQ3 How do the proposed algorithms perform in terms of wall-clock time and objective value accuracy on real-world UCI datasets?
• RQ4 What is the impact of problem size on the performance gap between the proposed methods and state-of-the-art solvers?
• RQ5 Can the use of adaptive penalty updates and closed-form projections lead to significant speedups in large-scale DRSVM optimization?

Key findings

• The proposed M-ISG and IPPA algorithms achieve orders-of-magnitude faster convergence than YALMIP and CPLEX, with performance gaps widening as problem size increases.
• On the a9a dataset, the M-ISG method completed in 3.018 seconds for $\ell_1$-DRSVM, while YALMIP required 642.160 seconds, a 213x speedup.
• For $\ell_\infty$-DRSVM on a9a, M-ISG took 3.368 seconds versus 753.330 seconds for YALMIP, a 224x speedup.
• All proposed methods achieved objective values within 1e-6 of the best-known solution, confirming high accuracy.
• The algorithms demonstrate consistent superiority across all 9 UCI datasets tested, with M-ISG and IPPA outperforming Hybrid and YALMIP in both speed and convergence.
• Theoretical analysis confirms the DRSVM problem satisfies a Hölderian growth condition with explicitly determined exponents, enabling convergence rate guarantees.