[Paper Review] Support vector machines and linear regression coincide with very high-dimensional features

Navid Ardeshir, Clayton Sanford+1 more|arXiv (Cornell University)|May 28, 2021

Sparse and Compressive Sensing Techniques43 references4 citations

TL;DR

This paper establishes that support vector machines (SVMs) and ordinary least squares (OLS) regression converge to identical solutions in high-dimensional settings due to support vector proliferation (SVP), where all training examples become support vectors. It proves a super-linear lower bound on dimension $d$ for SVP in independent subgaussian feature models, identifies a sharp phase transition at $d = 2n\log n$ in Gaussian features, and conjectures that $\ell_1$-SVM exhibits SVP only at significantly higher dimensions, supported by geometric duality analysis.

ABSTRACT

The support vector machine (SVM) and minimum Euclidean norm least squares regression are two fundamentally different approaches to fitting linear models, but they have recently been connected in models for very high-dimensional data through a phenomenon of support vector proliferation, where every training example used to fit an SVM becomes a support vector. In this paper, we explore the generality of this phenomenon and make the following contributions. First, we prove a super-linear lower bound on the dimension (in terms of sample size) required for support vector proliferation in independent feature models, matching the upper bounds from previous works. We further identify a sharp phase transition in Gaussian feature models, bound the width of this transition, and give experimental support for its universality. Finally, we hypothesize that this phase transition occurs only in much higher-dimensional settings in the $\ell_1$ variant of the SVM, and we present a new geometric characterization of the problem that may elucidate this phenomenon for the general $\ell_p$ case.

Motivation & Objective

• To understand the conditions under which SVM and OLS solutions coincide in high-dimensional settings.
• To establish a tight lower bound on the dimension $d$ required for support vector proliferation (SVP) in independent subgaussian feature models.
• To investigate whether a sharp phase transition separates SVP from non-SVP regimes, particularly in Gaussian feature models.
• To explore the generalization of SVP to $\ell_p$-norm SVMs, especially the $\ell_1$ case, and to develop a geometric characterization of the dual problem.
• To provide theoretical and experimental evidence for the universality and dimensionality dependence of the SVP phenomenon across different feature distributions.

Proposed method

• Proves a super-linear lower bound $d \gtrsim n\log n$ on the dimension required for SVP in independent subgaussian feature models, matching prior upper bounds.
• Analyzes the geometry of random convex polytopes formed by feature vectors to characterize SVP via the dual optimization problem.
• Uses rotational invariance and random matrix theory to bound the probability that an optimal dual solution lies in the positive orthant, indicating SVP.
• Establishes a sharp phase transition at $d = 2n\log n$ for isotropic Gaussian features and quantifies the width of the transition.
• Introduces a geometric characterization of the dual of the $\ell_p$-SVM problem to analyze SVP beyond $\ell_2$ norms.
• Employs a union bound argument on the number of facets of the convex hull of $\pm \mathbf{A}_{\cdot,i}$ to upper-bound the probability of SVP in $\ell_1$-SVM, though acknowledges looseness in the bound.

Experimental results

Research questions

• RQ1 What is the minimum dimension $d$ required for support vector proliferation (SVP) to occur in independent subgaussian feature models?
• RQ2 Does a sharp phase transition exist in the occurrence of SVP, and if so, at what threshold of $d$ for Gaussian features?
• RQ3 How does the SVP phenomenon generalize to $\ell_p$-norm SVMs, particularly $\ell_1$-SVM, and does it require significantly higher dimensions?
• RQ4 Can a geometric characterization of the dual problem for $\ell_p$-SVMs explain the conditions under which SVP occurs?
• RQ5 Is the SVP phase transition universal across different distributions of features, or is it specific to certain models like Gaussians?

Key findings

• A super-linear lower bound $d \gtrsim n\log n$ is proven for SVP to occur in independent subgaussian feature models, matching prior upper bounds and closing a $\log n$ gap.
• For isotropic Gaussian features, a sharp phase transition in SVP occurs at $d = 2n\log n$, with the width of the transition quantitatively bounded.
• Experimental results suggest the phase transition is universal across a wide range of distributions, not limited to Gaussians.
• For $\ell_1$-SVM, SVP is conjectured to occur only at much higher dimensions than in $\ell_2$-SVM, supported by preliminary experiments.
• The probability of SVP in $\ell_1$-SVM is shown to be negligible when $d < Cn$ for some universal $C \approx 1.29$, though this bound is considered loose due to union bound overstatement.
• The geometric duality framework reveals that SVP corresponds to the dual solution lying in the positive orthant, which is rare unless the feature dimension is sufficiently large.