[Paper Review] Non-Convex SGD Learns Halfspaces with Adversarial Label Noise

Ilias Diakonikolas, Vasilis Kontonis+2 more|arXiv (Cornell University)|Jun 11, 2020

Machine Learning and Algorithms15 references4 citations

TL;DR

This paper demonstrates that stochastic gradient descent (SGD) on a non-convex logistic loss surrogate efficiently learns homogeneous halfspaces under adversarial label noise when the data distribution is well-behaved—such as log-concave or sub-Gaussian. It shows that non-convex SGD achieves misclassification error of $O(\mathrm{opt}) + \epsilon$, while convex surrogates inherently suffer from error $\omega(\mathrm{opt})$, even under Gaussian marginals.

ABSTRACT

We study the problem of agnostically learning homogeneous halfspaces in the distribution-specific PAC model. For a broad family of structured distributions, including log-concave distributions, we show that non-convex SGD efficiently converges to a solution with misclassification error $O(\opt)+\eps$, where $\opt$ is the misclassification error of the best-fitting halfspace. In sharp contrast, we show that optimizing any convex surrogate inherently leads to misclassification error of $ω(\opt)$, even under Gaussian marginals.

Motivation & Objective

• To design a simple, practical algorithm for learning homogeneous halfspaces under adversarial label noise with near-optimal error guarantees.
• To show that non-convex optimization via SGD on a smooth 0-1 loss surrogate can achieve $O(\mathrm{opt}) + \epsilon$ error under well-behaved distributions.
• To demonstrate that convex surrogates inherently fail to achieve $O(\mathrm{opt}) + \epsilon$ error, even under Gaussian marginals.
• To establish that stationary points of the non-convex logistic loss correspond to near-optimal halfspaces under isotropic log-concave, sub-Gaussian, and heavy-tailed distributions.

Proposed method

• The method uses a non-convex surrogate loss $\mathcal{L}_{\sigma}(\mathbf{w}) = \mathbb{E}_{(\mathbf{x},y)\sim\mathcal{D}}[S_\sigma(-y\langle\mathbf{w},\mathbf{x}\rangle)]$, where $S_\sigma(t) = 1/(1 + e^{-t/\sigma})$ is the logistic function.
• Optimization is performed over the unit sphere $\|\mathbf{w}\|_2 = 1$ to ensure scale invariance and stability.
• The analysis relies on geometric concentration properties: for any distribution in the family, the probability mass in the region where $\|\mathbf{x}\|_2 \geq Z$ is bounded by $\mathrm{opt}$, with $Z$ chosen based on tail behavior.
• It proves that any halfspace with angle $\theta(\mathbf{w}, \mathbf{w}^*) \leq \theta$ from the optimal weight vector $\mathbf{w}^*$ has non-zero gradient, so cannot be a minimizer of the convex surrogate.
• The argument uses anti-concentration and radial symmetry to show that the gradient does not vanish near the optimal halfspace, even under adversarial label flips.
• The key insight is that the non-convex loss avoids spurious stationary points near the optimal solution, unlike convex surrogates.

Experimental results

Research questions

• RQ1 Can non-convex SGD on a smooth 0-1 loss surrogate learn homogeneous halfspaces under adversarial label noise with error close to the optimal misclassification error?
• RQ2 Why do convex surrogates fail to achieve $O(\mathrm{opt}) + \epsilon$ error even under Gaussian marginals?
• RQ3 What distributional assumptions ensure that stationary points of the non-convex logistic loss are near-optimal?
• RQ4 How does the geometry of the distribution (e.g., log-concave, heavy-tailed) affect the existence of spurious stationary points in the surrogate loss?
• RQ5 Can a simple, single-phase optimization method like SGD match the error guarantees of complex multi-phase algorithms in the agnostic halfspace learning setting?

Key findings

• Non-convex SGD on the logistic loss achieves misclassification error $O(\mathrm{opt}) + \epsilon$ for homogeneous halfspaces under isotropic log-concave distributions.
• For Gaussian marginals, any convex surrogate leads to misclassification error $\omega(\mathrm{opt})$, meaning it cannot achieve $O(\mathrm{opt}) + \epsilon$ error.
• Under isotropic log-concave distributions, the error bound is $O(\mathrm{opt}) + \epsilon$ with polynomial sample and runtime complexity.
• For heavy-tailed distributions with tail index $s > 2$, the error bound is $O(\mathrm{opt}^{1-1/s}) + \epsilon$, which remains near-optimal for small $\mathrm{opt}$.
• The gradient of the convex surrogate does not vanish in any $\theta$-neighborhood of the optimal halfspace, where $\theta = \Omega(\mathrm{opt}^{1-1/s})$, proving that such points cannot be minimizers.
• The analysis shows that the non-convex loss avoids spurious stationary points near the optimal solution, enabling SGD to converge to near-optimal solutions.