[Paper Review] List-Decodable Linear Regression

Sushrut Karmalkar, Adam R. Klivans+1 more|arXiv (Cornell University)|May 14, 2019

Machine Learning and Algorithms23 references4 citations

TL;DR

This paper presents the first polynomial-time algorithm for list-decodable linear regression, where more than half of the data points are adversarially corrupted. It uses a sum-of-squares-based framework to output a constant-sized list of hypotheses, one of which is close to the true regression vector, under the condition that inliers follow a certifiably anti-concentrated distribution—such as the standard Gaussian or uniform on the hypercube—providing the first efficient solution for this robust learning setting with strong theoretical guarantees.

ABSTRACT

We give the first polynomial-time algorithm for robust regression in the list-decodable setting where an adversary can corrupt a greater than $1/2$ fraction of examples. For any $α< 1$, our algorithm takes as input a sample $\{(x_i,y_i)\}_{i \leq n}$ of $n$ linear equations where $αn$ of the equations satisfy $y_i = \langle x_i,\ell^* angle +ζ$ for some small noise $ζ$ and $(1-α)n$ of the equations are {\em arbitrarily} chosen. It outputs a list $L$ of size $O(1/α)$ - a fixed constant - that contains an $\ell$ that is close to $\ell^*$. Our algorithm succeeds whenever the inliers are chosen from a \emph{certifiably} anti-concentrated distribution $D$. In particular, this gives a $(d/α)^{O(1/α^8)}$ time algorithm to find a $O(1/α)$ size list when the inlier distribution is standard Gaussian. For discrete product distributions that are anti-concentrated only in \emph{regular} directions, we give an algorithm that achieves similar guarantee under the promise that $\ell^*$ has all coordinates of the same magnitude. To complement our result, we prove that the anti-concentration assumption on the inliers is information-theoretically necessary. Our algorithm is based on a new framework for list-decodable learning that strengthens the `identifiability to algorithms' paradigm based on the sum-of-squares method. In an independent and concurrent work, Raghavendra and Yau also used the Sum-of-Squares method to give a similar result for list-decodable regression.

Motivation & Objective

• To design an efficient, polynomial-time algorithm for linear regression when more than half the training data is corrupted by an adversary.
• To identify the minimal distributional assumptions on inliers that make list-decodable regression information-theoretically possible.
• To establish a principled connection between certifiable anti-concentration and algorithmic solvability in robust estimation via the sum-of-squares method.
• To provide a tight information-theoretic lower bound showing that anti-concentration is necessary for any list-decodable regression algorithm.

Proposed method

• The algorithm leverages the sum-of-squares (SoS) method to construct a pseudo-distribution over candidate regression vectors that satisfies moment constraints derived from the corrupted data.
• It uses a new framework that strengthens the 'identifiability to algorithms' paradigm by incorporating certifiable anti-concentration properties of the inlier distribution.
• For Gaussian and spherically symmetric distributions with sub-exponential tails, the algorithm achieves a runtime of (d/α)^O(1/α⁸) and outputs a list of size O(1/α).
• For discrete product distributions, the algorithm requires the true regression vector to have coordinates of equal magnitude and still achieves the same list size and runtime under the same anti-concentration condition.
• The method constructs a family of distributions R_i that are statistically indistinguishable across different inlier directions, enabling a tight lower bound on list size.
• It proves that anti-concentration is information-theoretically necessary by constructing a family of distributions where the inlier distribution is uniform on {0,1}^d or [q]^d, showing that no algorithm can succeed with list size smaller than d.

Experimental results

Research questions

• RQ1 Can we design a polynomial-time algorithm for linear regression when more than half the data is adversarially corrupted, without requiring strong separation or condition-number assumptions on the true regression vector?
• RQ2 What distributional assumptions on the inlier data are necessary and sufficient to make list-decodable regression information-theoretically solvable?
• RQ3 Can the sum-of-squares method be extended to provide efficient algorithms for list-decodable regression under weak assumptions on the inlier distribution?
• RQ4 Is the anti-concentration property of the inlier distribution both necessary and sufficient for efficient list-decodable regression?
• RQ5 What is the minimal list size required for any algorithm to solve list-decodable regression, and can this be tightly bounded under natural distributions?

Key findings

• The algorithm runs in time (d/α)^O(1/α⁸) and outputs a list of size O(1/α) containing a regression vector that is close to the true vector ℓ* when inliers are drawn from a certifiably anti-concentrated distribution.
• For standard Gaussian inliers, the algorithm achieves a runtime of (d/α)^O(1/α⁸) and guarantees a constant-sized list, making it the first efficient solution in this setting.
• The algorithm succeeds under the condition that the inlier distribution is certifiably anti-concentrated, which includes spherically symmetric distributions with sub-exponential tails and discrete product distributions with regular anti-concentration.
• The paper proves that anti-concentration is information-theoretically necessary: for uniform distributions on {0,1}^d or [q]^d, no algorithm can achieve a list size smaller than d, even with no noise.
• The lower bound construction shows that for any q, the uniform distribution on [q]^d has P(⟨x,v⟩=0) ≤ 1/q, and this bound is tight, implying that list size must grow with d when q is small.
• The work establishes that list-decodable regression is possible only when inliers avoid concentration on low-dimensional subspaces, and certifiable anti-concentration provides a tractable condition for algorithmic use.