[Paper Review] Resilience: A Criterion for Learning in the Presence of Arbitrary Outliers

Jacob Steinhardt, Moses Charikar+1 more|arXiv (Cornell University)|Mar 15, 2017

Machine Learning and Algorithms15 references17 citations

TL;DR

This paper introduces resilience as a general criterion for robust learning in high-dimensional data with arbitrary outliers. It shows that if a dataset is resilient—meaning the mean of any large subset remains close to the overall mean—then accurate estimation of parameters like the mean is possible even when up to a constant fraction of data is adversarially corrupted. The key contribution is a new theoretical framework enabling dimension-independent robust estimation under minimal assumptions, with applications to mean estimation, distribution learning, and stochastic block models.

ABSTRACT

We introduce a criterion, resilience, which allows properties of a dataset (such as its mean or best low rank approximation) to be robustly computed, even in the presence of a large fraction of arbitrary additional data. Resilience is a weaker condition than most other properties considered so far in the literature, and yet enables robust estimation in a broader variety of settings. We provide new information-theoretic results on robust distribution learning, robust estimation of stochastic block models, and robust mean estimation under bounded $k$th moments. We also provide new algorithmic results on robust distribution learning, as well as robust mean estimation in $\ell_p$-norms. Among our proof techniques is a method for pruning a high-dimensional distribution with bounded $1$st moments to a stable "core" with bounded $2$nd moments, which may be of independent interest.

Motivation & Objective

• To identify a general, simple criterion that enables robust estimation in high-dimensional settings with adversarial outliers.
• To provide information-theoretic bounds for robust learning of distributions with bounded k-th moments, discrete distributions, and stochastic block models.
• To develop efficient algorithms for robust mean estimation in ℓp-norms and discrete distribution learning under resilience.
• To establish a connection between resilience and the existence of a stable core with bounded variance via pruning techniques.
• To unify and generalize prior results on robust estimation by introducing a weaker, more widely applicable condition than previous assumptions.

Proposed method

• Define resilience as a property of a dataset where the mean of any large subset remains close to the overall mean, formalized via a norm-bound on subset deviations.
• Use minimax duality and Khintchine’s inequality to prove that a resilient set with bounded first moments contains a core with bounded second moments.
• Leverage strong convexity of the norm to ensure the existence of a large core with bounded variance, enabling algorithmic applications.
• Apply the core extraction technique to derive efficient algorithms for robust mean estimation in ℓp-norms and discrete distribution learning.
• Use powering techniques to iteratively strengthen resilience, enabling robustness even when initial data has only bounded first moments.
• Prove that resilience implies robustness to adversarial contamination, with error bounds independent of dimension under mild conditions.

Experimental results

Research questions

• RQ1 Can a general, simple criterion be formulated that guarantees robust estimation of the mean in high-dimensional data with arbitrary outliers?
• RQ2 Under what conditions can robust mean estimation be achieved with error independent of dimension, even when the data has heavy-tailed or high-dimensional structure?
• RQ3 How can a dataset with bounded first moments be transformed into a stable core with bounded variance, enabling efficient algorithms?
• RQ4 What is the relationship between resilience and existing robustness criteria such as bounded moments or sub-Gaussian tails?
• RQ5 Can resilience be used to derive efficient algorithms for robust learning of discrete distributions and stochastic block models?

Key findings

• Resilience is a weaker condition than bounded k-th moments or sub-Gaussianity, yet it enables robust estimation under broader conditions.
• For any set with bounded first moments in a norm, if the norm is γ-strongly convex, then a core of size at least half the original set exists with variance bounded by 32σ²/γ.
• Robust mean estimation in ℓp-norms is possible with error O(ε) when the underlying distribution is resilient, independent of dimension.
• The paper establishes that resilience implies the existence of a stable core with bounded second moments, enabling algorithmic applications.
• For discrete distributions, an efficient polynomial-time algorithm for robust learning is achieved under the resilience condition.
• The framework provides new information-theoretic bounds for robust learning of stochastic block models and distributions with bounded k-th moments.