[Paper Review] Outlier-Robust Clustering of Non-Spherical Mixtures

Ainesh Bakshi, Pravesh K. Kothari|arXiv (Cornell University)|May 6, 2020

Bayesian Methods and Mixture Models30 references16 citations

TL;DR

This paper presents the first efficient, outlier-robust algorithm for clustering mixtures of $k$-dimensional Gaussians under total variation (TV) separation, a statistically minimal condition. By leveraging certifiable hypercontractivity and anti-concentration of low-degree polynomials, the algorithm achieves $d^{\operatorname{poly}(k/\eta)}$-time clustering with misclassification error $k^{O(k)}(\varepsilon + \eta)$, even when components share the same mean but differ in covariance along unknown directions.

ABSTRACT

We give the first outlier-robust efficient algorithm for clustering a mixture of $k$ statistically separated d-dimensional Gaussians (k-GMMs). Concretely, our algorithm takes input an $ε$-corrupted sample from a $k$-GMM and whp in $d^{ ext{poly}(k/η)}$ time, outputs an approximate clustering that misclassifies at most $k^{O(k)}(ε+η)$ fraction of the points whenever every pair of mixture components are separated by $1-\exp(- ext{poly}(k/η)^k)$ in total variation (TV) distance. Such a result was not previously known even for $k=2$. TV separation is the statistically weakest possible notion of separation and captures important special cases such as mixed linear regression and subspace clustering. Our main conceptual contribution is to distill simple analytic properties - (certifiable) hypercontractivity and bounded variance of degree 2 polynomials and anti-concentration of linear projections - that are necessary and sufficient for mixture models to be (efficiently) clusterable. As a consequence, our results extend to clustering mixtures of arbitrary affine transforms of the uniform distribution on the $d$-dimensional unit sphere. Even the information-theoretic clusterability of separated distributions satisfying these two analytic assumptions was not known prior to our work and is likely to be of independent interest. Our algorithms build on the recent sequence of works relying on certifiable anti-concentration first introduced in the works of Karmarkar, Klivans, and Kothari and Raghavendra, and Yau in 2019. Our techniques expand the sum-of-squares toolkit to show robust certifiability of TV-separated Gaussian clusters in data. This involves giving a low-degree sum-of-squares proof of statements that relate parameter (i.e. mean and covariances) distance to total variation distance by relying only on hypercontractivity and anti-concentration.

Motivation & Objective

• To develop an efficient, outlier-robust algorithm for clustering $k$-Gaussian mixtures when components are separated only in total variation (TV) distance, the weakest statistical separation notion.
• To identify and formalize analytic conditions—certifiable hypercontractivity and anti-concentration—that are both necessary and sufficient for efficient clustering of $k$-GMMs.
• To extend the sum-of-squares (SoS) framework to prove robust certifiability of TV-separated Gaussian clusters via low-degree SoS proofs of parameter-to-TV distance inequalities.
• To establish that the algorithm works not only for Gaussians but also for mixtures of arbitrary affine transforms of the uniform distribution on the $d$-dimensional unit sphere.
• To resolve the information-theoretic clusterability of distributions satisfying these analytic conditions, which was previously unknown.

Proposed method

• The algorithm uses the sum-of-squares (SoS) method to construct low-degree proofs of inequalities relating parameter distance (mean and covariance) to total variation distance.
• It relies on certifiable anti-concentration of linear projections and certifiable hypercontractivity of degree-2 polynomials to ensure robustness to adversarial outliers.
• The method proves a low-degree SoS version of the matrix inequality $\|AB\|_F^2 \leq \|A\|_{op}^2 \|B\|_F^2$, with a constant-factor loss, using Gaussian hypercontractivity.
• Cancellation lemmas in the SoS framework allow simplifying inequalities involving powers of indeterminates, enabling robust inference under constraints.
• The algorithm recursively applies partial clustering to handle outliers and achieve efficient runtime in $d^{\operatorname{poly}(k/\eta)}$ time.
• It establishes robust covariance estimation in Frobenius distance as a subroutine, crucial for handling non-spherical components.

Experimental results

Research questions

• RQ1 Can an efficient, outlier-robust clustering algorithm be designed for $k$-Gaussian mixtures when components are separated only in total variation distance?
• RQ2 What analytic properties are both necessary and sufficient for efficient clustering of $k$-GMMs in the presence of adversarial outliers?
• RQ3 Can the sum-of-squares framework be extended to prove robust certifiability of TV-separated clusters using only degree-2 polynomial concentration and anti-concentration?
• RQ4 Does the information-theoretic clusterability of distributions satisfying certifiable hypercontractivity and anti-concentration hold, and can it be algorithmically realized?
• RQ5 Can the algorithm be generalized beyond Gaussians to mixtures of affine-transformed uniform distributions on the $d$-sphere?

Key findings

• The algorithm achieves $d^{\operatorname{poly}(k/\eta)}$-time clustering of $k$-GMMs with misclassification error at most $k^{O(k)}(\varepsilon + \eta)$ under TV separation.
• TV separation is the weakest known separation condition under which clustering is information-theoretically possible, and the algorithm works even when components have identical means but differ in covariance along unknown directions.
• The paper establishes that certifiable hypercontractivity and anti-concentration of degree-2 polynomials are both necessary and sufficient for efficient clustering of $k$-GMMs.
• The results extend to mixtures of arbitrary affine transforms of the uniform distribution on the $d$-dimensional unit sphere, generalizing beyond Gaussians.
• The work provides the first low-degree sum-of-squares proof of inequalities linking parameter distance to total variation distance, relying only on second-order moment and anti-concentration properties.
• The algorithm is robust to $\varepsilon$-fraction of fully adversarial outliers, and the runtime is polynomial in $d$ for fixed $k$, even when $k=2$.