[Paper Review] Robustly Clustering a Mixture of Gaussians

Jia He, Santosh Vempala|arXiv (Cornell University)|Nov 26, 2019

Bayesian Methods and Mixture Models27 references4 citations

TL;DR

This paper presents an efficient algorithm for robustly clustering a mixture of two Gaussians under minimal separation assumptions—either mean or covariance separation—using a novel identifiability criterion based on isotropic position and the Fisher discriminant, along with a fixed-degree Sum-of-Squares convex relaxation. The method achieves near-optimal separation requirements and extends to strongly log-concave distributions, with total variation distance separation also sufficient for clustering.

ABSTRACT

We give an efficient algorithm for robustly clustering of a mixture of two arbitrary Gaussians, a central open problem in the theory of computationally efficient robust estimation, assuming only that the the means of the component Gaussians are well-separated or their covariances are well-separated. Our algorithm and analysis extend naturally to robustly clustering mixtures of well-separated strongly logconcave distributions. The mean separation required is close to the smallest possible to guarantee that most of the measure of each component can be separated by some hyperplane (for covariances, it is the same condition in the second degree polynomial kernel). We also show that for Gaussian mixtures, separation in total variation distance suffices to achieve robust clustering. Our main tools are a new identifiability criterion based on isotropic position and the Fisher discriminant, and a corresponding Sum-of-Squares convex programming relaxation, of fixed degree.

Motivation & Objective

• To address the central open problem in robust estimation: efficiently clustering a mixture of two Gaussians under computationally feasible assumptions.
• To identify minimal separation conditions between components that still allow for reliable clustering, approaching the information-theoretic limit.
• To extend the clustering framework beyond Gaussians to well-separated strongly log-concave distributions.
• To develop a convex optimization-based method with provable guarantees using a fixed-degree Sum-of-Squares relaxation.

Proposed method

• Introduces a new identifiability criterion based on transforming data into isotropic position and applying the Fisher discriminant to separate components.
• Uses a fixed-degree Sum-of-Squares (SOS) convex programming relaxation to solve the clustering problem efficiently.
• Establishes that mean or covariance separation sufficient for hyperplane-based separation implies robust clustering is possible.
• Applies the second-degree polynomial kernel to characterize the separation condition for covariances.
• Demonstrates that total variation distance between components is sufficient for robust clustering in the Gaussian case.
• Leverages properties of strongly log-concave distributions to extend the method beyond Gaussians.

Experimental results

Research questions

• RQ1 What is the minimal separation condition between two Gaussians that still allows for efficient and robust clustering?
• RQ2 Can the Fisher discriminant in isotropic position serve as a robust identifiability criterion for Gaussian mixtures?
• RQ3 To what extent can the algorithm be generalized to mixtures of strongly log-concave distributions?
• RQ4 Does total variation distance between components suffice to guarantee robust clustering in Gaussian mixtures?
• RQ5 Can a fixed-degree Sum-of-Squares relaxation achieve provable clustering performance under minimal separation?

Key findings

• The algorithm achieves robust clustering under mean or covariance separation conditions that are close to the information-theoretic minimum.
• The identifiability criterion based on isotropic position and the Fisher discriminant enables effective separation even with minimal component separation.
• The method generalizes naturally to mixtures of well-separated strongly log-concave distributions.
• Total variation distance between component Gaussians is sufficient to guarantee robust clustering.
• The fixed-degree Sum-of-Squares relaxation provides a computationally efficient and provably correct solution to the clustering problem.
• The separation condition for covariances is equivalent to that in the second-degree polynomial kernel, linking geometric and kernel-based perspectives.