[Paper Review] Minimax Theory for High-dimensional Gaussian Mixtures with Sparse Mean Separation

Martin Azizyan, Aarti Singh+1 more|arXiv (Cornell University)|Jun 9, 2013

Bayesian Methods and Mixture Models12 references36 citations

TL;DR

This paper establishes sharp minimax lower bounds for clustering Gaussian mixtures in high-dimensional settings with sparse mean separation. It shows that sample complexity depends only on the number of relevant (sparse) dimensions and mean separation, and that a simple, computationally efficient procedure nearly achieves the information-theoretic limit, providing theoretical justification for feature selection in clustering.

ABSTRACT

While several papers have investigated computationally and statistically efficient methods for learning Gaussian mixtures, precise minimax bounds for their statistical performance as well as fundamental limits in high-dimensional settings are not well-understood. In this paper, we provide precise information theoretic bounds on the clustering accuracy and sample complexity of learning a mixture of two isotropic Gaussians in high dimensions under small mean separation. If there is a sparse subset of relevant dimensions that determine the mean separation, then the sample complexity only depends on the number of relevant dimensions and mean separation, and can be achieved by a simple computationally efficient procedure. Our results provide the first step of a theoretical basis for recent methods that combine feature selection and clustering.

Motivation & Objective

• To establish precise information-theoretic bounds on clustering accuracy and sample complexity for high-dimensional Gaussian mixtures with small mean separation.
• To analyze the statistical performance of clustering in settings where only a sparse subset of dimensions contribute to mean separation between components.
• To demonstrate that a simple, computationally efficient procedure nearly achieves the information-theoretic sample complexity in sparse mean separation settings.
• To provide theoretical justification for combining feature selection with clustering in high-dimensional unsupervised learning.
• To resolve the misconception that there is a gap between statistical and computational complexity in learning two-component isotropic Gaussian mixtures under small mean separation.

Proposed method

• Formulates the clustering problem as minimizing the probability of misclustering relative to the Bayes optimal classifier, using a loss function that compares to the optimal clustering under the true distribution.
• Derives minimax lower bounds using a non-standard approach due to the loss function not satisfying the triangle inequality, relying on Le Cam's method and Fano-type inequalities.
• Applies a novel KL divergence bound between mixture distributions using geometric arguments involving angles between mean vectors, establishing KL(Pθ, Pθ') ≤ ξ⁴(1 − cos β) with ξ = ||μ||/(2σ).
• Constructs a finite set of parameter configurations (θω) with controlled pairwise KL divergences and misclassification losses to apply Fano’s inequality and derive lower bounds.
• Uses combinatorial constructions (e.g., Hamming balls) to ensure sufficient separation between hypotheses while keeping KL divergence bounded.
• Analyzes both non-sparse and sparse mean separation settings, with the sparse case restricting mean differences to s ≤ d dimensions, and derives bounds that scale with s, not d.

Experimental results

Research questions

• RQ1 What is the fundamental statistical limit (minimax risk) for clustering two isotropic Gaussian components in high dimensions with small mean separation?
• RQ2 How does the sample complexity scale when only a sparse subset of dimensions (s) contribute to the mean separation, rather than all d dimensions?
• RQ3 Can a computationally efficient procedure achieve the information-theoretic sample complexity in the sparse mean separation setting?
• RQ4 Is there a gap between statistical and computational complexity in learning two-component Gaussian mixtures under small mean separation?
• RQ5 To what extent does feature selection improve clustering performance in high-dimensional settings with sparse mean differences?

Key findings

• For the non-sparse case, the minimax expected misclustering loss satisfies: inf_Fn sup_θ∈Θλ EθLθ(Fn) ≥ (1/500) min(√(log 2)/3 * (σ²/λ²) * √((d−1)/n), 1/4), showing dependence on d and n.
• For the sparse case with s relevant dimensions, the minimax risk is bounded below by (1/600) min(√(8/45) * (σ²/λ²) * √(s/(s−1)) * √(n⁻¹ log((d−1)/(s−1))), 1/2), indicating sample complexity depends only on s, not d.
• The lower bounds match known sample complexity requirements of existing algorithms up to logarithmic factors, validating the tightness of the theoretical limits.
• A simple, computationally efficient procedure nearly achieves the information-theoretic sample complexity in the sparse mean separation setting, demonstrating that feature selection is statistically beneficial.
• The results debunk the myth that statistical and computational complexity are fundamentally mismatched for learning two-component isotropic Gaussian mixtures under small mean separation.
• The loss function used—misclustering probability relative to the Bayes optimal classifier—provides a meaningful benchmark that tends to zero as sample size increases, unlike alternative loss functions.