[Paper Review] Learning GMMs with Nearly Optimal Robustness Guarantees

Allen Liu, Ankur Moitra|arXiv (Cornell University)|Apr 19, 2021

Machine Learning and Algorithms35 references4 citations

TL;DR

This paper presents a computationally efficient algorithm for robustly learning high-dimensional Gaussian mixture models (GMMs) with $k$ components from $\epsilon$-corrupted data, achieving $\widetilde{O}(\epsilon)$ total variation error—nearly optimal up to logarithmic factors. The key innovation is a new framework called 'strong observability' that bypasses the need to learn individual components by using test functions whose measurements uniquely and robustly identify the full mixture distribution.

ABSTRACT

In this work we solve the problem of robustly learning a high-dimensional Gaussian mixture model with $k$ components from $ε$-corrupted samples up to accuracy $\widetilde{O}(ε)$ in total variation distance for any constant $k$ and with mild assumptions on the mixture. This robustness guarantee is optimal up to polylogarithmic factors. The main challenge is that most earlier works rely on learning individual components in the mixture, but this is impossible in our setting, at least for the types of strong robustness guarantees we are aiming for. Instead we introduce a new framework which we call {\em strong observability} that gives us a route to circumvent this obstacle.

Motivation & Objective

• To achieve nearly optimal robustness guarantees in learning high-dimensional Gaussian mixture models under adversarial corruption.
• To overcome the fundamental limitation that component-wise learning is impossible under strong robustness constraints.
• To develop a new algorithmic framework that enables robust density estimation without requiring individual component recovery.
• To establish that $\widetilde{O}(\epsilon)$ total variation error is achievable for any constant $k$ under mild assumptions.
• To provide a computationally efficient solution with dimension-independent robustness guarantees, improving upon prior $\epsilon^{\Omega_k(1)}$ error bounds.

Proposed method

• Introduce the concept of 'strong observability'—a framework where a family of distributions is uniquely determined by measurements of a set of test functions.
• Use test functions whose discrepancy bounds total variation distance up to polylogarithmic factors, enabling robust estimation.
• Apply a linear transformation to normalize the initial estimate of the first component, reducing the problem to a regularized form.
• Cluster components based on proximity in the transformed space and apply a robust learning algorithm to each connected component.
• Construct candidate density functions for each submixture and use hypothesis testing to select the best approximation from a list of candidates.
• Leverage a hypothesis testing lemma from prior work to identify a candidate within $O(\epsilon)$ total variation distance with high probability.

Experimental results

Research questions

• RQ1 Can we achieve $\widetilde{O}(\epsilon)$ total variation error in learning a $k$-component GMM under $\epsilon$-corruption, for any constant $k$?
• RQ2 Is it possible to design a robust learning algorithm that does not require learning individual components, especially when such recovery is information-theoretically impossible?
• RQ3 Can we construct a framework that ensures robustness guarantees independent of dimension while maintaining computational efficiency?
• RQ4 How can we ensure that test function measurements serve as accurate proxies for total variation distance in high-dimensional mixtures?
• RQ5 Can we achieve nearly optimal robustness in improper density estimation for GMMs, even when component-wise learning fails?

Key findings

• The proposed algorithm achieves $\widetilde{O}(\epsilon)$ total variation error in learning a $k$-component GMM from $\epsilon$-corrupted samples, which is optimal up to polylogarithmic factors.
• The framework of strong observability enables robust learning without requiring component-wise estimation, circumventing a major barrier in prior work.
• The algorithm runs in time polynomial in the sample size and $1/\epsilon$, with the number of candidate hypotheses bounded by $ (1/\epsilon)^{O_{k,A}(1)} $.
• The hypothesis testing step ensures that with high probability, one of the candidate functions is within $O(\epsilon)$ total variation distance of the true mixture.
• The method relies on transforming the mixture into a regularized form where known robust learning techniques can be applied, followed by inverse transformation and weight estimation.
• The final output is a density function that approximates the true GMM within $ (2 + \log 1/\epsilon + \chi)^{O_{k,A}(1)} \epsilon $ in $L^1$-norm, which translates to $\widetilde{O}(\epsilon)$ error under the stated assumptions.