[Paper Review] A Nearly Optimal and Agnostic Algorithm for Properly Learning a Mixture of k Gaussians, for any Constant k

Jerry Zheng Li, Ludwig Schmidt|arXiv (Cornell University)|Jun 3, 2015

Machine Learning and Algorithms29 references12 citations

TL;DR

This paper presents a nearly optimal and agnostic algorithm for properly learning a mixture of $k$ univariate Gaussians, achieving sample complexity $\widetilde{O}(k/\epsilon^2)$ and running time $(k \log \frac{1}{\epsilon})^{O(k^4)} + \widetilde{O}(k/\epsilon^2)$, which is nearly linear in the number of samples for fixed $k$. The method uses density estimation followed by fitting a Gaussian mixture to the estimated density via polynomial system solving, enabling agnostic learning even when the true distribution is not a Gaussian mixture.

ABSTRACT

Learning a Gaussian mixture model (GMM) is a fundamental problem in machine learning, learning theory, and statistics. One notion of learning a GMM is proper learning: here, the goal is to find a mixture of $k$ Gaussians $\mathcal{M}$ that is close to the density $f$ of the unknown distribution from which we draw samples. The distance between $\mathcal{M}$ and $f$ is typically measured in the total variation or $L_1$-norm. We give an algorithm for learning a mixture of $k$ univariate Gaussians that is nearly optimal for any fixed $k$. The sample complexity of our algorithm is $ ilde{O}(\frac{k}{ε^2})$ and the running time is $(k \cdot \log\frac{1}ε)^{O(k^4)} + ilde{O}(\frac{k}{ε^2})$. It is well-known that this sample complexity is optimal (up to logarithmic factors), and it was already achieved by prior work. However, the best known time complexity for proper learning a $k$-GMM was $ ilde{O}(\frac{1}{ε^{3k-1}})$. In particular, the dependence between $\frac{1}ε$ and $k$ was exponential. We significantly improve this dependence by replacing the $\frac{1}ε$ term with a $\log \frac{1}ε$ while only increasing the exponent moderately. Hence, for any fixed $k$, the $ ilde{O} (\frac{k}{ε^2})$ term dominates our running time, and thus our algorithm runs in time which is nearly-linear in the number of samples drawn. Achieving a running time of $ extrm{poly}(k, \frac{1}ε)$ for proper learning of $k$-GMMs has recently been stated as an open problem by multiple researchers, and we make progress on this question. Moreover, our approach offers an agnostic learning guarantee: our algorithm returns a good GMM even if the distribution we are sampling from is not a mixture of Gaussians. To the best of our knowledge, our algorithm is the first agnostic proper learning algorithm for GMMs.

Motivation & Objective

• To develop a proper learning algorithm for mixtures of $k$ univariate Gaussians with nearly optimal sample and time complexity.
• To resolve the open problem of achieving $\text{poly}(k, 1/\epsilon)$ time complexity for proper learning of $k$-GMMs.
• To provide an agnostic learning guarantee, ensuring good performance even when the true distribution is not a Gaussian mixture.
• To generalize the approach to other parametric univariate distributions such as exponentials and Laplace distributions.

Proposed method

• The algorithm begins with a nonparametric density estimation step to obtain a close approximation of the true distribution's density.
• It then formulates the problem of fitting a $k$-Gaussian mixture to the estimated density as a deterministic optimization problem.
• The optimization is reduced to solving a sequence of systems of polynomial inequalities using Renegar’s algorithm.
• The method leverages the fact that differences of $k$-Gaussian densities have bounded zero crossings and that densities can be approximated by piecewise polynomials with polynomial dependence on parameters.
• Robust parametrization ensures that small parameter perturbations lead to small $L_1$-distance changes in the density.
• The framework generalizes to other distributions (e.g., exponentials, Laplace) by verifying structural conditions on zero crossings, parametrization, and polynomial approximation.

Experimental results

Research questions

• RQ1 Can we achieve nearly optimal sample complexity and nearly-linear running time for proper learning of a $k$-Gaussian mixture for any fixed $k$?
• RQ2 Can we design a proper learning algorithm that remains effective even when the true distribution is not a Gaussian mixture, i.e., an agnostic learning algorithm?
• RQ3 Is it possible to reduce the time complexity dependence on $1/\epsilon$ from exponential to logarithmic while maintaining proper learning guarantees?
• RQ4 Can the proposed method be generalized to other parametric families of univariate distributions beyond Gaussians?

Key findings

• The algorithm achieves sample complexity $\widetilde{O}(k/\epsilon^2)$, which is optimal up to logarithmic factors.
• The running time is $(k \log \frac{1}{\epsilon})^{O(k^4)} + \widetilde{O}(k/\epsilon^2)$, significantly improving over the prior $\widetilde{O}(1/\epsilon^{3k-1})$ bound.
• The algorithm provides an agnostic learning guarantee, returning a good $k$-Gaussian mixture even if the data does not come from a Gaussian mixture.
• The method generalizes to mixtures of $k$ exponential and $k$ Laplace distributions, achieving the same sample and time complexity bounds.
• The framework is the first to achieve nearly-linear running time in the number of samples for proper learning of $k$-GMMs with fixed $k$.
• The approach reduces proper learning to solving polynomial systems, enabling a general template for learning other parametric distributions with similar structural properties.