[Paper Review] Polynomial Learning of Distribution Families

Mikhail Belkin, K. P. Sinha|arXiv (Cornell University)|Apr 27, 2010

Machine Learning and Algorithms16 references143 citations

TL;DR

This paper resolves the long-standing open problem of polynomial-time learning of high-dimensional Gaussian mixture models with an arbitrary fixed number of components, without requiring separation between components. It introduces polynomial families—distributions with polynomial moments in parameters—and uses tools from real algebraic geometry to show that parameters of such families can be learned in polynomial time and sample complexity, enabling efficient learning of Gaussian mixtures via dimensionality reduction and moment-based estimation.

ABSTRACT

The question of polynomial learnability of probability distributions, particularly Gaussian mixture distributions, has recently received significant attention in theoretical computer science and machine learning. However, despite major progress, the general question of polynomial learnability of Gaussian mixture distributions still remained open. The current work resolves the question of polynomial learnability for Gaussian mixtures in high dimension with an arbitrary fixed number of components. The result on learning Gaussian mixtures relies on an analysis of distributions belonging to what we call "polynomial families" in low dimension. These families are characterized by their moments being polynomial in parameters and include almost all common probability distributions as well as their mixtures and products. Using tools from real algebraic geometry, we show that parameters of any distribution belonging to such a family can be learned in polynomial time and using a polynomial number of sample points. The result on learning polynomial families is quite general and is of independent interest. To estimate parameters of a Gaussian mixture distribution in high dimensions, we provide a deterministic algorithm for dimensionality reduction. This allows us to reduce learning a high-dimensional mixture to a polynomial number of parameter estimations in low dimension. Combining this reduction with the results on polynomial families yields our result on learning arbitrary Gaussian mixtures in high dimensions.

Motivation & Objective

• To resolve the open problem of polynomial-time learning of Gaussian mixture models in high dimensions without separation assumptions.
• To develop a general framework for learning distributions in polynomial families, where moments are polynomial in parameters.
• To establish that parameters of any distribution in a polynomial family can be learned in polynomial time and sample complexity.
• To apply this framework to high-dimensional Gaussian mixtures through deterministic dimensionality reduction and iterative parameter estimation.
• To extend the method to learn products of low-dimensional distributions and other structured families.

Proposed method

• The paper introduces the concept of polynomial families, defined by moments that are polynomial functions of the parameters.
• It uses tools from real algebraic geometry to prove that parameters of distributions in such families can be identified and estimated using the method of moments.
• A deterministic dimensionality reduction technique is developed to project high-dimensional mixtures into low-dimensional subspaces where parameters can be estimated efficiently.
• The algorithm estimates the radius of identifiability in projections and selects a subspace that preserves sufficient information for parameter recovery.
• Iterative estimation is performed by extending the subspace one coordinate at a time, using projections to estimate missing mean and covariance entries.
• Corollary 2.11 is applied to estimate mixing weights, projected means, and covariance minors in increasing subspaces, ensuring convergence to full parameter recovery.

Experimental results

Research questions

• RQ1 Can Gaussian mixture models in high dimensions be learned in polynomial time without requiring separation between components?
• RQ2 Are there general classes of distributions for which parameter estimation via the method of moments is polynomial-time feasible?
• RQ3 Can algebraic geometry tools be used to establish identifiability and efficient estimation for parametric families with polynomial moments?
• RQ4 Is it possible to reduce high-dimensional parameter estimation to a sequence of low-dimensional estimations with bounded error?
• RQ5 Can the framework be extended to learn products of distributions or other structured families beyond Gaussian mixtures?

Key findings

• The paper establishes that any distribution in a polynomial family—encompassing most common distributions and their mixtures—can be learned in polynomial time and with a polynomial number of samples.
• The method achieves polynomial learnability of Gaussian mixtures in high dimensions without any separation assumptions, even when components have identical means but different covariances.
• The algorithm uses a deterministic dimensionality reduction step that preserves the radius of identifiability within a factor of 1/n, enabling stable parameter recovery.
• The number of required samples and runtime are polynomial in the dimension n and the number of components k, with no dependence on separation parameters.
• The framework applies to learning products of distributions, such as n-dimensional mixtures formed from k-component mixtures of d-dimensional Gaussians.
• The results are of independent interest and open new avenues for applying algebraic geometry to statistical learning and moment-based inference.