[Paper Review] Convergence Analysis of Gradient EM for Multi-component Gaussian Mixture

Bowei Yan, Mingzhang Yin+1 more|arXiv (Cornell University)|May 23, 2017

Bayesian Methods and Mixture Models24 references3 citations

TL;DR

This paper provides a convergence analysis of the gradient EM algorithm for multi-component Gaussian Mixture Models (GMMs) with arbitrary numbers of components and known mixing coefficients. It derives a near-optimal local contraction radius and convergence rate dependent on mixing coefficients, pairwise center distances, and dimensionality, using tools from learning theory and empirical processes to establish non-asymptotic guarantees under good initialization.

ABSTRACT

In this paper, we study convergence properties of the gradient Expectation-Maximization algorithm \cite{lange1995gradient} for Gaussian Mixture Models for general number of clusters and mixing coefficients. We derive the convergence rate depending on the mixing coefficients, minimum and maximum pairwise distances between the true centers and dimensionality and number of components; and obtain a near-optimal local contraction radius. While there have been some recent notable works that derive local convergence rates for EM in the two equal mixture symmetric GMM, in the more general case, the derivations need structurally different and non-trivial arguments. We use recent tools from learning theory and empirical processes to achieve our theoretical results.

Motivation & Objective

• To understand the convergence behavior of gradient EM in general multi-component Gaussian Mixture Models beyond the two-component symmetric case.
• To derive non-asymptotic convergence rates and local contraction radii that depend on model parameters such as mixing coefficients, center separations, and dimensionality.
• To establish a near-optimal contraction region for gradient EM, improving upon prior work with suboptimal radii.
• To provide theoretical guarantees that extend to Stochastic Gradient EM via sample-based uniform concentration bounds.

Proposed method

• The analysis uses empirical process theory and concentration inequalities to bound the uniform deviation between population and empirical gradient operators.
• A covering argument with $ e^{2d} $-net is employed to control the supremum of gradient differences over a parameter region.
• The paper introduces a uniform deviation bound $ \epsilon^{\text{unif}}(n) \propto M^{3/2}(1+3R_{\max})^3 \max\{1,\log \kappa\} \sqrt{\frac{d \log n}{n}} $ for the difference between true and empirical gradients.
• It leverages a sub-Gaussian concentration argument for the gradient difference, relying on boundedness of the gradient components via $ R_{\max} $.
• The contraction region is defined as $ \mathbb{A} = \prod_{i=1}^M \mathbb{B}(\bm{\mu}_i^*, a) $, and the analysis ensures convergence within this set under good initialization.
• A probabilistic initialization bound is derived using combinatorial arguments and tail bounds, showing that $ \mathcal{O}\left( \frac{\log(1/\delta)}{\sqrt{2\pi M}} \left( \frac{e}{1 - e^{-a\sqrt{d}/2}} \right)^M \right) $ initializations suffice to achieve a good start with high probability.

Experimental results

Research questions

• RQ1 What is the convergence rate of gradient EM for general multi-component GMMs with arbitrary mixing coefficients and component centers?
• RQ2 How does the local contraction radius of gradient EM scale with the number of components, dimensionality, and minimum separation between centers?
• RQ3 Can a near-optimal contraction radius be established for gradient EM in the general GMM setting, beyond the two-equal-component case?
• RQ4 What is the required number of initializations to ensure a good starting point with high probability in high-dimensional GMMs?
• RQ5 How do sample-based uniform concentration bounds enable non-asymptotic convergence guarantees for gradient EM and its stochastic variant?

Key findings

• The convergence rate of gradient EM is shown to depend on the mixing coefficients, the minimum and maximum pairwise distances between true centers, and the dimensionality and number of components.
• A near-optimal local contraction radius is derived, improving upon the suboptimal radius in prior work for two-equal-component GMMs.
• The uniform deviation between population and empirical gradients is bounded by $ \epsilon^{\text{unif}}(n) = cM^{3/2}(1+3R_{\max})^3\max\{1,\log(\kappa)\}\sqrt{\frac{d\log n}{n}} $ with high probability.
• With high probability, the gradient EM algorithm converges linearly within the contraction region $ \mathbb{A} $, provided the initial estimate lies within a neighborhood of the true parameters.
• The required number of initializations to achieve a good start is bounded by $ \mathcal{O}\left( \frac{\log(1/\delta)}{\sqrt{2\pi M}} \left( \frac{e}{1 - e^{-a\sqrt{d}/2}} \right)^M \right) $, ensuring a high-probability good initialization.
• The theoretical framework immediately extends to Stochastic Gradient EM, providing non-asymptotic convergence guarantees under the same conditions.