[Paper Review] Estimating the Coefficients of a Mixture of Two Linear Regressions by Expectation Maximization

Jason M. Klusowski, Dana Yang+1 more|arXiv (Cornell University)|Apr 26, 2017

Bayesian Methods and Mixture Models27 references3 citations

TL;DR

This paper establishes convergence guarantees for the Expectation-Maximization (EM) algorithm in estimating the coefficients of a symmetric mixture of two linear regressions. It proves that EM iterates converge to the true parameter vector at the parametric rate when initialized within an unbounded cone defined by a sufficiently large cosine angle with the target, and shows the population EM operator is anti-contractive when this angle is too small, necessitating the conical initialization condition.

ABSTRACT

We give convergence guarantees for estimating the coefficients of a symmetric mixture of two linear regressions by expectation maximization (EM). In particular, we show that the empirical EM iterates converge to the target parameter vector at the parametric rate, provided the algorithm is initialized in an unbounded cone. In particular, if the initial guess has a sufficiently large cosine angle with the target parameter vector, a sample-splitting version of the EM algorithm converges to the true coefficient vector with high probability. Interestingly, our analysis borrows from tools used in the problem of estimating the centers of a symmetric mixture of two Gaussians by EM. We also show that the population EM operator for mixtures of two regressions is anti-contractive from the target parameter vector if the cosine angle between the input vector and the target parameter vector is too small, thereby establishing the necessity of our conic condition. Finally, we give empirical evidence supporting this theoretical observation, which suggests that the sample based EM algorithm performs poorly when initial guesses are drawn accordingly. Our simulation study also suggests that the EM algorithm performs well even under model misspecification (i.e., when the covariate and error distributions violate the model assumptions).

Motivation & Objective

• To provide theoretical convergence guarantees for the EM algorithm in estimating parameters of a symmetric mixture of two linear regressions.
• To identify conditions under which EM converges to the true coefficient vector with high probability.
• To establish the necessity of a conical initialization condition by analyzing the anti-contractive behavior of the population EM operator.
• To connect the analysis to similar problems in Gaussian mixture estimation, leveraging tools from that domain.
• To validate theoretical findings with empirical evidence under model misspecification.

Proposed method

• The authors analyze the population EM operator for a symmetric two-component linear regression mixture model with i.i.d. standard normal covariates and Rademacher mixing weights.
• They prove that the population EM operator is contractive within an unbounded cone around the true parameter vector, ensuring convergence from within this region.
• The analysis uses tools from the theory of EM for Gaussian location mixtures, particularly bounding tail probabilities and expectations involving the standard normal CDF and PDF.
• Key inequalities are derived using moment generating functions and Jensen’s inequality to bound the contraction modulus and statistical error terms.
• A sample-splitting version of EM is analyzed to decouple algorithmic and statistical convergence, enabling high-probability convergence guarantees.
• The anti-contraction property of the EM operator is established when the cosine angle between the initial guess and the true parameter is too small, justifying the conical initialization condition.

Experimental results

Research questions

• RQ1 Under what conditions does the EM algorithm converge to the true parameter vector in a symmetric mixture of two linear regressions?
• RQ2 What is the rate of convergence of the empirical EM iterates, and can it achieve the parametric rate?
• RQ3 Why is initialization within a specific conical region around the true parameter necessary for convergence?
• RQ4 How does the population EM operator behave when the initial guess is poorly aligned with the true parameter?
• RQ5 Does the EM algorithm remain robust under model misspecification, such as non-Gaussian covariates or errors?

Key findings

• The empirical EM iterates converge to the true parameter vector at the parametric rate of $ O(1/ ext{sample size}) $ when initialized in an unbounded cone with a sufficiently large cosine angle to the target.
• The population EM operator is anti-contractive from the target parameter vector when the cosine angle between input and target is too small, proving the necessity of the conical initialization condition.
• A sample-splitting version of EM achieves high-probability convergence to the true parameter vector under the same conical initialization condition.
• Theoretical bounds on the contraction modulus and statistical error terms are derived using moment generating functions and tail bounds on the standard normal distribution.
• Empirical results confirm that EM performs poorly when initial guesses are drawn from regions with small cosine angles, supporting the anti-contraction analysis.
• The algorithm remains robust under model misspecification, as shown by simulation studies where EM performs well even when covariate or error distributions deviate from model assumptions.