[Paper Review] Alternating Minimization for Mixed Linear Regression

Xinyang Yi, Constantine Caramanis+1 more|arXiv (Cornell University)|Oct 14, 2013

Sparse and Compressive Sensing Techniques13 references78 citations

TL;DR

This paper proposes a provably convergent alternating minimization algorithm for mixed linear regression by introducing a novel spectral initialization based on the leading two eigenvectors of a data matrix. With this initialization, the EM-based algorithm converges exponentially fast to the true regression vectors using only $ O(k\log^2 k) $ samples, achieving optimal sample complexity up to logarithmic factors and providing the first theoretical guarantees for EM in this setting.

ABSTRACT

Mixed linear regression involves the recovery of two (or more) unknown vectors from unlabeled linear measurements; that is, where each sample comes from exactly one of the vectors, but we do not know which one. It is a classic problem, and the natural and empirically most popular approach to its solution has been the EM algorithm. As in other settings, this is prone to bad local minima; however, each iteration is very fast (alternating between guessing labels, and solving with those labels). In this paper we provide a new initialization procedure for EM, based on finding the leading two eigenvectors of an appropriate matrix. We then show that with this, a re-sampled version of the EM algorithm provably converges to the correct vectors, under natural assumptions on the sampling distribution, and with nearly optimal (unimprovable) sample complexity. This provides not only the first characterization of EM's performance, but also much lower sample complexity as compared to both standard (randomly initialized) EM, and other methods for this problem.

Motivation & Objective

• To address the lack of theoretical guarantees for the EM algorithm in mixed linear regression, which is prone to local minima despite its empirical popularity.
• To develop a provably correct initialization scheme that enables EM to converge to the global optimum.
• To achieve nearly optimal sample complexity for recovering two unknown regression vectors from unlabeled linear measurements.
• To establish the first analytical performance bounds for the EM algorithm in the context of mixed linear regression.

Proposed method

• Propose a spectral initialization that computes the leading two eigenvectors of a matrix constructed from the data, which serves as a good initial estimate of the true regression vectors.
• Use this initialization to seed a re-sampled version of the EM algorithm, alternating between label assignment and regression coefficient updates.
• Apply concentration inequalities and random matrix theory to bound the error in each EM iteration, showing exponential convergence.
• Derive bounds on the minimum singular values of submatrices formed by correctly and incorrectly labeled samples to control estimation error.
• Use Hoeffding’s inequality and standard concentration results to control the number of correct and incorrect label assignments at each iteration.
• Establish that the error halves in each iteration under mild assumptions, leading to exponential convergence to the true vectors.

Experimental results

Research questions

• RQ1 Can a spectral initialization be designed to make EM provably converge to the global optimum in mixed linear regression?
• RQ2 What is the minimum number of samples required for EM with this initialization to recover the true regression vectors with high probability?
• RQ3 Does the proposed method achieve optimal or near-optimal sample complexity compared to existing approaches?
• RQ4 Can theoretical performance guarantees be established for EM in mixed linear regression, which has previously lacked such analysis?

Key findings

• The proposed spectral initialization enables the EM algorithm to converge exponentially fast to the true regression vectors under natural sampling assumptions.
• The algorithm achieves exact recovery with $ O(k\log^2 k) $ samples, which is optimal up to logarithmic factors in the dimension $ k $.
• The sample complexity is significantly lower than that of standard EM and other existing methods, which require more samples or lack theoretical guarantees.
• The error in the estimated regression vectors is shown to decrease by at least half in each EM iteration, leading to exponential convergence.
• Theoretical analysis confirms that the method recovers the ground truth exactly with high probability under the noiseless setting.
• The method provides the first analytical performance characterization of EM for mixed linear regression, resolving a long-standing open problem.