[Paper Review] Alternating Minimization Converges Super-Linearly for Mixed Linear Regression

Avishek Ghosh, Kannan Ramchandran|arXiv (Cornell University)|Apr 23, 2020

Advanced Statistical Methods and Models28 references4 citations

TL;DR

This paper establishes the first theoretical super-linear convergence rate of Alternating Minimization (AM) for mixed linear regression with two components, showing that AM converges in O(log log(1/ε)) iterations to achieve ε error in ℓ₂ norm when properly initialized. The analysis reveals that AM's empirical speedup over gradient methods is theoretically justified by its faster-than-linear convergence in specific parameter regimes.

ABSTRACT

We address the problem of solving mixed random linear equations. We have unlabeled observations coming from multiple linear regressions, and each observation corresponds to exactly one of the regression models. The goal is to learn the linear regressors from the observations. Classically, Alternating Minimization (AM) (which is a variant of Expectation Maximization (EM)) is used to solve this problem. AM iteratively alternates between the estimation of labels and solving the regression problems with the estimated labels. Empirically, it is observed that, for a large variety of non-convex problems including mixed linear regression, AM converges at a much faster rate compared to gradient based algorithms. However, the existing theory suggests similar rate of convergence for AM and gradient based methods, failing to capture this empirical behavior. In this paper, we close this gap between theory and practice for the special case of a mixture of $2$ linear regressions. We show that, provided initialized properly, AM enjoys a \emph{super-linear} rate of convergence in certain parameter regimes. To the best of our knowledge, this is the first work that theoretically establishes such rate for AM. Hence, if we want to recover the unknown regressors upto an error (in $\ell_2$ norm) of $ε$, AM only takes $\mathcal{O}(\log \log (1/ε))$ iterations. Furthermore, we compare AM with a gradient based heuristic algorithm empirically and show that AM dominates in iteration complexity as well as wall-clock time.

Motivation & Objective

• To bridge the theoretical gap between the observed fast convergence of Alternating Minimization (AM) in practice and its slow convergence rates predicted by existing theory.
• To analyze the convergence behavior of AM for the special case of a mixture of two linear regressions.
• To establish that AM achieves super-linear convergence under proper initialization, explaining its empirical superiority over gradient-based methods.
• To provide theoretical justification for the observed fast convergence of AM in non-convex problems with latent variables, such as mixed linear regression.

Proposed method

• The authors analyze the AM algorithm by modeling its iterates through an empirical process, focusing on the label estimation and regression update steps.
• They derive a high-probability upper bound on the number of samples assigned to each component using sub-Gaussian and sub-exponential concentration inequalities.
• The analysis leverages spectral properties of Gaussian random matrices to convert prediction error into estimation error.
• A key step involves bounding the quadratic form ∑⟨xᵢ, u⟩² over the set of misclassified samples, using sub-exponential tail bounds.
• The proof uses Lemma 1 to control the size of the misclassified set, ensuring the error shrinks cubically with distance.
• By combining error propagation and spectral lower bounds, the authors show that the estimation error decreases as O(dist³), implying super-linear convergence.

Experimental results

Research questions

• RQ1 Does Alternating Minimization (AM) exhibit super-linear convergence in the mixed linear regression problem with two components, despite existing theory suggesting only linear convergence?
• RQ2 Why does AM converge significantly faster than gradient-based methods in practice, especially in non-convex settings with latent variables?
• RQ3 Can the empirical observation of rapid convergence in AM be theoretically justified for the two-component mixed linear regression model?
• RQ4 What role does proper initialization play in enabling super-linear convergence of AM?
• RQ5 Is the observed convergence exponent of approximately 1.75–1.8 in experiments consistent with a theoretical super-linear rate?

Key findings

• AM achieves a super-linear convergence rate for the two-component mixed linear regression problem when initialized properly, with convergence in O(log log(1/ε)) iterations to reach ε error in ℓ₂ norm.
• The estimation error decreases as O(dist³), where dist is the ℓ₂ distance to the true parameters, confirming super-linear convergence.
• The theoretical analysis shows that the number of misclassified samples is tightly controlled via sub-Gaussian and sub-exponential concentration bounds.
• The spectral norm of the design matrix ensures that prediction error translates to estimation error with a favorable lower bound, enabling the cubic decay in error.
• Empirical results confirm that AM outperforms gradient-based methods in both iteration count and wall-clock time, validating the theoretical speedup.
• The convergence exponent observed in experiments (1.75–1.8) is consistent with super-linear behavior, though the theoretical analysis suggests a cubic decay rate.