[Paper Review] Efficient Algorithms for Estimating the Parameters of Mixed Linear Regression Models

Babak Barazandeh, Ali Ghafelebashi+2 more|arXiv (Cornell University)|May 12, 2021

Statistical Methods and Inference33 references4 citations

TL;DR

This paper proposes a novel ADMM-EM hybrid algorithm for maximum likelihood estimation in mixed linear regression (MLR) models with non-Gaussian noise, specifically Laplacian noise, where traditional EM fails due to lack of closed-form updates. The method reformulates the MLE problem to enable closed-form updates in each ADMM sub-problem, achieving superior statistical accuracy and computational efficiency over EM in non-Gaussian settings.

ABSTRACT

Mixed linear regression (MLR) model is among the most exemplary statistical tools for modeling non-linear distributions using a mixture of linear models. When the additive noise in MLR model is Gaussian, Expectation-Maximization (EM) algorithm is a widely-used algorithm for maximum likelihood estimation of MLR parameters. However, when noise is non-Gaussian, the steps of EM algorithm may not have closed-form update rules, which makes EM algorithm impractical. In this work, we study the maximum likelihood estimation of the parameters of MLR model when the additive noise has non-Gaussian distribution. In particular, we consider the case that noise has Laplacian distribution and we first show that unlike the the Gaussian case, the resulting sub-problems of EM algorithm in this case does not have closed-form update rule, thus preventing us from using EM in this case. To overcome this issue, we propose a new algorithm based on combining the alternating direction method of multipliers (ADMM) with EM algorithm idea. Our numerical experiments show that our method outperforms the EM algorithm in statistical accuracy and computational time in non-Gaussian noise case.

Motivation & Objective

• To address the limitation of the EM algorithm in mixed linear regression when noise is non-Gaussian, such as Laplacian, where EM sub-problems lack closed-form solutions.
• To develop a scalable and efficient optimization method for maximum likelihood estimation in MLR models under non-Gaussian noise.
• To overcome the computational bottleneck of EM in Laplacian noise scenarios by combining ADMM with EM-inspired updates.
• To provide a numerically stable and fast algorithm that maintains high statistical accuracy in high-dimensional and large-scale MLR problems.

Proposed method

• Reformulates the maximum likelihood estimation (MLE) objective for mixed linear regression to make it amenable to ADMM optimization.
• Introduces a variable splitting technique to decouple the estimation of component weights and regression coefficients.
• Uses ADMM to solve the reformulated problem, ensuring each sub-problem has a closed-form update, enabling efficient computation.
• Combines the EM idea of soft assignment (via posterior probabilities) with ADMM’s alternating minimization framework.
• Employs a dual ascent step with penalty parameter ρ to enforce consensus between variables, ensuring convergence.
• Derives closed-form updates for the regression coefficients and auxiliary variables in both Gaussian and Laplacian noise cases.

Experimental results

Research questions

• RQ1 Can we develop an efficient algorithm for maximum likelihood estimation in mixed linear regression when the noise is non-Gaussian, such as Laplacian?
• RQ2 Why does the standard EM algorithm fail to provide closed-form updates under Laplacian noise, and can this be overcome?
• RQ3 Can ADMM be effectively combined with EM-like iterative updates to solve non-convex MLR problems with non-Gaussian noise?
• RQ4 Does the proposed hybrid ADMM-EM algorithm outperform EM in terms of estimation accuracy and computational speed under Laplacian noise?
• RQ5 Is the proposed method scalable and robust across varying numbers of components and dimensions in mixed linear regression?

Key findings

• The proposed ADMM-EM hybrid algorithm achieves significantly lower recovery error than EM in the Laplacian noise case, with a paired t-test rejecting the hypothesis that EM performs better (p < 0.05).
• The algorithm outperforms EM in computational efficiency, with EM taking 1.6 to 10 minutes longer per run under Laplacian noise due to solving expensive linear programs in each EM step.
• The proposed method maintains closed-form updates in all sub-problems, enabling fast and scalable optimization even in high-dimensional settings.
• In the Gaussian noise case, both methods perform comparably in accuracy, but the proposed algorithm remains faster due to its efficient update rules.
• Numerical experiments with 30 repetitions across 1950 data points confirm the robustness and consistency of the proposed method across varying K (2–14 components) and d (1–5 dimensions).
• The algorithm converges reliably and achieves high statistical accuracy, with average recovery error (mean ± std) reported in tables for both Gaussian and Laplacian noise scenarios.