[Paper Review] Convergence of Parameter Estimates for Regularized Mixed Linear Regression Models

Taiyao Wang, Ioannis Ch. Paschalidis|arXiv (Cornell University)|Mar 21, 2019

Statistical Methods and Inference28 references4 citations

TL;DR

This paper proposes a mixed-integer programming (MIP) formulation for regularized mixed linear regression (MLR) with convergence guarantees. It establishes almost sure convergence of parameter estimates to the true coefficients in the noiseless case and under cluster separability, while showing convergence under martingale difference noise for single-cluster models. For multiple clusters with noise, a counterexample demonstrates that strong consistency may fail due to parameter non-identifiability.

ABSTRACT

We consider {\em Mixed Linear Regression (MLR)}, where training data have been generated from a mixture of distinct linear models (or clusters) and we seek to identify the corresponding coefficient vectors. We introduce a {\em Mixed Integer Programming (MIP)} formulation for MLR subject to regularization constraints on the coefficient vectors. We establish that as the number of training samples grows large, the MIP solution converges to the true coefficient vectors in the absence of noise. Subject to slightly stronger assumptions, we also establish that the MIP identifies the clusters from which the training samples were generated. In the special case where training data come from a single cluster, we establish that the corresponding MIP yields a solution that converges to the true coefficient vector even when training data are perturbed by (martingale difference) noise. We provide a counterexample indicating that in the presence of noise, the MIP may fail to produce the true coefficient vectors for more than one clusters. We also provide numerical results testing the MIP solutions in synthetic examples with noise.

Motivation & Objective

• To establish strong consistency of parameter estimates for mixed linear regression models under general noise and feature conditions.
• To develop a mixed-integer programming (MIP) formulation that incorporates norm-based regularization for MLR.
• To analyze convergence of MIP solutions to true coefficient vectors as sample size increases.
• To investigate the conditions under which MIP can correctly identify cluster memberships in MLR.
• To examine the robustness of the MIP approach under noise, particularly in multi-cluster settings.

Proposed method

• Formulates mixed linear regression as a mixed-integer program (MIP) with regularization constraints on coefficient vectors.
• Uses a MIP framework to jointly estimate cluster assignments and regression coefficients.
• Applies techniques from strong consistency theory of least-squares estimates, adapting them to the MIP formulation under weaker assumptions.
• Employs a counterexample to demonstrate failure of strong consistency in multi-cluster MLR under noise due to parameter non-identifiability.
• Conducts numerical experiments using GUROBI 8.0 and Python on synthetic data with Gaussian and uniform noise.
• Evaluates convergence via the norm difference $\|\boldsymbol{\beta}^n_k - \boldsymbol{\beta}_k\|$ as a function of sample size $n$.

Experimental results

Research questions

• RQ1 Does the proposed MIP formulation for regularized MLR yield parameter estimates that converge almost surely to the true coefficients in the noiseless case?
• RQ2 Under what conditions can the MIP solution correctly identify the cluster membership of each training sample?
• RQ3 Can the MIP approach maintain strong consistency when data are perturbed by martingale difference noise, particularly in the single-cluster case?
• RQ4 Why does the MIP fail to recover true parameters in multi-cluster models under noise, and what structural limitations cause this?
• RQ5 How does the convergence rate of MIP estimates behave under different noise distributions and cluster separability?

Key findings

• In the noiseless case, the MIP solution converges almost surely to the true coefficient vectors as the number of training samples increases.
• Under cluster separability assumptions, the MIP solution correctly identifies the cluster from which each sample was generated.
• For a single cluster with martingale difference noise, the MIP solution converges to the true coefficient vector under weak distributional assumptions.
• A counterexample shows that in multi-cluster models with noise, the MIP may fail to recover the true parameters due to non-identifiability, even when the objective function value is minimized.
• Numerical results confirm that convergence is nearly equivalent to separate linear regression when clusters are well-separated, even under Gaussian and uniform noise.
• The convergence rate of parameter estimates improves with increasing sample size, and the MIP formulation outperforms standard EM-type methods in terms of consistency guarantees.