[Paper Review] Pattern Alternating Maximization Algorithm for Missing Data in Large P, Small N Problems

Nicolas Städler, Daniel J. Stekhoven+1 more|arXiv (Cornell University)|May 3, 2010

Gene expression and cancer classification29 references3 citations

TL;DR

This paper proposes MissPALasso, a computationally efficient algorithm for imputing missing values in high-dimensional 'large P, small N' data using a novel pattern alternating maximization approach. By replacing matrix inversion with soft-thresholding and incorporating Lasso penalties, it achieves sparse regression coefficients and outperforms existing methods like k-NN, nuclear norm minimization, and MissGLasso in simulations and real microarray datasets.

ABSTRACT

We propose a new and computationally efficient algorithm for maximizing the observed log-likelihood for a multivariate normal data matrix with missing values. We show that our procedure based on iteratively regressing the missing on the observed variables, generalizes the standard EM algorithm by alternating between different complete data spaces and performing the E-Step incrementally. In this non-standard setup we prove numerical convergence to a stationary point of the observed log-likelihood. For high-dimensional data, where the number of variables may greatly exceed sample size, we add a Lasso penalty in the regression part of our algorithm and perform coordinate descent approximations. This leads to a computationally very attractive technique with sparse regression coefficients for missing data imputation. Simulations and results on four microarray datasets show that the new method often outperforms other imputation techniques as k-nearest neighbors imputation, nuclear norm minimization or a penalized likelihood approach with an l1-penalty on the inverse covariance matrix.

Motivation & Objective

• Address the computational inefficiency and lack of sparsity in existing EM-based methods for missing data in high-dimensional settings.
• Develop a likelihood-based approach that generalizes the EM algorithm by alternating between different complete data spaces.
• Introduce a sparse estimation framework via Lasso penalties to improve interpretability and accuracy in 'large P, small N' scenarios.
• Prove numerical convergence of the algorithm to a stationary point of the observed log-likelihood.
• Demonstrate superior performance over k-NN, nuclear norm minimization, and penalized likelihood methods on real and simulated microarray data.

Proposed method

• Propose MissPA, a non-standard EM-type algorithm that alternates between different complete data spaces and performs incremental E-Steps.
• Replace matrix inversion in the E-Step with a soft-thresholding operator to improve computational efficiency.
• Introduce MissPALasso by adding Lasso penalties to the regression step, enabling sparse estimation of missing data imputations.
• Use coordinate descent approximations to efficiently solve the penalized regression problems in high-dimensional settings.
• Formulate the optimization as maximizing a penalized variational free energy, with iterative updates across missingness patterns.
• Prove convergence to a stationary point of the observed log-likelihood using compactness and sandwich inequality arguments.

Experimental results

Research questions

• RQ1 Can a modified EM algorithm that alternates between different complete data spaces achieve better computational efficiency in high-dimensional missing data problems?
• RQ2 Does replacing matrix inversion in the E-Step with soft-thresholding improve computational scalability and sparsity in imputation?
• RQ3 Can Lasso regularization in the regression step lead to more accurate and interpretable imputations in 'large P, small N' settings?
• RQ4 How does the performance of MissPALasso compare to k-NN, nuclear norm minimization, and MissGLasso on real microarray datasets?
• RQ5 Is the proposed algorithm numerically stable and guaranteed to converge to a stationary point of the observed log-likelihood?

Key findings

• MissPALasso significantly improves computational efficiency by replacing matrix inversion with soft-thresholding in the E-Step, avoiding costly operations on large covariance matrices.
• The algorithm achieves sparse regression coefficients through Lasso penalties, enhancing interpretability and reducing overfitting in high-dimensional settings.
• Simulations and real data analyses on four microarray datasets show that MissPALasso outperforms k-NN, nuclear norm minimization, and MissGLasso in imputation accuracy.
• The method converges numerically to a stationary point of the observed log-likelihood, as rigorously proven via compactness and sandwich inequality arguments.
• MissPALasso demonstrates robust performance even when the data do not follow a low-rank structure, outperforming convex matrix completion methods in non-low-rank scenarios.
• The proposed approach generalizes the standard EM algorithm by allowing alternating complete data spaces, enabling more flexible and efficient optimization in missing data problems.