[Paper Review] Imputation and low-rank estimation with Missing Not At Random data

Aude Sportisse, Claire Boyer+1 more|arXiv (Cornell University)|Dec 29, 2018

Tensor decomposition and applications41 references4 citations

TL;DR

This paper proposes two novel matrix completion methods for Missing Not At Random (MNAR) data, combining model-based estimation via EM algorithm with FISTA and a computationally efficient surrogate approach that jointly models data and missingness patterns using low-rank structure on a concatenated data-mask matrix. The method achieves robust imputation and estimation under MNAR mechanisms, validated on a real trauma registry with clinical predictors for tranexamic acid use.

ABSTRACT

Missing values challenge data analysis because many supervised and unsupervised learning methods cannot be applied directly to incomplete data. Matrix completion based on low-rank assumptions are very powerful solution for dealing with missing values. However, existing methods do not consider the case of informative missing values which are widely encountered in practice. This paper proposes matrix completion methods to recover Missing Not At Random (MNAR) data. Our first contribution is to suggest a model-based estimation strategy by modelling the missing mechanism distribution. An EM algorithm is then implemented, involving a Fast Iterative Soft-Thresholding Algorithm (FISTA). Our second contribution is to suggest a computationally efficient surrogate estimation by implicitly taking into account the joint distribution of the data and the missing mechanism: the data matrix is concatenated with the mask coding for the missing values; a low-rank structure for exponential family is assumed on this new matrix, in order to encode links between variables and missing mechanisms. The methodology that has the great advantage of handling different missing value mechanisms is robust to model specification errors.The performances of our methods are assessed on the real data collected from a trauma registry (TraumaBase ) containing clinical information about over twenty thousand severely traumatized patients in France. The aim is then to predict if the doctors should administrate tranexomic acid to patients with traumatic brain injury, that would limit excessive bleeding.

Motivation & Objective

• To address the challenge of informative missing data in matrix completion, where missingness depends on unobserved values.
• To develop a model-based estimation strategy that explicitly accounts for the missing data mechanism in low-rank matrix recovery.
• To propose a computationally efficient surrogate method that implicitly models the joint distribution of data and missingness through a low-rank structure on a data-mask augmented matrix.
• To ensure robustness to model misspecification in the presence of MNAR mechanisms.
• To evaluate performance on real-world clinical data for predicting tranexamic acid administration in traumatic brain injury patients.

Proposed method

• Uses an EM algorithm with a model-based estimation strategy, where the missing mechanism is explicitly modeled via a logistic regression on the data and mask.
• Implements FISTA (Fast Iterative Soft-Thresholding Algorithm) for nuclear norm-regularized low-rank matrix estimation in the M-step.
• Introduces a surrogate method that concatenates the data matrix with a binary mask of missing values and assumes a low-rank structure on the combined matrix for exponential family distributions.
• Applies importance sampling via SIR (Sequential Importance Resampling) to approximate the posterior distribution of missing values given current parameter estimates.
• Uses generalized linear models with binomial link to estimate the missingness mechanism parameters in the M-step, updating the missingness model based on imputed data.
• Employs a soft-impute-like update in the M-step to solve the low-rank matrix estimation problem with nuclear norm penalty.

Experimental results

Research questions

• RQ1 Can a model-based EM approach with FISTA effectively recover low-rank matrices when data are Missing Not At Random (MNAR)?
• RQ2 Does jointly modeling data and missingness through a low-rank structure on a data-mask concatenated matrix improve imputation accuracy and robustness?
• RQ3 How does the proposed method perform under model misspecification compared to standard matrix completion techniques?
• RQ4 Can the method reliably predict clinical outcomes (e.g., tranexamic acid use) in real-world trauma data with informative missingness?
• RQ5 What is the impact of different missingness mechanisms on the performance of low-rank matrix completion in MNAR settings?

Key findings

• The proposed EM-FISTA method with explicit modeling of the missing mechanism significantly improves imputation accuracy on MNAR data compared to standard low-rank methods.
• The surrogate method, which treats the data-mask matrix as a single low-rank object, achieves competitive performance with reduced computational cost and greater robustness to model misspecification.
• The method successfully recovers low-rank structure in the presence of informative missingness, as demonstrated on the TraumaBase® dataset with over 20,000 patients.
• The SIR-based imputation step effectively approximates the posterior distribution of missing values, enabling stable parameter updates in the EM algorithm.
• The use of generalized linear models with binomial link for missingness modeling allows flexible and interpretable estimation of the MNAR mechanism.
• Empirical results show that the method improves prediction performance for tranexamic acid administration, a clinically relevant outcome, in the presence of MNAR data.