[Paper Review] Approximate nonparametric maximum likelihood inference for mixture models via convex optimization

Long Feng, Lee H. Dicker|arXiv (Cornell University)|Jun 7, 2016

Bayesian Methods and Mixture Models28 references3 citations

TL;DR

This paper proposes a convex optimization-based approach to approximate nonparametric maximum likelihood estimation (NPML) for multivariate mixture models, enabling scalable and accurate inference without strong parametric assumptions. The method efficiently estimates high-dimensional mixing distributions and demonstrates superior performance over univariate and parametric alternatives in real-world applications including baseball analytics, microarray classification, and blood-glucose prediction.

ABSTRACT

Nonparametric maximum likelihood (NPML) for mixture models is a technique for estimating mixing distributions that has a long and rich history in statistics going back to the 1950s, and is closely related to empirical Bayes methods. Historically, NPML-based methods have been considered to be relatively impractical because of computational and theoretical obstacles. However, recent work focusing on approximate NPML methods suggests that these methods may have great promise for a variety of modern applications. Building on this recent work, a class of flexible, scalable, and easy to implement approximate NPML methods is studied for problems with multivariate mixing distributions. Concrete guidance on implementing these methods is provided, with theoretical and empirical support; topics covered include identifying the support set of the mixing distribution, and comparing algorithms (across a variety of metrics) for solving the simple convex optimization problem at the core of the approximate NPML problem. Additionally, three diverse real data applications are studied to illustrate the methods' performance: (i) A baseball data analysis (a classical example for empirical Bayes methods), (ii) high-dimensional microarray classification, and (iii) online prediction of blood-glucose density for diabetes patients. Among other things, the empirical results demonstrate the relative effectiveness of using multivariate (as opposed to univariate) mixing distributions for NPML-based approaches.

Motivation & Objective

• To address the long-standing computational and theoretical challenges of nonparametric maximum likelihood (NPML) estimation for multivariate mixing distributions.
• To develop a practical, scalable, and theoretically grounded method for fitting arbitrary multivariate mixing distributions using convex optimization.
• To provide concrete implementation guidance for approximate NPML, including support set identification and algorithm comparison.
• To demonstrate the empirical advantages of multivariate over univariate mixing distributions in real data applications.

Proposed method

• The method formulates approximate NPML estimation as a convex optimization problem, leveraging interior-point methods for efficient computation.
• It uses a convex approximation to the NPMLE that maintains statistical consistency while avoiding the computational burden of exact NPML.
• The support set of the estimated mixing distribution is identified via a theoretical result showing it lies within the convex hull of maximum likelihood estimates.
• Algorithms are compared across metrics such as convergence speed, accuracy, and scalability on high-dimensional data.
• The approach is applied to three real data problems: baseball player performance, high-dimensional microarray classification, and continuous glucose monitoring.
• The method incorporates state-space models (e.g., Kalman filters) with nonparametric mixing distributions to model time-varying parameters.

Experimental results

Research questions

• RQ1 Can convex optimization be used to make approximate NPML estimation computationally feasible and scalable for multivariate mixture models?
• RQ2 How does the performance of multivariate mixing distributions compare to univariate ones in empirical Bayes settings?
• RQ3 What is an effective and reliable way to identify the support set of the estimated mixing distribution in high-dimensional problems?
• RQ4 How do different optimization algorithms compare in terms of accuracy, speed, and robustness for solving the core convex problem?
• RQ5 Can the proposed method outperform standard parametric and empirical Bayes approaches in real-world applications like genomics and medical monitoring?

Key findings

• The proposed convex optimization approach achieves significant performance gains in blood-glucose prediction, reducing mean squared error (MSE) by 5.7% relative to the combined Kalman filter model.
• In microarray classification, the multivariate NPML approach outperformed univariate and parametric alternatives, demonstrating improved accuracy in high-dimensional settings.
• The NPMLE-based method reduced MSE by 1.51 relative to CGM in the blood-glucose dataset, outperforming both individual and combined models.
• The support set of the estimated mixing distribution was shown to be a subset of the convex hull of MLEs under elliptical unimodal likelihoods, enabling efficient computation.
• The Kalman filter with nonparametric mixing distributions reduced MSE to 1.03, significantly outperforming the linear model (MSE 1.56) and the combined model (MSE 1.54).
• Empirical results confirm that multivariate mixing distributions with correlated components yield better inference than univariate counterparts in all three real data applications.