[Paper Review] Estimation and Feature Selection in Mixtures of Generalized Linear Experts Models

Bao Huynh, Faïcel Chamroukhi|arXiv (Cornell University)|Jul 14, 2019

Bayesian Methods and Mixture Models34 references4 citations

TL;DR

This paper proposes a regularized maximum likelihood estimation framework for mixtures of generalized linear experts (MoE) models, integrating feature selection via $λ_1$-penalized likelihood and a proximal-Newton EM algorithm for efficient parameter estimation in high-dimensional settings. The method achieves sparse, stable solutions and outperforms state-of-the-art competitors in recovering true feature patterns and clustering heterogeneous regression data.

ABSTRACT

Mixtures-of-Experts (MoE) are conditional mixture models that have shown their performance in modeling heterogeneity in data in many statistical learning approaches for prediction, including regression and classification, as well as for clustering. Their estimation in high-dimensional problems is still however challenging. We consider the problem of parameter estimation and feature selection in MoE models with different generalized linear experts models, and propose a regularized maximum likelihood estimation that efficiently encourages sparse solutions for heterogeneous data with high-dimensional predictors. The developed proximal-Newton EM algorithm includes proximal Newton-type procedures to update the model parameter by monotonically maximizing the objective function and allows to perform efficient estimation and feature selection. An experimental study shows the good performance of the algorithms in terms of recovering the actual sparse solutions, parameter estimation, and clustering of heterogeneous regression data, compared to the main state-of-the art competitors.

Motivation & Objective

• To address the challenge of parameter estimation and feature selection in high-dimensional mixtures of generalized linear experts (MoE) models.
• To overcome the instability and lack of sparsity in standard maximum likelihood estimation (MLE) when predictors are numerous and correlated.
• To develop a computationally efficient algorithm that monotonically maximizes the regularized objective function while promoting sparse solutions.
• To extend regularization techniques (e.g., Lasso, SCAD) to MoE models with generalized linear experts for improved model interpretability and performance.
• To provide theoretical and empirical validation of the algorithm’s ability to recover true sparse structures in heterogeneous data.

Proposed method

• Uses a regularized maximum-likelihood objective with $λ_1$-norm penalties to induce sparsity in the expert model coefficients.
• Develops a proximal-Newton-type EM algorithm that iteratively updates parameters by solving a sequence of quadratic approximations to the log-likelihood.
• Applies Taylor expansion to construct a local quadratic approximation of the log-likelihood function around current parameter estimates.
• For Poisson and multinomial outputs, derives explicit expressions for the first and second derivatives of the approximated objective function.
• Implements a coordinate-wise update strategy for the proximal-Newton steps, ensuring monotonic increase in the objective function at each iteration.
• Uses soft-thresholding in the optimization steps to enforce sparsity, effectively performing feature selection.

Experimental results

Research questions

• RQ1 Can regularized maximum likelihood estimation improve parameter estimation and feature selection in high-dimensional MoE models?
• RQ2 Does the proposed proximal-Newton EM algorithm converge monotonically and efficiently in the presence of high-dimensional predictors?
• RQ3 How does the performance of the algorithm compare to state-of-the-art methods in recovering true sparse structures in heterogeneous regression data?
• RQ4 Can the method effectively handle different exponential family distributions (Poisson, multinomial) in the expert components?
• RQ5 What is the impact of nonconvex penalties (e.g., SCAD) versus convex penalties (e.g., Lasso) on model selection and estimation accuracy?

Key findings

• The proposed proximal-Newton EM algorithm monotonically increases the regularized objective function, ensuring stable convergence.
• The method successfully recovers the true underlying sparse feature patterns in simulated high-dimensional regression data, outperforming competitors in feature selection accuracy.
• For Poisson and multinomial outputs, the algorithm achieves accurate parameter estimation with reduced bias compared to standard MLE in high-dimensional settings.
• The algorithm demonstrates robust performance in clustering heterogeneous regression data, correctly identifying underlying subpopulations.
• The use of quadratic approximation and proximal updates enables efficient computation even when the number of predictors is large.
• Empirical results confirm that the method maintains good estimation and selection performance across various data configurations and noise levels.