[Paper Review] Neural Mixture Distributional Regression

David Rügamer, Florian Pfisterer+1 more|arXiv (Cornell University)|Oct 14, 2020

Bayesian Methods and Mixture Models32 references4 citations

TL;DR

This paper introduces Neural Mixture Distributional Regression (NMDR), a deep learning-based framework for estimating complex finite mixtures of parametric distributional regression models using flexible additive predictors. By leveraging optimization techniques from deep learning, NMDR enables scalable, assumption-free estimation of high-dimensional mixtures with diverse distributions, achieving state-of-the-art performance in both synthetic and real-world applications, including a movie rating prediction task with 18 beta components and text embeddings.

ABSTRACT

We present neural mixture distributional regression (NMDR), a holistic framework to estimate complex finite mixtures of distributional regressions defined by flexible additive predictors. Our framework is able to handle a large number of mixtures of potentially different distributions in high-dimensional settings, allows for efficient and scalable optimization and can be applied to recent concepts that combine structured regression models with deep neural networks. While many existing approaches for mixture models address challenges in optimization of such and provide results for convergence under specific model assumptions, our approach is assumption-free and instead makes use of optimizers well-established in deep learning. Through extensive numerical experiments and a high-dimensional deep learning application we provide evidence that the proposed approach is competitive to existing approaches and works well in more complex scenarios.

Motivation & Objective

• To address the limitations of traditional mixture model estimation, particularly in high-dimensional settings with complex, non-convex likelihoods.
• To develop a scalable, optimization-friendly framework for estimating finite mixtures of parametric distributional regressions without restrictive model assumptions.
• To integrate structured additive predictors with deep neural networks for enhanced flexibility and interpretability in mixture regression modeling.
• To demonstrate the efficacy of the proposed framework on complex, high-dimensional real-world data, such as movie ratings with textual and covariate features.

Proposed method

• NMDR models a mixture of parametric distributions (e.g., beta, Gaussian) where each component's parameters are predicted via flexible additive predictors.
• The framework uses deep neural networks to model the linear predictors of distribution parameters (e.g., shape parameters c₀, c₁) and mixture weights π, enabling high-dimensional, non-linear relationships.
• Mixture component parameters and weights are estimated jointly via end-to-end backpropagation using optimizers from deep learning (e.g., Adam, AdaDelta), avoiding reliance on EM or MCMC.
• Structured additive effects (e.g., smooth terms s_j) are separated from deep neural network components to ensure identifiability and interpretability of smooth effects.
• Textual features are embedded via a 300-dimensional embedding layer, followed by dense layers to predict distribution parameters or mixture weights, enabling joint modeling of text and tabular covariates.
• The model is trained using mini-batch optimization with a fixed batch size of 256 and evaluated via repeated train-test splits with RMSE as the primary metric.

Experimental results

Research questions

• RQ1 Can a deep learning-based optimization framework effectively estimate complex finite mixtures of distributional regressions without strong parametric assumptions?
• RQ2 How does the integration of deep neural networks with structured additive predictors improve modeling flexibility and predictive performance in high-dimensional settings?
• RQ3 To what extent do learned text embeddings from movie descriptions improve the estimation of mixture component probabilities and distribution parameters?
• RQ4 How do different architectural choices (e.g., number of units in dense layers, routing of embeddings) affect model performance and stability in mixture modeling?

Key findings

• Model (V), which uses a single-unit dense layer to predict observation-specific mixture weights π, achieved the lowest mean RMSE of 0.117 (std. 0.026) across 10 train-test splits, outperforming all other configurations.
• Models incorporating deep neural networks for distribution parameters or mixture weights showed greater variability in estimated mixture probabilities, with some models yielding unimodal π distributions and others showing multimodal support.
• The addition of deep neural networks led to a preference for constant or linear effects over smooth effects in structured additive components, suggesting a trade-off between flexibility and identifiability.
• The t-SNE visualization of the learned embedding space in model (V) revealed meaningful clustering of the 50 most common words, indicating effective semantic representation learning.
• Model (I), the baseline without neural networks, achieved a mean RMSE of 0.242, demonstrating that deep learning integration significantly improves predictive accuracy.
• Model (IV), which used a 36-unit dense layer for both shape parameters, performed poorly with a mean RMSE of 0.321, indicating suboptimal architectural design for this task.