[Paper Review] Machine Learning with High-Cardinality Categorical Features in Actuarial Applications

Benjamin Avanzi, G. B. Taylor+2 more|arXiv (Cornell University)|Jan 30, 2023

Insurance, Mortality, Demography, Risk Management4 citations

TL;DR

This paper proposes GLMMNet, a novel neural network architecture that integrates generalised linear mixed models (GLMMs) with deep learning to model high-cardinality categorical features in actuarial data. By using variational inference to handle complex response distributions and retaining interpretable random effects, GLMMNet achieves predictive performance comparable to or better than entity-embedded neural networks, while offering transparency and probabilistic uncertainty estimates.

ABSTRACT

High-cardinality categorical features are pervasive in actuarial data (e.g. occupation in commercial property insurance). Standard categorical encoding methods like one-hot encoding are inadequate in these settings. In this work, we present a novel _Generalised Linear Mixed Model Neural Network_ ("GLMMNet") approach to the modelling of high-cardinality categorical features. The GLMMNet integrates a generalised linear mixed model in a deep learning framework, offering the predictive power of neural networks and the transparency of random effects estimates, the latter of which cannot be obtained from the entity embedding models. Further, its flexibility to deal with any distribution in the exponential dispersion (ED) family makes it widely applicable to many actuarial contexts and beyond. We illustrate and compare the GLMMNet against existing approaches in a range of simulation experiments as well as in a real-life insurance case study. Notably, we find that the GLMMNet often outperforms or at least performs comparably with an entity embedded neural network, while providing the additional benefit of transparency, which is particularly valuable in practical applications. Importantly, while our model was motivated by actuarial applications, it can have wider applicability. The GLMMNet would suit any applications that involve high-cardinality categorical variables and where the response cannot be sufficiently modelled by a Gaussian distribution.

Motivation & Objective

• Address the limitations of one-hot encoding and entity embeddings in modeling high-cardinality categorical features common in actuarial data.
• Overcome the lack of interpretability in black-box neural network embeddings while preserving predictive power.
• Extend the LMMNN framework to support the full exponential dispersion family of distributions, enabling better modeling of skewed insurance data.
• Develop a computationally efficient method for handling large-scale, high-cardinality categorical variables with uneven category distributions.
• Provide a transparent, interpretable, and flexible alternative to existing deep learning models in actuarial applications.

Proposed method

• Integrate a generalised linear mixed model (GLMM) into a deep neural network framework, treating high-cardinality categorical features as random effects.
• Replace the non-linear transformation on categorical features in the original LMMNN to preserve interpretability of random effect estimates.
• Use variational inference to approximate the intractable likelihood integral in GLMMs, enabling efficient training with non-Gaussian response distributions.
• Allow the response variable to follow any distribution in the exponential dispersion family (e.g., Poisson, Gamma, Bernoulli), enhancing applicability to actuarial data.
• Train the model end-to-end using backpropagation, with stochastic gradient descent and regularisation to prevent overfitting.
• Enable probabilistic predictions through posterior predictive distributions, including uncertainty estimates for random effects.

Experimental results

Research questions

• RQ1 Can a hybrid neural network–GLMM model outperform standard entity-embedded neural networks in predictive accuracy for high-cardinality categorical features?
• RQ2 To what extent does the GLMMNet model maintain interpretability through transparent random effect estimates compared to black-box embeddings?
• RQ3 How well does the GLMMNet perform across different response distributions (e.g., Poisson, Gamma, Bernoulli) common in actuarial data?
• RQ4 Can variational inference effectively approximate the intractable likelihood in GLMMs while maintaining computational efficiency and model accuracy?
• RQ5 Does the GLMMNet model remain robust under low signal-to-noise ratios, especially when regularisation is applied?

Key findings

• GLMMNet consistently outperforms or matches the performance of entity-embedded neural networks across multiple simulation experiments and a real-world insurance case study.
• The model achieves superior predictive performance in high-cardinality settings with skewed, overdispersed response variables, such as claim counts and severities.
• Random effect estimates from GLMMNet are interpretable and allow for statistical inference, such as identifying occupations with significantly higher or lower risk, which is not possible with standard entity embeddings.
• The use of variational inference enables efficient training with complex distributions (e.g., Gamma, Poisson) without analytical likelihood solutions, broadening the model’s applicability.
• Regularisation in GLMMNet restores model performance in low signal-to-noise ratio environments, where boosting models may otherwise dominate.
• The model produces valid probabilistic predictions, including 95% credible intervals for random effects, enhancing its utility in risk assessment and decision-making.