[Paper Review] LMLFM: Longitudinal Multi-Level Factorization Machine

Junjie Liang, Dongkuan Xu+2 more|arXiv (Cornell University)|Nov 11, 2019

Recommender Systems and Techniques34 references4 citations

TL;DR

LMLFM is a novel longitudinal multi-level factorization machine that jointly selects fixed and random effects in high-dimensional longitudinal data with complex correlation structures (longitudinal and cluster correlation). It uses a hierarchical Bayesian framework with two-layer Laplace priors and iterated conditional modes optimization, achieving superior predictive accuracy, sparsity, and scalability compared to state-of-the-art methods on simulated and real-world data with over 5,000 variables.

ABSTRACT

We consider the problem of learning predictive models from longitudinal data, consisting of irregularly repeated, sparse observations from a set of individuals over time. Such data often exhibit {\em longitudinal correlation} (LC) (correlations among observations for each individual over time), {\em cluster correlation} (CC) (correlations among individuals that have similar characteristics), or both. These correlations are often accounted for using {\em mixed effects models} that include {\em fixed effects} and {\em random effects}, where the fixed effects capture the regression parameters that are shared by all individuals, whereas random effects capture those parameters that vary across individuals. However, the current state-of-the-art methods are unable to select the most predictive fixed effects and random effects from a large number of variables, while accounting for complex correlation structure in the data and non-linear interactions among the variables. We propose Longitudinal Multi-Level Factorization Machine (LMLFM), to the best of our knowledge, the first model to address these challenges in learning predictive models from longitudinal data. We establish the convergence properties, and analyze the computational complexity, of LMLFM. We present results of experiments with both simulated and real-world longitudinal data which show that LMLFM outperforms the state-of-the-art methods in terms of predictive accuracy, variable selection ability, and scalability to data with large number of variables. The code and supplemental material is available at \url{https://github.com/junjieliang672/LMLFM}.

Motivation & Objective

• To address the challenge of learning predictive models from high-dimensional longitudinal data where fixed and random effects are a priori unspecified.
• To jointly select fixed and random effects while accounting for complex correlation structures, including longitudinal correlation (LC) and cluster correlation (CC).
• To reduce reliance on hyper-parameter tuning and improve scalability in data with large numbers of variables.
• To enable effective modeling of nonlinear interactions and sparse, interpretable representations in longitudinal settings.
• To provide a provably convergent, computationally efficient alternative to existing mixed-effects models that scale poorly with variable count.

Proposed method

• LMLFM extends Factorization Machines (FM) to longitudinal data using a hierarchical Bayesian formulation with two layers of Laplace priors.
• The first Laplace prior induces sparsity in the latent factor representations, enabling efficient learning from high-dimensional data.
• The second Laplace prior differentiates between fixed effects (shared across individuals) and random effects (varying by individual or cluster), enabling automatic effect selection.
• The model is trained using the iterated conditional modes (ICM) algorithm, which ensures strong convergence guarantees and computational efficiency.
• LMLFM jointly models nonlinear interactions and correlation structures (LC and CC) by integrating individual-level and group-level effects within a unified probabilistic framework.
• The framework allows for automatic variable selection without extensive hyper-parameter tuning, enhancing model interpretability and scalability.

Experimental results

Research questions

• RQ1 Can a unified model jointly select fixed and random effects in high-dimensional longitudinal data while accounting for complex correlation structures?
• RQ2 How does LMLFM perform in predictive accuracy and variable selection compared to state-of-the-art baselines on simulated and real-world longitudinal data?
• RQ3 To what extent does LMLFM scale to datasets with over 5,000 variables, where traditional mixed-effects models fail?
• RQ4 Does LMLFM effectively identify the presence and dominance of longitudinal correlation (LC) and cluster correlation (CC) in real data?
• RQ5 Can LMLFM produce sparse, interpretable models that align with domain-specific findings in health and social science data?

Key findings

• LMLFM successfully models longitudinal data with over 5,000 variables, whereas traditional mixed-effects models fail when the number of variables exceeds 100.
• On the GSS dataset with 1,553 variables, LMLFM achieved the highest R² score among all baselines, significantly outperforming FM, MLLASSO, and RF in predictive accuracy.
• LMLFM identified 126 non-zero fixed effects out of 1,199, with only 6 features having absolute effects > 0.1, indicating high sparsity and effective variable selection.
• The sparsity rate of the random effect matrix Θ^O was 84.9% on the GSS data, suggesting strong cluster correlation (CC) dominates over longitudinal correlation (LC).
• LMLFM’s selected variables were consistent with published findings on happiness predictors, including marriage, health, financial satisfaction, and religiosity.
• LMLFM ruled out longitudinal correlation (LC) in the GSS data, likely due to long gaps between survey observations, and confirmed the dominance of cluster correlation (CC).