[Paper Review] Mixed Effects Random Forests for Personalised Predictions of Clinical Depression Severity

Robert A. Lewis, Asma Ghandeharioun+4 more|arXiv (Cornell University)|Jan 24, 2023

Mental Health Research TopicsPsychology8 citations

TL;DR

The paper demonstrates that mixed effects random forests (MERF) can improve HDRS-17 depression score predictions over standard random forests and personal baselines when trained on multimodal longitudinal data from patients with major depressive disorder, though gains are limited to scenarios with labelled training data for the individuals.

ABSTRACT

This work demonstrates how mixed effects random forests enable accurate predictions of depression severity using multimodal physiological and digital activity data collected from an 8-week study involving 31 patients with major depressive disorder. We show that mixed effects random forests outperform standard random forests and personal average baselines when predicting clinical Hamilton Depression Rating Scale scores (HDRS_17). Compared to the latter baseline, accuracy is significantly improved for each patient by an average of 0.199-0.276 in terms of mean absolute error (p<0.05). This is noteworthy as these simple baselines frequently outperform machine learning methods in mental health prediction tasks. We suggest that this improved performance results from the ability of the mixed effects random forest to personalise model parameters to individuals in the dataset. However, we find that these improvements pertain exclusively to scenarios where labelled patient data are available to the model at training time. Investigating methods that improve accuracy when generalising to new patients is left as important future work.

Motivation & Objective

• Address heterogeneity in longitudinal depression data by applying mixed effects modeling to a random forest.
• Evaluate whether MERF outperforms standard random forests and personal baselines in predicting HDRS-17 scores.
• Assess scenario-specific performance and robustness across random, time-split, and user-split evaluations.

Proposed method

• Apply mixed effects random forest (MERF) with a random intercept per patient and a fixed-effect random forest f(X) to predict HDRS-17 scores.
• Model: Y_i = f(X_i) + Z_i b_i + ε_i with b_i ~ N(0, D) and ε_i ~ N(0, σ^2 I).
• Estimate parameters using an Expectation-Maximisation (EM) procedure to maximize a generalized log likelihood.
• Use EM to alternate between updating f(X_i) and patient-specific random effects b_i, with convergence monitored by the GLL.
• Evaluate predictions on group and participant levels using mean absolute error (MAE).

Figure 1: Mean absolute error (MAE) of testing set HDRS 17 prediction by evaluation scenario and model type. The MAE is reported here at the group-level : i.e., it is the average error across all observations in the testing set. The scenarios and models are described in Sections 4.2 & 4.3 , respecti

Experimental results

Research questions

• RQ1 Does MERF improve HDRS-17 prediction accuracy over standard random forests and simple personal baselines?
• RQ2 How does MERF perform under different data-splitting schemes (random, time-based, and user holdout)?
• RQ3 Is the improvement primarily due to patient-specific random effects or shared fixed effects learned by the forest?
• RQ4 Can MERF maintain robustness across participants as measured by worst-case error?

Key findings

• MERF outperforms standard random forests and personal baselines in Random Split and Time Split scenarios.
• Compared to personal baselines, MERF yields average MAE improvements of 0.199 (Random Split) and 0.276 (Time Split) with p-values 0.000 and 0.004 respectively.
• In the User Split scenario, MERF does not provide a lift over baselines and can perform similarly to standard RF.
• Participant-level analysis shows significant user lift in the first two scenarios (permutation tests, p ≤ 0.05), indicating per-patient improvements.
• MERF shows modest improvement in worst-case error in Random Split and Time Split, indicating some robustness gains.
• The study uses 1,643 days of data from 31 patients, with 149 observations used for HDRS-17 prediction, and 2,820 engineered features from multimodal sensors.

Figure 2: Extended results set including standard machine learning models as additional baseline comparisons. Mean absolute error (MAE) of testing set HDRS 17 prediction by evaluation scenario and model type. The MAE is reported here at the group-level : i.e., it is the average error across all obse