[Paper Review] Trees-Based Models for Correlated Data

Assaf Rabinowicz, Saharon Rosset|arXiv (Cornell University)|Feb 16, 2021

Gaussian Processes and Bayesian Inference33 references4 citations

TL;DR

This paper introduces RETCO (Regression Tree for Correlated Data), a novel trees-based model that explicitly incorporates correlation structures—such as clustered, longitudinal, or spatial data—into regression trees, random forests, and gradient boosting. By modifying splitting criteria, stopping rules, and leaf predictions to account for correlation via linear mixed models and Gaussian processes, RETCO significantly improves prediction accuracy over standard methods, as validated through simulations and real-world data across diverse domains.

ABSTRACT

This paper presents a new approach for trees-based regression, such as simple regression tree, random forest and gradient boosting, in settings involving correlated data. We show the problems that arise when implementing standard trees-based regression models, which ignore the correlation structure. Our new approach explicitly takes the correlation structure into account in the splitting criterion, stopping rules and fitted values in the leaves, which induces some major modifications of standard methodology. The superiority of our new approach over trees-based models that do not account for the correlation is supported by simulation experiments and real data analyses.

Motivation & Objective

• To address the limitation of standard trees-based models in handling correlated data, which often leads to biased predictions and inaccurate error estimation.
• To develop a unified framework that integrates random effects and random fields into tree-based models to improve predictive performance while enabling statistical inference.
• To adapt model evaluation and selection techniques for correlated data, ensuring unbiased generalization error estimation in the presence of dependence structures.
• To demonstrate the superiority of the proposed method over standard trees-based models and existing alternatives in both simulated and real-world settings.

Proposed method

• Proposes RETCO, a new algorithm that modifies the splitting criterion, stopping rules, and fitted values in tree leaves to explicitly account for known correlation structures in the data.
• Integrates linear mixed models (LMM) and Gaussian process regression into tree-based frameworks by modeling the conditional mean using fixed and random effects.
• Uses a corrected cross-validation loss function (CV_c) to estimate generalization error unbiasedly in correlated data, replacing standard CV in model selection.
• Applies kernel-based covariance functions (e.g., exponential) to model dependence between observations, particularly in spatial or temporal settings.
• Employs a recursive partitioning strategy where each split minimizes a loss function adjusted for correlation, ensuring that predictions reflect the underlying dependence structure.
• Supports ensemble methods like random forest and gradient boosting by applying the correlation-aware splitting and prediction rules across multiple trees.

Experimental results

Research questions

• RQ1 How does ignoring correlation in standard trees-based models affect prediction accuracy and error estimation in correlated data?
• RQ2 Can a modified tree-based model that explicitly incorporates correlation structures outperform standard regression trees, random forests, and gradient boosting in terms of prediction error?
• RQ3 How can model selection and error estimation be corrected for correlated data to avoid optimistic bias in cross-validation?
• RQ4 What is the performance gain of RETCO compared to existing methods such as RE-EM or standard RF when applied to real-world correlated datasets?

Key findings

• RETCO significantly reduces prediction error compared to standard trees-based models in all simulated and real-world scenarios involving correlated data.
• In simulations with varying levels of random effects variance (σ²_b), RETCO consistently outperformed standard models, with error reductions up to 30% in high-correlation settings.
• The corrected cross-validation loss (CV_c) provided unbiased generalization error estimates, while standard CV underestimated error in correlated data, leading to overfitting.
• In real data analyses, RETCO achieved lower prediction error than standard RF on five diverse datasets: FIFA player values, US crime rates, South Korean temperature forecasts, California housing prices, Parkinson’s telemonitoring, and wage data.
• RETCO outperformed the RE-EM method in both scenarios (b*⊥b and b*=b), with larger performance gains observed when the correlation structure was strong.
• The method maintained stable performance across different data types—clustered (FIFA, crimes), spatial (Korea temperature), longitudinal (Parkinson’s, wages)—demonstrating broad applicability.