[Paper Review] Combining Gradient Boosting Machines with Collective Inference to Predict Continuous Values

Iman Al-Odah, Jennifer Neville|arXiv (Cornell University)|Jul 1, 2016

Bayesian Modeling and Causal Inference9 references3 citations

TL;DR

This paper proposes a novel gradient boosting algorithm, MGB, that integrates collective inference into regression tasks on relational data by iteratively learning relational models to predict residuals. The method improves prediction accuracy over standard gradient boosting by leveraging relational dependencies, with empirical results showing superior performance on real-world network datasets, particularly when using current boosted predictions rather than neighbor residuals as features.

ABSTRACT

Gradient boosting of regression trees is a competitive procedure for learning predictive models of continuous data that fits the data with an additive non-parametric model. The classic version of gradient boosting assumes that the data is independent and identically distributed. However, relational data with interdependent, linked instances is now common and the dependencies in such data can be exploited to improve predictive performance. Collective inference is one approach to exploit relational correlation patterns and significantly reduce classification error. However, much of the work on collective learning and inference has focused on discrete prediction tasks rather than continuous. %target values has not got that attention in terms of collective inference. In this work, we investigate how to combine these two paradigms together to improve regression in relational domains. Specifically, we propose a boosting algorithm for learning a collective inference model that predicts a continuous target variable. In the algorithm, we learn a basic relational model, collectively infer the target values, and then iteratively learn relational models to predict the residuals. We evaluate our proposed algorithm on a real network dataset and show that it outperforms alternative boosting methods. However, our investigation also revealed that the relational features interact together to produce better predictions.

Motivation & Objective

• To address the limitation of traditional gradient boosting in relational data, where instances are interdependent rather than i.i.d.
• To extend collective inference—previously used mainly for classification—into the regression setting with continuous target variables.
• To investigate whether combining gradient boosting with collective inference enhances predictive performance in relational regression tasks.
• To evaluate the impact of different relational features, especially those based on neighbors' residuals versus current boosted predictions.
• To determine whether collective inference on residuals improves model accuracy compared to standard boosting or non-collective relational models.

Proposed method

• Proposes a modified gradient boosting (MGB) algorithm that learns a basic relational model and then iteratively fits models to predict residuals using collective inference.
• At each boosting stage, the algorithm computes residuals as the difference between true and predicted values, using the negative gradient of the loss function.
• Relational features are constructed based on neighbors' target values, current boosted predictions, and residuals, with initial values set to median known neighbor values or zero.
• Collective inference is applied to update predictions across the network simultaneously, exploiting relational correlations to improve estimation.
• The boosting process proceeds stage-wise, updating the model as $ F_m = F_{m-1} + \rho_m \cdot h_m $, where $ h_m $ is a weak learner fitted to the residuals.
• The algorithm uses line search to optimize the multiplier $ \rho_m $, ensuring minimal loss at each stage.

Experimental results

Research questions

• RQ1 Can collective inference improve gradient boosting performance in relational regression tasks?
• RQ2 How do different relational features—especially those based on neighbors' residuals versus current predictions—affect model accuracy?
• RQ3 Does integrating collective inference into the residual learning phase lead to better generalization than standard gradient boosting on relational data?
• RQ4 What is the relative contribution of various relational features to the final predictive performance?
• RQ5 Is collective inference on residuals more effective than using current boosted predictions as features in the relational model?

Key findings

• The MGB algorithm outperforms standard gradient boosting and non-collective relational models on a real-world network dataset, demonstrating improved regression accuracy through collective inference.
• The inclusion of relational feature Rf₄ (based on neighbors' current boosted predictions) significantly improves performance, as shown by ablation studies.
• Relational features based on neighbors' residuals (Rf₂ and Rf₃) were less effective and sometimes degraded performance, indicating difficulty in estimating residuals accurately via collective inference.
• The best performance was achieved only when both Rf₁ (median of known neighbors' values) and Rf₄ (current boosted predictions) were used together, highlighting feature interaction effects.
• Results suggest that using current boosted predictions as features is more effective than relying on collective inference of residuals, even though residuals are correlated with error.
• The algorithm's performance is sensitive to feature initialization, with median-based initialization for Rf₁ and Rf₄ yielding better results than random or zero initialization.