[Paper Review] Multi-Layered Gradient Boosting Decision Trees

Ji Feng, Yu Yang+1 more|arXiv (Cornell University)|May 31, 2018

Neural Networks and Applications63 citations

TL;DR

The paper introduces multi-layered GBDT forests (mGBDTs) that learn hierarchical representations by stacking regression GBDTs and training via a target-propagation-like procedure without backpropagation. It demonstrates representation learning and competitive performance on tabular data.

ABSTRACT

Multi-layered representation is believed to be the key ingredient of deep neural networks especially in cognitive tasks like computer vision. While non-differentiable models such as gradient boosting decision trees (GBDTs) are the dominant methods for modeling discrete or tabular data, they are hard to incorporate with such representation learning ability. In this work, we propose the multi-layered GBDT forest (mGBDTs), with an explicit emphasis on exploring the ability to learn hierarchical representations by stacking several layers of regression GBDTs as its building block. The model can be jointly trained by a variant of target propagation across layers, without the need to derive back-propagation nor differentiability. Experiments and visualizations confirmed the effectiveness of the model in terms of performance and representation learning ability.

Motivation & Objective

• Motivate the need for deep representations in non-differentiable models like GBDTs for tabular data.
• Propose a multi-layered GBDT architecture to learn hierarchical representations.
• Develop a training procedure that without back-propagation jointly optimizes all layers via pseudo-labels.
• Demonstrate representation learning and competitive performance on real-world datasets.
• Explore potential extensions and applications of non-differentiable deep models.

Proposed method

• Construct a multi-layer feedforward structure with M-1 intermediate layers and a final output layer.
• Define forward mappings F_i as non-differentiable GBDT-based layers and introduce inverse mappings G_i to form pseudo-inverse pairs.
• Use a variant of target propagation: compute pseudo-labels for each layer by propagating top-layer targets downward through inverse mappings.
• Update each F_i by gradient-boosting steps toward pseudo-residuals derived from the pseudo-labels; train G_i to map forward outputs back toward previous layer representations.
• Inject Gaussian noise in the inverse path to regularize and stabilize learning; initialize with tiny trees and small Gaussian outputs to start.
• Provide a bottom-up update schedule where F_i are updated before higher layers, and iterate for E epochs.

Experimental results

Research questions

• RQ1 Can a multi-layered model built from non-differentiable components (GBDTs) be trained end-to-end without backpropagation?
• RQ2 Do stacked GBDT layers learn hierarchical, distributed representations similar to deep neural networks?
• RQ3 How does mGBDT perform in supervised and unsupervised settings compared to neural networks and single GBDTs?
• RQ4 Does increasing depth improve representation quality and classification/regression performance on tabular data?

Key findings

• mGBDT achieves the highest accuracy among compared methods on the Income and Protein datasets in the reported experiments.
• In the Income dataset, mGBDT achieves 0.8742 accuracy, compared to XGBoost 0.8719, XGBoost Stacking 0.8697, NN(TargetProp) 0.8491, NN(BackProp) 0.8534, and similar trends on the Protein dataset.
• Unsupervised mGBDT autoencoders can produce distributed representations, as visualization shows meaningful encodings in higher layers.
• Deeper mGBDT stacks yield progressively better representations in visualization and improve performance on real data tasks.
• Compared with neural networks trained by backpropagation or target propagation, mGBDT can converge faster and reach competitive or superior accuracy on the tested datasets.
• Variations in layer depth indicate robustness of the mGBDT approach, with target-propagation-based neural nets showing less robustness as depth increases.