[Paper Review] Deep differentiable forest with sparse attention for the tabular data

Yingshi Chen|arXiv (Cornell University)|Feb 29, 2020

Advanced Neural Network Applications19 references4 citations

TL;DR

This paper proposes a deep differentiable forest with sparse attention for tabular data, combining the interpretability of decision trees with the optimization power of neural networks. By enforcing sparsity through data-aware initialization of attention weights, the model achieves higher accuracy than XGBoost on large tabular datasets while maintaining full differentiability for gradient-based training.

ABSTRACT

We present a general architecture of deep differentiable forest and its sparse attention mechanism. The differentiable forest has the advantages of both trees and neural networks. Its structure is a simple binary tree, easy to use and understand. It has full differentiability and all variables are learnable parameters. We would train it by the gradient-based optimization method, which shows great power in the training of deep CNN. We find and analyze the attention mechanism in the differentiable forest. That is, each decision depends on only a few important features, and others are irrelevant. The attention is always sparse. Based on this observation, we improve its sparsity by data-aware initialization. We use the attribute importance to initialize the attention weight. Then the learned weight is much sparse than that from random initialization. Our experiment on some large tabular dataset shows differentiable forest has higher accuracy than GBDT, which is the state of art algorithm for tabular datasets. The source codes are available at https://github.com/closest-git/QuantumForest

Motivation & Objective

• To develop a deep, differentiable decision forest architecture that combines tree interpretability with neural network optimization.
• To identify and exploit attention mechanisms in differentiable forests, where only a few features influence each decision.
• To enhance model sparsity by initializing attention weights using feature importance, reducing reliance on random initialization.
• To outperform state-of-the-art gradient-boosted decision trees (GBDT) on large tabular datasets.
• To enable end-to-end training of tree-based models using gradient descent through differentiable components.

Proposed method

• The model uses a simple binary tree structure with differentiable decision functions parameterized by learnable weights.
• Each split in the tree is computed via a differentiable sigmoid function applied to weighted features, enabling backpropagation.
• Attention is modeled as a sparse selection of features per node, where only a few features contribute significantly to decisions.
• Sparsity is enhanced by initializing attention weights using feature importance scores from a pre-trained tree, promoting early pruning of irrelevant features.
• The entire architecture is trained via standard backpropagation using gradient-based optimization.
• The method supports end-to-end training on tabular data with full differentiability across all parameters.

Experimental results

Research questions

• RQ1 Can a differentiable decision forest architecture achieve higher accuracy than traditional gradient-boosted trees on tabular data?
• RQ2 Does attention in differentiable forests naturally exhibit sparsity, with only a few features influencing each decision?
• RQ3 Can data-aware initialization of attention weights improve model sparsity and performance compared to random initialization?
• RQ4 To what extent does differentiability enable effective end-to-end training of tree-based models?
• RQ5 How does the proposed model compare to state-of-the-art tabular learning methods like XGBoost?

Key findings

• The deep differentiable forest achieves higher accuracy than XGBoost on large tabular datasets, demonstrating superior performance.
• The attention mechanism in the differentiable forest is inherently sparse, with only a small subset of features contributing to each decision.
• Data-aware initialization of attention weights results in significantly sparser attention patterns compared to random initialization.
• The model's full differentiability enables effective training via gradient-based optimization, similar to deep neural networks.
• The method maintains interpretability through a tree structure while achieving performance on par with or better than GBDT.
• The source code is publicly available, supporting reproducibility and further research.