[Paper Review] Deep Neural Decision Trees

Yongxin Yang, Irene Garcia Morillo+1 more|arXiv (Cornell University)|Jun 19, 2018

Explainable Artificial Intelligence (XAI)21 references104 citations

TL;DR

DNDT presents a neural-network implementation of decision trees, enabling gradient-based training and end-to-end learning while preserving interpretability of tree structures.

ABSTRACT

Deep neural networks have been proven powerful at processing perceptual data, such as images and audio. However for tabular data, tree-based models are more popular. A nice property of tree-based models is their natural interpretability. In this work, we present Deep Neural Decision Trees (DNDT) -- tree models realised by neural networks. A DNDT is intrinsically interpretable, as it is a tree. Yet as it is also a neural network (NN), it can be easily implemented in NN toolkits, and trained with gradient descent rather than greedy splitting. We evaluate DNDT on several tabular datasets, verify its efficacy, and investigate similarities and differences between DNDT and vanilla decision trees. Interestingly, DNDT self-prunes at both split and feature-level.

Motivation & Objective

• Motivate interpretability in predictive models, especially for ethics- and safety-critical domains.
• Introduce a model that combines neural networks with decision trees to achieve interpretability plus gradient-based training.
• Enable easy implementation and integration of tree-like models in standard NN toolkits.
• Show that DNDT can train with back-propagation and support mini-batch and GPU acceleration.

Proposed method

• Use a soft binning function to create differentiable bin boundaries via trainable cut-points for each feature.
• Construct a tree via Kronecker product of feature binning outputs to identify final leaf nodes.
• Attach a linear classifier to each leaf to produce final predictions.
• Train all parameters end-to-end with stochastic gradient descent in a single pass.
• Optionally apply temperature-based smoothing or Gumbel-Softmax for hard leaf assignments during training.
• Discuss scalability limitations with Kronecker product and propose subspace ensembles as a practical workaround.

Experimental results

Research questions

• RQ1 Can a neural network-inspired architecture realize a decision tree that remains intrinsically interpretable?
• RQ2 Does end-to-end SGD training of both split decisions and leaf classifiers yield competitive performance on tabular data?
• RQ3 How does DNDT compare to traditional decision trees and vanilla neural networks in accuracy and interpretability?
• RQ4 What self-regularization effects (e.g., active cut-points and features) emerge during DNDT learning?
• RQ5 Can GPU acceleration and feature subspace ensembles mitigate scalability challenges?

Key findings

• DNDT is interpretable as a tree yet implemented as a neural network enabling gradient-based optimization.
• DNDT can be trained end-to-end with SGD and supports mini-batch and GPU acceleration.
• DNDT tends to self-prune by reducing the number of active cut-points and unused features during training.
• On a suite of 14 tabular datasets, DNDT often matches or slightly surpasses vanilla neural networks and can approach or exceed traditional DT performance depending on data.
• Using multiple DNDT trees trained on random feature subspaces improves scalability for high-dimensional datasets.
• DNDT enables analysis of feature importance via the usage and exclusion of features across runs, providing a way to gauge which features drive decisions.