[Paper Review] Neural Decision Trees

Randall Balestriero|arXiv (Cornell University)|Feb 23, 2017

Neural Networks and ApplicationsComputer Science27 references19 citations

TL;DR

This paper introduces Neural Decision Trees (NDT), a differentiable architecture that unifies decision trees and neural networks by replacing traditional splitting nodes with multilayer perceptrons (MLPs), enabling global optimization and end-to-end training. The framework achieves superior modeling capacity by learning arbitrary nonlinear decision boundaries and reduces the number of parameters needed for classification from C (softmax) to log₂(C) via a Hashing Neural Network (HNN) formulation.

ABSTRACT

In this paper we propose a synergistic melting of neural networks and decision trees (DT) we call neural decision trees (NDT). NDT is an architecture a la decision tree where each splitting node is an independent multilayer perceptron allowing oblique decision functions or arbritrary nonlinear decision function if more than one layer is used. This way, each MLP can be seen as a node of the tree. We then show that with the weight sharing asumption among those units, we end up with a Hashing Neural Network (HNN) which is a multilayer perceptron with sigmoid activation function for the last layer as opposed to the standard softmax. The output units then jointly represent the probability to be in a particular region. The proposed framework allows for global optimization as opposed to greedy in DT and differentiability w.r.t. all parameters and the input, allowing easy integration in any learnable pipeline, for example after CNNs for computer vision tasks. We also demonstrate the modeling power of HNN allowing to learn union of disjoint regions for final clustering or classification making it more general and powerful than standard softmax MLP requiring linear separability thus reducing the need on the inner layer to perform complex data transformations. We finally show experiments for supervised, semi-suppervised and unsupervised tasks and compare results with standard DTs and MLPs.

Motivation & Objective

• To overcome the limitations of greedy, non-differentiable decision trees by integrating them with deep learning through a differentiable, end-to-end trainable architecture.
• To enable global optimization of tree structures by formulating the decision process as a differentiable hashing mechanism using multilayer perceptrons.
• To reduce the number of required parameters for classification from C (standard softmax) to log₂(C) by modeling class regions via a Hashing Neural Network (HNN).
• To improve generalization and modeling capacity by allowing arbitrary nonlinear decision functions through multi-layered splitting nodes.
• To enable seamless integration into deep learning pipelines, such as those involving CNNs, for tasks like semi-supervised clustering.

Proposed method

• Replace each decision tree splitting node with an independent multilayer perceptron (MLP), allowing oblique and nonlinear decision functions instead of axis-aligned or linear splits.
• Enforce weight sharing across all MLP nodes to form a Hashing Neural Network (HNN), where the final layer uses sigmoid activation instead of softmax.
• Define the output of the HNN as the probability of input belonging to a specific region, with the final layer jointly representing the probability distribution over regions.
• Train the entire architecture end-to-end using a global loss function that minimizes the entropy of the leaf node predictions, enabling joint optimization of all parameters.
• Leverage differentiability w.r.t. both parameters and input to allow integration into any differentiable machine learning pipeline, such as those with convolutional features.
• Use the HNN’s ability to learn unions of disjoint regions to reduce the need for complex feature transformations in inner layers, improving generalization.

Experimental results

Research questions

• RQ1 Can a differentiable, globally optimized decision tree architecture be constructed by replacing splitting nodes with multilayer perceptrons?
• RQ2 How does the proposed Neural Decision Tree (NDT) compare to standard decision trees and MLPs in terms of parameter efficiency and modeling capacity?
• RQ3 To what extent does the HNN formulation reduce the number of required neurons for classification compared to standard softmax-based MLPs?
• RQ4 Can the differentiable nature of NDT enable effective integration into deep learning pipelines, such as those using CNN features?
• RQ5 Does the ability to learn arbitrary nonlinear decision boundaries improve performance in unsupervised and semi-supervised learning tasks?

Key findings

• The proposed Neural Decision Tree (NDT) achieves global optimization of tree structure through end-to-end differentiable training, avoiding the local optima issues common in greedy decision tree learning.
• The HNN formulation reduces the minimum number of neurons required for C-class classification from C (softmax) to log₂(C), significantly improving parameter efficiency.
• The framework demonstrates strong performance in supervised, semi-supervised, and unsupervised tasks, with experiments showing improved clustering and classification results compared to standard decision trees and MLPs.
• The differentiability of the model enables seamless integration into deep learning pipelines, such as those combining CNN features with NDT for robust semi-supervised clustering.
• The HNN can learn arbitrary unions of disjoint regions in input space, reducing the need for complex data transformations in hidden layers and improving generalization.
• Experiments show that the HNN with 4 neurons can effectively model complex decision boundaries, with training dynamics (loss and regularization) stabilizing over time as shown in Figures 16–18.