[Paper Review] Learning Optimal Decision Trees from Large Datasets

Florent Avellaneda|arXiv (Cornell University)|Apr 12, 2019

Machine Learning and Data Classification9 references4 citations

TL;DR

This paper presents a scalable, SAT-based method for learning optimal decision trees with minimal depth and node count from large datasets. By incrementally generating Boolean formulas and refining the tree using counterexamples, the approach achieves significantly faster inference—processing the 'Mouse' dataset in 75ms—outperforming prior exact methods in speed and scalability while maintaining high accuracy.

ABSTRACT

Inferring a decision tree from a given dataset is one of the classic problems in machine learning. This problem consists of buildings, from a labelled dataset, a tree such that each node corresponds to a class and a path between the tree root and a leaf corresponds to a conjunction of features to be satisfied in this class. Following the principle of parsimony, we want to infer a minimal tree consistent with the dataset. Unfortunately, inferring an optimal decision tree is known to be NP-complete for several definitions of optimality. Hence, the majority of existing approaches relies on heuristics, and as for the few exact inference approaches, they do not work on large data sets. In this paper, we propose a novel approach for inferring a decision tree of a minimum depth based on the incremental generation of Boolean formula. The experimental results indicate that it scales sufficiently well and the time it takes to run grows slowly with the size of dataset.

Motivation & Objective

• To address the NP-complete problem of learning optimal decision trees with minimal depth and node count from large datasets.
• To overcome the scalability limitations of existing SAT-based exact inference methods that struggle with large training sets.
• To develop a practical, efficient algorithm for exact decision tree learning suitable for critical systems requiring explainable AI.
• To improve upon prior SAT formulations by enabling incremental processing of training examples to reduce runtime growth with dataset size.
• To demonstrate that optimal decision trees can be learned efficiently on well-known, large-scale datasets previously considered infeasible for exact methods.

Proposed method

• Formulates decision tree learning as a Boolean satisfiability (SAT) problem, encoding tree structure and consistency constraints with training examples.
• Introduces an incremental algorithm that processes training examples one at a time, refining the tree using counterexamples when inconsistencies arise.
• Uses a two-phase approach: first, learns a minimal-depth tree; second, prunes it to minimize nodes while preserving depth optimality.
• Employs a compact SAT encoding that avoids monolithic formula construction, reducing memory and runtime overhead.
• Leverages modern SAT solvers to efficiently solve the incremental formula generation, enabling scalability to large datasets.
• Applies feature encoding techniques to handle non-binary and numerical features by transforming them into binary Boolean features.

Experimental results

Research questions

• RQ1 Can exact decision tree learning be made scalable to large datasets while maintaining optimality in depth and node count?
• RQ2 How does incremental formula generation compare to monolithic SAT encoding in terms of runtime and memory usage?
• RQ3 What is the impact of dataset size, number of features, and depth on inference time in exact decision tree learning?
• RQ4 Can optimal decision trees be learned for well-known benchmark datasets previously considered too large for exact methods?
• RQ5 Does the incremental approach maintain or improve prediction accuracy compared to heuristic or non-incremental exact methods?

Key findings

• The proposed method processes the 'Mouse' dataset in 75 milliseconds, a 7.7x improvement over the prior best SAT-based method (577 seconds).
• The algorithm scales sub-linearly with dataset size, as inference time plateaus after a certain number of examples, unlike previous SAT-based methods.
• For depth-4 trees, the method achieves 92.6% accuracy on the balance-scale dataset, outperforming the BinOCT* heuristic method (78.9%).
• Execution time grows almost polynomially with the number of nodes, despite the NP-completeness of the problem, indicating strong practical scalability.
• The number of features has the strongest negative impact on inference time, suggesting feature selection or pruning could further improve performance.
• The method successfully infers optimal decision trees for datasets previously considered infeasible for exact inference, including large real-world benchmarks.