[Paper Review] Decision trees as partitioning machines to characterize their generalization properties

Jean-Samuel Leboeuf, Frédéric LeBlanc+1 more|arXiv (Cornell University)|Oct 14, 2020

Machine Learning and Data Classification31 references4 citations

TL;DR

This paper introduces the partitioning function as a novel framework to analyze the generalization properties of decision trees with real-valued features. It derives the exact VC dimension of decision stumps as the largest integer d satisfying 2ℓ ≥ C(d, ⌊d/2⌋), and establishes an upper bound on the VC dimension of binary trees with N internal nodes as O(N log(Nℓ)), enabling a new pruning algorithm that outperforms CART without requiring cross-validation.

ABSTRACT

Decision trees are popular machine learning models that are simple to build and easy to interpret. Even though algorithms to learn decision trees date back to almost 50 years, key properties affecting their generalization error are still weakly bounded. Hence, we revisit binary decision trees on real-valued features from the perspective of partitions of the data. We introduce the notion of partitioning function, and we relate it to the growth function and to the VC dimension. Using this new concept, we are able to find the exact VC dimension of decision stumps, which is given by the largest integer $d$ such that $2\ell \ge \binom{d}{\left\lfloor\frac{d}{2} ight floor}$, where $\ell$ is the number of real-valued features. We provide a recursive expression to bound the partitioning functions, resulting in a upper bound on the growth function of any decision tree structure. This allows us to show that the VC dimension of a binary tree structure with $N$ internal nodes is of order $N \log(N\ell)$. Finally, we elaborate a pruning algorithm based on these results that performs better than the CART algorithm on a number of datasets, with the advantage that no cross-validation is required.

Motivation & Objective

• To address the lack of tight upper bounds on the VC dimension and growth function of binary decision trees with real-valued features.
• To provide a theoretical characterization of generalization error in decision trees using a new concept: the partitioning function.
• To derive the exact VC dimension of decision stumps on real-valued features, resolving a previously unknown bound.
• To extend the partitioning function framework to general binary tree structures and derive asymptotic bounds on their VC dimension.
• To develop a pruning algorithm based on theoretical bounds that outperforms CART on multiple datasets without requiring cross-validation.

Proposed method

• Introduces the partitioning function as a measure of the number of distinct data partitions induced by a decision tree structure on real-valued features.
• Relates the partitioning function to the growth function and VC dimension, enabling theoretical analysis of generalization error.
• Uses graph-theoretic methods to derive the exact VC dimension of decision stumps as the largest d such that 2ℓ ≥ C(d, ⌊d/2⌋), where ℓ is the number of features.
• Develops a recursive upper bound on the partitioning function for general binary tree structures, leading to an O(N log(Nℓ)) upper bound on the VC dimension for trees with N internal nodes.
• Proposes a pruning algorithm based on the derived VC dimension bounds, using theoretical risk bounds instead of cross-validation.
• Employs sample-compression and VC-based generalization bounds to guide tree pruning while maintaining high test accuracy.

Experimental results

Research questions

• RQ1 What is the exact VC dimension of decision stumps constructed from ℓ real-valued features?
• RQ2 How can the partitioning function be used to bound the growth function and VC dimension of general binary decision tree structures?
• RQ3 What is the asymptotic behavior of the VC dimension for a binary decision tree with N internal nodes and ℓ real-valued features?
• RQ4 Can theoretical bounds on VC dimension be leveraged to design a pruning algorithm that outperforms CART without cross-validation?
• RQ5 How does the proposed pruning method compare in performance and efficiency to CART across diverse datasets?

Key findings

• The exact VC dimension of decision stumps on ℓ real-valued features is the largest integer d such that 2ℓ ≥ C(d, ⌊d/2⌋), resolving a previously unknown bound.
• The VC dimension of a binary decision tree with N internal nodes and ℓ real-valued features is O(N log(Nℓ)), providing a tight asymptotic upper bound.
• The proposed pruning algorithm based on theoretical VC bounds achieves higher test accuracy than CART on 10 out of 11 datasets tested, including Ionosphere, Iris, Parkinson, and Spambase.
• The new pruning method reduces the number of leaves and tree height while maintaining or improving test accuracy, with significantly faster runtime than CART on larger datasets.
• The algorithm achieves better generalization than CART on datasets like Yeast and Planning Relax, where CART overfits due to high variance in cross-validation.
• The bound computed by the proposed method (e.g., 22.3 ± 0.3 on Yeast) correlates with generalization performance and is used to guide pruning without requiring cross-validation.