[Paper Review] Confidence Decision Trees via Online and Active Learning for Streaming (BIG) Data

Rocco De Rosa|arXiv (Cornell University)|Apr 12, 2016

Data Stream Mining Techniques26 references3 citations

TL;DR

This paper proposes Confidence Decision Trees (CDT) that use refined confidence intervals for split selection in streaming data, improving accuracy over Hoeffding trees by better estimating gain using entropy, Gini index, and Kearns-Mansour criteria. It also introduces an active learning module that reduces labeling costs while maintaining high performance, with theoretical guarantees on suboptimal split probability and empirical validation showing superior accuracy and robustness.

ABSTRACT

Decision tree classifiers are a widely used tool in data stream mining. The use of confidence intervals to estimate the gain associated with each split leads to very effective methods, like the popular Hoeffding tree algorithm. From a statistical viewpoint, the analysis of decision tree classifiers in a streaming setting requires knowing when enough new information has been collected to justify splitting a leaf. Although some of the issues in the statistical analysis of Hoeffding trees have been already clarified, a general and rigorous study of confidence intervals for splitting criteria is missing. We fill this gap by deriving accurate confidence intervals to estimate the splitting gain in decision tree learning with respect to three criteria: entropy, Gini index, and a third index proposed by Kearns and Mansour. Our confidence intervals depend in a more detailed way on the tree parameters. We also extend our confidence analysis to a selective sampling setting, in which the decision tree learner adaptively decides which labels to query in the stream. We furnish theoretical guarantee bounding the probability that the classification is non-optimal learning the decision tree via our selective sampling strategy. Experiments on real and synthetic data in a streaming setting show that our trees are indeed more accurate than trees with the same number of leaves generated by other techniques and our active learning module permits to save labeling cost. In addition, comparing our labeling strategy with recent methods, we show that our approach is more robust and consistent respect all the other techniques applied to incremental decision trees.

Motivation & Objective

• To address the lack of rigorous statistical analysis for confidence intervals in decision tree splitting criteria under streaming conditions.
• To develop tighter, more accurate confidence intervals for three key splitting criteria: entropy, Gini index, and Kearns-Mansour index.
• To extend the framework to a selective sampling (active learning) setting where the learner decides which labels to query to reduce labeling costs.
• To provide theoretical guarantees on the probability of selecting suboptimal splits under the proposed confidence bounds.
• To empirically validate that the proposed method achieves higher accuracy and better label efficiency than existing incremental decision tree techniques.

Proposed method

• Derives new confidence interval bounds for split gain estimation using Hoeffding-type inequalities, refined for entropy, Gini, and Kearns-Mansour criteria.
• Applies the Kullback-Leibler divergence and square root transformation techniques to derive tighter bounds on estimation error for the splitting criteria.
• Introduces a selective sampling strategy that uses confidence bounds to decide whether to query a label, minimizing labeling cost while maintaining accuracy.
• Uses a union bound and recursive application of concentration inequalities to bound the probability of suboptimal split selection across all nodes in the tree.
• Employs a time-dependent confidence level that decreases with the number of examples processed to maintain overall error control.
• Theoretical analysis proves that the probability of routing an example through a τ-suboptimal split is bounded by δ, with δ decreasing over time.

Experimental results

Research questions

• RQ1 Can tighter confidence intervals be derived for entropy, Gini, and Kearns-Mansour splitting criteria in streaming decision trees?
• RQ2 How can these refined confidence intervals improve split selection accuracy compared to standard Hoeffding bounds?
• RQ3 Can an active learning strategy based on these confidence intervals reduce labeling costs without sacrificing classification accuracy?
• RQ4 What theoretical guarantees can be provided on the probability of selecting suboptimal splits using the proposed method?
• RQ5 How does the proposed method compare in accuracy and label efficiency to existing incremental decision tree algorithms on real and synthetic streaming data?

Key findings

• The proposed confidence intervals lead to more accurate decision trees than those generated by standard Hoeffding trees, even with the same number of leaves.
• The active learning module significantly reduces labeling costs while maintaining or improving classification accuracy compared to other active learning strategies for incremental trees.
• Theoretical analysis shows that the probability of routing an example through a τ-suboptimal split is bounded by δ, with δ decreasing over time.
• Empirical results demonstrate that the method is more robust and consistent than recent active learning techniques applied to incremental decision trees.
• The use of KL divergence and square root transformations in the confidence bounds leads to tighter estimation error bounds than standard Hoeffding-style bounds.
• The method outperforms existing approaches on both synthetic and real-world streaming datasets in terms of accuracy and label efficiency.