[Paper Review] Cost-Sensitive Tree of Classifiers

Zhixiang Xu, Matt J. Kusner+2 more|PolyPublie (École Polytechnique de Montréal)|Oct 9, 2012

Machine Learning and Data ClassificationComputer Science29 references75 citations

TL;DR

This paper proposes Cost-Sensitive Tree of Classifiers (CSTC), a tree-structured ensemble that dynamically allocates expensive features to specific input subregions based on cost-accuracy trade-offs. By modeling expected test-time cost via a mixed-norm relaxation and jointly optimizing all classifiers under a single global loss, CSTC achieves state-of-the-art accuracy at a fraction of the computational cost, especially effective in high-cost feature scenarios like web search ranking.

ABSTRACT

Recently, machine learning algorithms have successfully entered large-scale real-world industrial applications (e.g. search engines and email spam filters). Here, the CPU cost during test time must be budgeted and accounted for. In this paper, we address the challenge of balancing the test-time cost and the classifier accuracy in a principled fashion. The test-time cost of a classifier is often dominated by the computation required for feature extraction-which can vary drastically across eatures. We decrease this extraction time by constructing a tree of classifiers, through which test inputs traverse along individual paths. Each path extracts different features and is optimized for a specific sub-partition of the input space. By only computing features for inputs that benefit from them the most, our cost sensitive tree of classifiers can match the high accuracies of the current state-of-the-art at a small fraction of the computational cost.

Motivation & Objective

• To address the challenge of balancing test-time computational cost and classifier accuracy in large-scale machine learning applications.
• To reduce energy waste and monetary costs by avoiding unnecessary feature extraction for inputs that do not benefit from expensive features.
• To enable dynamic, input-dependent feature selection that allocates costly features only where they improve classification performance.
• To provide a principled framework for cost-sensitive learning that explicitly models expected test-time cost during training.
• To outperform existing cascade-based and regularization-based methods in cost-accuracy trade-offs on real-world data.

Proposed method

• Constructs a tree of classifiers where each path corresponds to a unique feature set and input subregion.
• Models the expected test-time cost of an input traversing the tree using a probabilistic traversal framework.
• Relaxes the expected cost using a mixed-norm relaxation to enable continuous optimization.
• Optimizes all classifiers jointly via a single global loss function that balances accuracy and test-time cost.
• Uses a cost-sensitive training objective that prioritizes feature extraction in deeper, less frequently traversed paths.
• Applies fine-tuning to prediction nodes to improve ranking performance without increasing test-time cost.

Experimental results

Research questions

• RQ1 How can we model and minimize the expected test-time cost of a tree-structured classifier system?
• RQ2 Can we jointly optimize classifier parameters and feature allocation to balance accuracy and computational cost?
• RQ3 Does a tree-based approach with input-dependent feature selection outperform linear cascades in balanced, feature-rich learning tasks?
• RQ4 How does the allocation of expensive features vary across different input subregions in the tree?
• RQ5 To what extent does CSTC maintain high ranking performance (e.g., NDCG) under strict computational budgets?

Key findings

• CSTC significantly outperforms state-of-the-art methods like Cronus and early-exit baselines in the cost-accuracy trade-off, especially under low-cost budgets.
• On the Yahoo! web-search ranking dataset, CSTC maintains high NDCG scores at a fraction of the computational cost of baseline models.
• Expensive features (cost ≥20) are predominantly extracted in deeper nodes, where they are used only for small, hard-to-classify input subgroups.
• Cheap features (cost ≤5) are extracted early in the tree, ensuring low-cost filtering of easily classifiable inputs.
• The Jaccard similarity between classifiers decreases with increasing distance in the tree, confirming that different regions use distinct feature sets.
• Fine-tuning provides minimal improvement in NDCG, indicating that the main performance gains come from the tree structure and cost-aware feature allocation.