[Paper Review] Exploiting Categorical Structure Using Tree-Based Methods

Brian Lucena|arXiv (Cornell University)|Apr 15, 2020

Data Mining Algorithms and Applications15 references4 citations

TL;DR

This paper proposes Structured Categorical Decision Trees (SCDT), a method that leverages inherent structure in categorical variables—such as geographical adjacency or temporal cycles—by defining a terrain of meaningful groupings for splitting. By integrating this structure into gradient boosting, SCDT significantly outperforms one-hot and ordinal encoding, even when searching only a small fraction of possible splits, demonstrating that structured categorical modeling improves predictive performance without exhaustive search.

ABSTRACT

Standard methods of using categorical variables as predictors either endow them with an ordinal structure or assume they have no structure at all. However, categorical variables often possess structure that is more complicated than a linear ordering can capture. We develop a mathematical framework for representing the structure of categorical variables and show how to generalize decision trees to make use of this structure. This approach is applicable to methods such as Gradient Boosted Trees which use a decision tree as the underlying learner. We show results on weather data to demonstrate the improvement yielded by this approach.

Motivation & Objective

• To address the limitation of standard methods that treat categorical variables as either unordered or linearly ordered, which fails to capture complex real-world structures.
• To develop a mathematical framework for representing categorical variable structure beyond ordinal or unordered assumptions.
• To design a decision tree variant that can exploit this structure during learning, particularly in gradient boosting frameworks.
• To empirically validate that structured categorical modeling improves predictive accuracy on real-world data with non-trivial variable structure.
• To demonstrate that high performance can be achieved with minimal search over the partition space, avoiding overfitting.

Proposed method

• Define a 'terrain' as a set of subsets of categorical variable levels that represent meaningful groupings, such as neighboring states or related months.
• Use the terrain to guide splits in decision trees, ensuring that splits respect the underlying structure of the categorical variable.
• Integrate the structured decision tree into a gradient boosting framework, enabling end-to-end learning with structured categorical features.
• Limit the number of candidate splits per node (e.g., max-splits-to-search) to control complexity and prevent overfitting.
• Use proximity based on the terrain to aggregate information across structurally similar categories, improving generalization.
• Implement and evaluate the method on weather data with structured predictors (e.g., U.S. states, months), comparing against one-hot and ordinal encodings.

Experimental results

Research questions

• RQ1 Can categorical variables with non-ordinal structure—such as geographical adjacency or circular time—be effectively modeled to improve predictive performance?
• RQ2 How can a mathematical framework be constructed to represent and exploit complex categorical structures beyond linear ordering?
• RQ3 Does using structured splits in decision trees lead to better generalization than standard one-hot or ordinal encoding in gradient boosting?
• RQ4 Is it possible to achieve strong performance with only a small subset of candidate splits, avoiding exhaustive search over the full partition space?
• RQ5 Can the proposed method generalize to other structured problems, such as image classification with semantically similar classes?

Key findings

• The Structured Categorical Decision Tree (SCDT) significantly outperforms both one-hot and ordinal encoding on weather prediction tasks with structured categorical predictors.
• On smaller datasets, the performance gap between SCDT and baseline methods is largest, indicating that structured aggregation provides substantial gains when data is limited.
• SCDT achieves strong performance while searching only a tiny fraction of the total possible splits—e.g., 5 out of 36 for a 9-level county variable—demonstrating efficiency and robustness.
• Increasing the number of candidate splits per node worsened performance on smaller datasets, likely due to overfitting, suggesting that limited search acts as a regularizer.
• The method remains effective even without advanced regularization techniques like shrinkage, highlighting its inherent inductive bias from structural constraints.
• The results confirm that meaningful signal exists in categorical variable structure beyond what ordinal or unordered encodings can capture, and that this signal can be effectively exploited in tree-based models.