[Paper Review] Encoding categorical data: Is there yet anything 'hotter' than one-hot encoding?

Ekaterina Poslavskaya, А. А. Королев|arXiv (Cornell University)|Dec 28, 2023

Machine Learning and Data Classification9 citations

TL;DR

The study evaluates multiple encoding schemes across a large OpenML dataset collection using linear mixed-effects models, finding that in multiclass tasks one-hot and Helmert coding outperform target-based encoders, while in binary tasks differences are modest.

ABSTRACT

Categorical features are present in about 40% of real world problems, highlighting the crucial role of encoding as a preprocessing component. Some recent studies have reported benefits of the various target-based encoders over classical target-agnostic approaches. However, these claims are not supported by any statistical analysis, and are based on a single dataset or a very small and heterogeneous sample of datasets. The present study explores the encoding effects in an exhaustive sample of classification problems from OpenML repository. We fitted linear mixed-effects models to the experimental data, treating task ID as a random effect, and the encoding scheme and the various characteristics of categorical features as fixed effects. We found that in multiclass tasks, one-hot encoding and Helmert contrast coding outperform target-based encoders. In binary tasks, there were no significant differences across the encoding schemes; however, one-hot encoding demonstrated a marginally positive effect on the outcome. Importantly, we found no significant interactions between the encoding schemes and the characteristics of categorical features. This suggests that our findings are generalizable to a wide variety of problems across domains.

Motivation & Objective

• Assess how encoding type impacts predictive performance across a wide, exhaustively sampled set of classification problems.
• Determine whether target-based encoders provide gains over target-agnostic encoders in real-world datasets.
• Evaluate whether encoding choices affect training/inference efficiency and model-based feature importance.
• Investigate interactions between encoding schemes and dataset characteristics (categorical feature prevalence, cardinality, etc.).
• Examine generalizability of encoding effects across binary and multiclass tasks.

Proposed method

• Assembled an exhaustive OpenML dataset collection of binary and multiclass classification tasks.
• Applied five encoders (label, one-hot, Helmert, target, weight of evidence) to all categorical features.
• Used H2O AutoML to train multiple base models and stacked ensembles on each encoded dataset.
• Fitted linear mixed-effects models with task ID as random effect and encoding type and feature characteristics as fixed effects.
• Compared performance across linear and non-linear models (GLM and XGBoost) and across binary vs multiclass tasks.

Experimental results

Research questions

• RQ1 Does encoding type influence the predictive power of a heterogeneous ensemble (stacked ensemble) across binary and multiclass tasks?
• RQ2 Does encoding type affect the predictive power of individual base estimators (linear vs non-linear)?
• RQ3 How do encoding schemes impact training duration and efficiency across different dataset characteristics?
• RQ4 Are there interactions between encoding schemes and categorical feature characteristics in determining performance?
• RQ5 Do results generalize across diverse real-world classification problems from OpenML?

Key findings

• In binary tasks, the proportion of categorical features negatively affects ensemble performance, while higher average cardinality can improve it; high cardinality can hurt performance.
• In multiclass tasks, the number of target classes is a major determinant of performance, with OHE and Helmert encodings showing some improvement over target-based encoders for the stacked ensemble.
• For linear models, label encoding tends to reduce predictive power in binary tasks, and target-based/indicator encodings generally outperform label encoding in multiclass tasks.
• Across models, encoding type had limited or no significant impact on the ensemble’s performance in binary tasks, but some benefits of OHE and Helmert were observed in multiclass GLM/XGBoost results.
• Target-based encoders and OHE can offer advantages over label encoding in certain multiclass scenarios, though results vary by model type.