[Paper Review] CatBoost: gradient boosting with categorical features support

Anna Veronika Dorogush, Vasily Ershov+1 more|arXiv (Cornell University)|Oct 24, 2018

Music and Audio ProcessingComputer Science15 references1,325 citations

TL;DR

CatBoost is an open-source gradient boosting library that natively handles categorical features and, on several public datasets, outperforms XGBoost, LightGBM, and H2O in accuracy (logloss) with GPU and CPU implementations.

ABSTRACT

In this paper we present CatBoost, a new open-sourced gradient boosting library that successfully handles categorical features and outperforms existing publicly available implementations of gradient boosting in terms of quality on a set of popular publicly available datasets. The library has a GPU implementation of learning algorithm and a CPU implementation of scoring algorithm, which are significantly faster than other gradient boosting libraries on ensembles of similar sizes.

Motivation & Objective

• Motivate the need for gradient boosting methods that effectively utilize categorical features without heavy preprocessing.
• Introduce CatBoost as a new gradient boosting algorithm tailored for categorical data.
• Show that CatBoost achieves higher predictive quality than existing GBDT libraries on diverse datasets.
• Describe how CatBoost mitigates overfitting through a new leaf-value computation scheme and gradient bias handling.

Proposed method

• Present a gradient boosting algorithm that handles categorical features during training using a novel statistics-based encoding with priors.
• Use oblivious trees as base predictors with a balanced, fixed-depth structure.
• Introduce a gradient-bias fighting mechanism that trains separate models to estimate unbiased gradients for each example.
• Employ a fast CPU scorer by binarizing features and using binary feature vectors for leaf indexing.
• Provide a GPU-accelerated training path using histogram-based split finding and group-based histograms to improve throughput.
• Support for feature combinations and on-the-fly generation of combination features while avoiding combinatorial explosion.

Experimental results

Research questions

• RQ1 Can CatBoost improve predictive accuracy on datasets with high-cardinality categorical features compared to XGBoost, LightGBM, and H2O?
• RQ2 Does CatBoost provide tangible training speed advantages on GPUs while maintaining or improving model quality?
• RQ3 How does CatBoost mitigate gradient bias and overfitting in gradient boosting with categorical features?
• RQ4 What are the practical implications of using feature combinations in CatBoost for capturing interactions among categorical features?

Key findings

• CatBoost outperforms XGBoost, LightGBM, and H2O on a set of public benchmark datasets in terms of logloss for classification tasks.
• GPU implementation of CatBoost trains significantly faster than CPU versions and outperforms competing GPU GBMs on ensembles of similar size.
• CatBoost provides faster model scoring on CPU compared with XGBoost and LightGBM for comparable ensemble sizes.
• Using several permutations for statistics calculation and a prior-based encoding reduces overfitting for high-cardinality categories.
• Leaf value calculation and gradient-bias mitigation strategies contribute to improved generalization over standard GBDT approaches.
• On the Epsilon and other datasets, CatBoost shows favorable training speed and scoring performance compared to XGBoost and LightGBM when configured with reasonable bin counts.