[Paper Review] Superensemble Classifier for Improving Predictions in Imbalanced Datasets

Tanujit Chakraborty, Ashis Kumar Chakraborty|arXiv (Cornell University)|Oct 25, 2018

Imbalanced Data Classification Techniques39 references10 citations

TL;DR

This paper proposes a novel distribution-free superensemble classifier that combines Hellinger Distance Decision Trees (HDDT) and Radial Basis Function Networks (RBFN) to improve prediction performance on imbalanced datasets without data sampling. The method leverages HDDT's robustness to class imbalance and RBFN's fast convergence and universal consistency, achieving state-of-the-art performance with minimal hyperparameter tuning and enhanced interpretability across multiple real-world datasets.

ABSTRACT

Learning from an imbalanced dataset is a tricky proposition. Because these datasets are biased towards one class, most existing classifiers tend not to perform well on minority class examples. Conventional classifiers usually aim to optimize the overall accuracy without considering the relative distribution of each class. This article presents a superensemble classifier, to tackle and improve predictions in imbalanced classification problems, that maps Hellinger distance decision trees (HDDT) into radial basis function network (RBFN) framework. Regularity conditions for universal consistency and the idea of parameter optimization of the proposed model are provided. The proposed distribution-free model can be applied for feature selection cum imbalanced classification problems. We have also provided enough numerical evidence using various real-life data sets to assess the performance of the proposed model. Its effectiveness and competitiveness with respect to different state-of-the-art models are shown.

Motivation & Objective

• To address the challenge of poor minority class performance in imbalanced datasets, where conventional classifiers favor the majority class.
• To develop a distribution-free ensemble model that avoids data sampling, preserving original data structure and reducing computational overhead.
• To combine the strengths of HDDT (robust to class skew) and RBFN (fast convergence, universal consistency) into a single, interpretable superensemble framework.
• To provide a method that supports both feature selection and classification in small-to-medium-sized imbalanced datasets.
• To demonstrate competitive performance against state-of-the-art models, including sampling-based and single-classifier approaches.

Proposed method

• The proposed superensemble classifier maps trained HDDT models into the RBFN framework, using HDDT nodes as RBF centers to construct a hybrid model.
• Hellinger distance is used as the splitting criterion in HDDT to ensure robustness to class imbalance and reduce bias toward the majority class.
• The RBFN component uses radial basis functions with Gaussian kernels, enabling fast training and universal consistency under mild regularity conditions.
• Model parameters, including RBF widths and weights, are optimized via a gradient-based or heuristic method to minimize classification error.
• The ensemble framework ensures diversity and generalization by combining multiple HDDT-RBFN mappings, reducing overfitting and variance.
• The method is designed to be interpretable, with feature importance derived from RBFN weights and HDDT split rules.

Experimental results

Research questions

• RQ1 Can a hybrid HDDT-RBFN superensemble classifier outperform traditional classifiers and sampling-based methods on imbalanced datasets without modifying class distributions?
• RQ2 How does the proposed superensemble model maintain universal consistency while handling class imbalance?
• RQ3 To what extent does the model reduce the need for data sampling techniques like SMOTE or SMOTE+ENN in imbalanced classification?
• RQ4 How does the integration of HDDT and RBFN improve generalization and reduce overfitting compared to standalone models?
• RQ5 What is the impact of the proposed model on feature selection and interpretability in small-to-medium-sized imbalanced datasets?

Key findings

• The proposed superensemble classifier achieved the highest AUC in 5 out of 8 datasets, outperforming both single classifiers and sampling-based methods.
• On the Pima Diabetes dataset, the model achieved an AUC of 0.748 with SMOTE+ENN preprocessing, outperforming baseline models like k-NN and RF.
• In the Indian Business School dataset, the model reached an AUC of 0.914 with SMOTE+ENN, significantly outperforming RBFN (0.905) and RF (0.875) on the same data.
• The model demonstrated universal consistency and required minimal hyperparameter tuning, making it practical for real-world applications.
• The method showed strong performance on high-dimensional datasets like Page Blocks, achieving an AUC of 0.949 with SMOTE+ENN, close to the theoretical upper bound.
• The model maintained high accuracy and interpretability, outperforming complex black-box models while avoiding data sampling and preserving original data structure.