[Paper Review] A Nonparametric Ensemble Binary Classifier and its Statistical Properties

Tanujit Chakraborty, Ashis Kumar Chakraborty+1 more|arXiv (Cornell University)|Apr 29, 2018

Neural Networks and ApplicationsComputer Science33 references18 citations

TL;DR

This paper proposes a nonparametric ensemble binary classifier combining classification trees (CT) and artificial neural networks (ANN) to achieve universal consistency and improved accuracy in high-dimensional, small-to-medium medical datasets. The method uses CT for feature selection and as an input feature in a single-hidden-layer ANN, with an analytically derived optimal number of neurons, resulting in superior performance and reduced hyperparameter tuning compared to state-of-the-art models.

ABSTRACT

In this work, we propose an ensemble of classification trees (CT) and artificial neural networks (ANN). Several statistical properties including universal consistency and upper bound of an important parameter of the proposed classifier are shown. Numerical evidence is also provided using various real life data sets to assess the performance of the model. Our proposed nonparametric ensemble classifier doesn't suffer from the `curse of dimensionality' and can be used in a wide variety of feature selection cum classification problems. Performance of the proposed model is quite better when compared to many other state-of-the-art models used for similar situations.

Motivation & Objective

• To develop a nonparametric ensemble classifier that combines the strengths of classification trees (CT) and artificial neural networks (ANN) for improved classification accuracy.
• To establish theoretical guarantees, including universal consistency and an upper bound on the number of hidden neurons, for the proposed ensemble model.
• To reduce the number of hyperparameters and improve interpretability compared to deep or complex neural networks.
• To enable effective feature selection and classification in high-dimensional, small-to-medium-sized datasets, especially in medical applications.
• To bridge the gap between empirical success and theoretical justification in hybrid CT-ANN models.

Proposed method

• First, train a classification tree (CT) on the data to identify important features and generate class predictions.
• Use the CT-predicted class labels as an additional input feature in a single-hidden-layer feedforward neural network (ANN).
• Train the ANN using only the selected features from CT and the CT output, with sigmoid activation and min-max normalized input data.
• Derive the optimal number of hidden neurons as O(√(n / (dm log n))), where n is the number of training samples and dm is the number of input features in the ANN.
• Apply the ensemble model to real-world medical datasets, comparing performance against state-of-the-art classifiers including RF, SVM, and DNDT.
• Use the neuralnet R package for ANN training, ensuring low computational cost and memory usage.

Experimental results

Research questions

• RQ1 Can a hybrid CT-ANN ensemble model achieve universal consistency while maintaining low computational cost and minimal hyperparameter tuning?
• RQ2 Does incorporating CT-predicted class labels as an input feature enhance the classification performance of the subsequent ANN?
• RQ3 What is the optimal number of neurons in the hidden layer of the ANN component for the proposed ensemble model?
• RQ4 How does the proposed model compare in accuracy and robustness to existing state-of-the-art classifiers on high-dimensional, small-to-medium medical datasets?
• RQ5 To what extent does the CT-based feature selection improve model performance and interpretability in the ensemble framework?

Key findings

• The proposed ensemble CT-ANN model achieves universal consistency, providing a strong theoretical foundation for its reliability.
• The optimal number of neurons in the hidden layer is theoretically derived as O(√(n / (dm log n))), significantly reducing the need for hyperparameter tuning.
• On six real medical datasets, the model achieved the highest overall accuracy (97.30% on Wisconsin breast cancer) and F-measure (0.98) among all compared models.
• The model outperformed state-of-the-art models such as Random Forest, SVM, and Deep Neural Decision Trees (DNDT) on most datasets, especially in feature selection and classification accuracy.
• The model demonstrated low computational cost and memory usage, with training times and requirements significantly lower than GPU-dependent models like DNDT.
• The inclusion of CT output as an input feature increased class separability and contributed to improved performance, particularly in complex, high-dimensional feature spaces.