[Paper Review] RaSE: Random Subspace Ensemble Classification

Ye Tian, Yang Feng|arXiv (Cornell University)|Jun 16, 2020

Face and Expression Recognition53 references4 citations

TL;DR

This paper proposes RaSE, a novel ensemble classification framework for high-dimensional sparse classification that selects optimal subspaces using a new Ratio Information Criterion (RIC) based on weighted Kullback-Leibler divergence. The method improves classification accuracy and feature selection by iteratively refining subspace selection, achieving theoretical consistency and low misclassification rates even with limited random subspaces.

ABSTRACT

We propose a flexible ensemble classification framework, Random Subspace Ensemble (RaSE), for sparse classification. In the RaSE algorithm, we aggregate many weak learners, where each weak learner is a base classifier trained in a subspace optimally selected from a collection of random subspaces. To conduct subspace selection, we propose a new criterion, ratio information criterion (RIC), based on weighted Kullback-Leibler divergence. The theoretical analysis includes the risk and Monte-Carlo variance of the RaSE classifier, establishing the screening consistency and weak consistency of RIC, and providing an upper bound for the misclassification rate of the RaSE classifier. In addition, we show that in a high-dimensional framework, the number of random subspaces needs to be very large to guarantee that a subspace covering signals is selected. Therefore, we propose an iterative version of the RaSE algorithm and prove that under some specific conditions, a smaller number of generated random subspaces are needed to find a desirable subspace through iteration. An array of simulations under various models and real-data applications demonstrate the effectiveness and robustness of the RaSE classifier and its iterative version in terms of low misclassification rate and accurate feature ranking. The RaSE algorithm is implemented in the R package RaSEn on CRAN.

Motivation & Objective

• To address the challenge of sparse classification in high-dimensional data where most features are irrelevant.
• To develop a theoretically grounded ensemble method that avoids naively aggregating all random subspaces, which may include no informative signals.
• To enable accurate feature ranking and selection by optimizing subspace selection through a new information criterion.
• To reduce the number of required random subspaces by introducing an iterative refinement process that improves subspace quality over time.
• To establish theoretical consistency and upper bounds on misclassification error for the RaSE classifier.

Proposed method

• The RaSE framework constructs an ensemble by training base classifiers on randomly generated subspaces, but only retains the best-performing subspace within each group of subspaces.
• A new information criterion, the Ratio Information Criterion (RIC), is proposed to evaluate and select optimal subspaces based on weighted Kullback-Leibler divergence between class-conditional densities.
• Subspace selection is performed iteratively: in each iteration, feature weights are updated based on classification performance, and new subspaces are sampled with higher probability for features with higher estimated importance.
• The iterative process improves the likelihood of selecting subspaces that contain the true signal set $ S^* $, reducing the number of subspaces needed compared to non-iterative approaches.
• Theoretical analysis establishes screening consistency and weak consistency of RIC, and provides an upper bound on the misclassification rate.
• The algorithm is implemented in the R package RaSEn on CRAN for practical use in high-dimensional classification tasks.

Experimental results

Research questions

• RQ1 Can a random subspace ensemble method achieve theoretical consistency in high-dimensional sparse classification settings?
• RQ2 Does the proposed Ratio Information Criterion (RIC) lead to better subspace selection than naive aggregation of all random subspaces?
• RQ3 Can an iterative subspace selection procedure significantly reduce the number of required random subspaces while maintaining or improving classification performance?
• RQ4 How does RaSE perform in terms of feature ranking accuracy and misclassification rate compared to existing ensemble methods in high-dimensional data?
• RQ5 What theoretical guarantees can be established for the misclassification rate and subspace selection consistency of the RaSE classifier?

Key findings

• The RaSE classifier achieves screening consistency and weak consistency under the proposed RIC criterion, ensuring that the true signal set $ S^* $ is asymptotically recovered.
• Theoretical upper bounds on the misclassification rate are derived, showing that RaSE maintains low error rates even in high-dimensional settings.
• The iterative version of RaSE reduces the required number of random subspaces to find a subspace containing $ S^* $, as it progressively refines feature importance estimates.
• Simulations and real-data applications demonstrate that RaSE achieves low misclassification rates and accurate feature ranking across diverse models and data structures.
• The probability of selecting a subspace containing the true signal set $ S^* $ increases with the number of iterations and ensemble size, with the bound converging to 1 as $ n, B_1, B_2 o iginfty $.
• The R package RaSEn is available on CRAN, enabling practical deployment of the RaSE framework in real-world high-dimensional classification tasks.