[Paper Review] Robust Classification of High Dimension Low Sample Size Data

Neclâ Gündüz, Ernest Fokoué|arXiv (Cornell University)|Jan 3, 2015

Advanced Statistical Methods and Models24 references20 citations

TL;DR

This paper evaluates robust classification methods for high-dimensional, low-sample-size (HDLSS) data, comparing techniques like robust discriminant analysis, robust PCA, and random forest. Despite being non-robust by design, random forest consistently outperforms all other methods in predictive accuracy across real and simulated HDLSS datasets with contamination, demonstrating inherent robustness through bootstrap subsampling and variable selection.

ABSTRACT

The robustification of pattern recognition techniques has been the subject of intense research in recent years. Despite the multiplicity of papers on the subject, very few articles have deeply explored the topic of robust classification in the high dimension low sample size context. In this work, we explore and compare the predictive performances of robust classification techniques with a special concentration on robust discriminant analysis and robust PCA applied to a wide variety of large $p$ small $n$ data sets. We also explore the performance of random forest by way of comparing and contrasting the differences single model methods and ensemble methods in this context. Our work reveals that Random Forest, although not inherently designed to be robust to outliers, substantially outperforms the existing techniques specifically designed to achieve robustness. Indeed, random forest emerges as the best predictively on both real life and simulated data.

Motivation & Objective

• To evaluate and compare the predictive performance of robust classification techniques in high-dimensional, low-sample-size (HDLSS) data settings where n ≪ p.
• To investigate how contamination and high dimensionality affect the performance of traditional and robust classification methods.
• To assess whether ensemble methods like random forest, despite not being explicitly robust, achieve superior predictive performance in HDLSS contexts with outliers.
• To explore the limitations of projection pursuit and robust discriminant analysis in multicategorical and high-dimensional scenarios.
• To identify the most practical and robust classification techniques for real-world HDLSS data, particularly in biomedical applications like cancer classification.

Proposed method

• Employs a repeated cross-validation framework with R replications to estimate average test error (AVTE) using zero-one loss for predictive performance evaluation.
• Applies robust classification techniques including Minimum Covariance Determinant (MCD), projection pursuit, SIMCA, regularized discriminant analysis, and robust PCA.
• Uses random forest with bootstrap sampling and random feature subspace selection to inherently reduce the influence of outliers and handle high dimensionality.
• Evaluates methods on both real HDLSS datasets (prostate, lymphoma, lung, colon, leukemia, brain cancer) and simulated data with controlled contamination levels.
• Compares performance across varying dimensions (p), contamination rates (κ), and correlation structures (ρ) in simulated data.
• Analyzes the effect of variable correlation and class count (G=2 or G=3) on method robustness and predictive accuracy.

Experimental results

Research questions

• RQ1 How do robust classification techniques perform in high-dimensional, low-sample-size (HDLSS) data with contamination?
• RQ2 Does random forest, despite not being explicitly designed for robustness, achieve superior predictive performance in HDLSS settings with outliers?
• RQ3 Why does projection pursuit fail in multicategorical classification despite success in binary classification under high correlation?
• RQ4 How does the combination of variable selection and bootstrap subsampling in random forest contribute to its robustness in HDLSS data?
• RQ5 What is the relative performance of SIMCA and MCD-based methods compared to random forest and other robust techniques under varying contamination levels and dimensionality?

Key findings

• Random forest consistently achieved the best or second-best predictive performance across all real and simulated HDLSS datasets, with no instance of worst performance.
• In strongly contaminated simulated data with g=2 and ρ=0.75, random forest achieved the lowest average test error, outperforming even MCD and projection pursuit.
• Projection pursuit performed best only in binary classification with high variable correlation (ρ=0.75), failing in multicategorical and low-correlation settings.
• SIMCA performed well under strong contamination, often ranking second to random forest, indicating strong practicality in contaminated HDLSS scenarios.
• Robust discriminant analysis and PCA-based methods showed limited effectiveness in high-dimensional, low-sample-size settings, especially with multicategorical data.
• The bootstrap mechanism in random forest inherently excludes approximately 37% of training samples (e⁻¹), which likely contributes to outlier resistance by averaging out their influence.