[Paper Review] Spectral feature scaling method for supervised dimensionality reduction

Momo Matsuda, Keiichi Morikuni+1 more|arXiv (Cornell University)|May 18, 2018

Face and Expression Recognition20 references3 citations

TL;DR

This paper proposes a supervised dimensionality reduction method that uses spectral feature scaling to improve clustering and classification by learning optimal feature scaling factors from labeled partial data. By formulating a generalized eigenvalue problem based on the Fiedler vector and a linear matrix pencil, the method adaptively rescales features to enhance separability in the reduced space, outperforming existing methods in both synthetic and real-world gene expression datasets, especially in high-dimensional, low-sample regimes.

ABSTRACT

Spectral dimensionality reduction methods enable linear separations of complex data with high-dimensional features in a reduced space. However, these methods do not always give the desired results due to irregularities or uncertainties of the data. Thus, we consider aggressively modifying the scales of the features to obtain the desired classification. Using prior knowledge on the labels of partial samples to specify the Fiedler vector, we formulate an eigenvalue problem of a linear matrix pencil whose eigenvector has the feature scaling factors. The resulting factors can modify the features of entire samples to form clusters in the reduced space, according to the known labels. In this study, we propose new dimensionality reduction methods supervised using the feature scaling associated with the spectral clustering. Numerical experiments show that the proposed methods outperform well-established supervised methods for toy problems with more samples than features, and are more robust regarding clustering than existing methods. Also, the proposed methods outperform existing methods regarding classification for real-world problems with more features than samples of gene expression profiles of cancer diseases. Furthermore, the feature scaling tends to improve the clustering and classification accuracies of existing unsupervised methods, as the proportion of training data increases.

Motivation & Objective

• To address the challenge of poor clustering performance in high-dimensional data due to feature irregularities and uncertainties.
• To improve spectral clustering and classification by learning feature scaling factors using partial label information.
• To develop a supervised method that enhances linear separability in the reduced feature space through adaptive feature scaling.
• To demonstrate robustness and superiority over existing methods in both toy problems and real-world gene expression datasets.

Proposed method

• The method formulates a generalized eigenvalue problem involving a linear matrix pencil, where the eigenvector represents the feature scaling factors.
• It uses prior knowledge of partial sample labels to specify the Fiedler vector, which guides the scaling process toward desired cluster structures.
• The feature scaling is applied to all samples, modifying variances and means to improve separability in the reduced space.
• The approach integrates with spectral clustering by transforming features before applying the Laplacian eigenmap, enabling better cluster formation.
• The method is designed to be robust in high-dimensional, low-sample settings, common in gene expression analysis.
• It is applicable as a preprocessing step to enhance existing unsupervised methods like LPP and LFDA.

Experimental results

Research questions

• RQ1 Can feature scaling based on partial label information improve the performance of spectral clustering in high-dimensional data?
• RQ2 How does supervised feature scaling compare to existing supervised dimensionality reduction methods in terms of clustering and classification accuracy?
• RQ3 To what extent does feature scaling enhance the robustness of dimensionality reduction in real-world gene expression datasets with more features than samples?
• RQ4 Can the proposed method improve the performance of unsupervised spectral clustering when combined with learned scaling factors?
• RQ5 Does the proposed method maintain or improve performance as the proportion of labeled training data increases?

Key findings

• For toy problems with more samples than features, the proposed method outperformed established supervised methods in clustering and classification accuracy.
• On real-world gene expression datasets (e.g., ovarian, pancreatic, IPF, colorectal), the proposed method achieved higher Rand Index (RI) and Normalized Mutual Information (NMI) scores than existing methods.
• For the IPF dataset, SC-S achieved a RI of 76.1% ± 0.00 and NMI of 0.093 ± 0.000, significantly outperforming baseline methods.
• In classification tasks, SC-S achieved 97.4% ± 4.43 accuracy on the IPF dataset, surpassing other methods including KLFDA and LFDA.
• The method demonstrated improved robustness in clustering, particularly when the proportion of labeled training data increased.
• Feature scaling consistently enhanced the performance of existing unsupervised methods like LPP and LFDA, especially in high-dimensional settings.