[Paper Review] Let the Data Choose its Features: Differentiable Unsupervised Feature Selection.

Ofir Lindenbaum, Uri Shaham+3 more|arXiv (Cornell University)|Jul 9, 2020

Neural Networks and Applications4 citations

TL;DR

This paper proposes a differentiable, unsupervised feature selection method for clustering that uses a continuous relaxation of Bernoulli variables to gate features and a graph Laplacian to score low-frequency, informative features. The approach jointly optimizes feature selection and clustering by learning which features to retain through a differentiable loss, demonstrating superior performance over baselines on real-world datasets.

ABSTRACT

Scientific observations often consist of a large number of variables (features). Identifying a subset of meaningful features is often ignored in unsupervised learning, despite its potential for unraveling clear patterns hidden in the ambient space. In this paper, we present a method for unsupervised feature selection, tailored for the task of clustering. We propose a differentiable loss function which combines the graph Laplacian with a gating mechanism based on continuous approximation of Bernoulli random variables. The Laplacian is used to define a scoring term that favors low-frequency features, while the parameters of the Bernoulli variables are trained to enable selection of the most informative features. We mathematically motivate the proposed approach and demonstrate that in the high noise regime, it is crucial to compute the Laplacian on the gated inputs, rather than on the full feature set. Experimental demonstration of the efficacy of the proposed approach and its advantage over current baselines is provided using several real-world examples.

Motivation & Objective

• To address the lack of feature selection in unsupervised learning, especially in clustering tasks.
• To identify a subset of meaningful features from high-dimensional data without labeled supervision.
• To develop a differentiable mechanism that enables end-to-end training of feature selection and clustering.
• To mathematically justify the use of the graph Laplacian on gated (selected) inputs rather than full input space in high-noise regimes.
• To demonstrate the method's effectiveness on real-world datasets, showing advantages over existing unsupervised feature selection baselines.

Proposed method

• The method introduces a differentiable loss function combining a graph Laplacian with a gating mechanism based on continuous relaxation of Bernoulli random variables.
• The graph Laplacian is computed on the gated feature inputs, favoring low-frequency components that correspond to informative, stable features.
• The Bernoulli parameters are differentiable and trained end-to-end to learn which features to select, enabling gradient-based optimization.
• The gating mechanism allows soft selection of features, where each feature's contribution is determined by a learnable gate parameter.
• The loss function balances clustering quality and feature selection by penalizing high-frequency components and promoting sparse, informative feature sets.
• The approach is theoretically justified in high-noise regimes, where computing the Laplacian on full data leads to poor performance compared to on gated inputs.

Experimental results

Research questions

• RQ1 How can unsupervised feature selection be made differentiable to enable end-to-end training with clustering objectives?
• RQ2 What is the optimal way to score and select features in high-dimensional, unlabeled data for clustering?
• RQ3 Why is computing the graph Laplacian on gated inputs more effective than on the full feature set in high-noise settings?
• RQ4 Can a continuous relaxation of feature selection via Bernoulli gates improve clustering performance compared to discrete or non-differentiable methods?
• RQ5 How does the proposed method compare to existing unsupervised feature selection baselines on real-world datasets?

Key findings

• The method achieves improved clustering performance by learning to select informative features through a differentiable, continuous gating mechanism.
• The use of the graph Laplacian on gated inputs, rather than the full input space, is critical in high-noise regimes for maintaining performance.
• The continuous relaxation of Bernoulli variables enables effective backpropagation through the feature selection process, allowing joint optimization of selection and clustering.
• Empirical results on real-world datasets show the proposed method outperforms existing unsupervised feature selection baselines in clustering accuracy and stability.
• The method demonstrates robustness to noise and high dimensionality, with clear improvements in identifying meaningful feature subsets.
• The theoretical analysis confirms that the Laplacian on gated features better captures low-frequency, stable patterns in noisy data.