[Paper Review] Feature Grouping as a Stochastic Regularizer for High-Dimensional Structured Data

Sergül Aydöre, Bertrand Thirion+1 more|arXiv (Cornell University)|Jul 31, 2018

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes a stochastic regularizer that groups correlated features via clustering and applies them randomly during stochastic gradient descent (SGD), effectively acting as a denoising regularizer. It improves generalization and convergence speed in high-dimensional, small-sample settings like fMRI and face images, outperforming $β$-regularization, dropout, and CNNs, especially under high noise and low data regimes.

ABSTRACT

In many applications where collecting data is expensive, for example neuroscience or medical imaging, the sample size is typically small compared to the feature dimension. It is challenging in this setting to train expressive, non-linear models without overfitting. These datasets call for intelligent regularization that exploits known structure, such as correlations between the features arising from the measurement device. However, existing structured regularizers need specially crafted solvers, which are difficult to apply to complex models. We propose a new regularizer specifically designed to leverage structure in the data in a way that can be applied efficiently to complex models. Our approach relies on feature grouping, using a fast clustering algorithm inside a stochastic gradient descent loop: given a family of feature groupings that capture feature covariations, we randomly select these groups at each iteration. We show that this approach amounts to enforcing a denoising regularizer on the solution. The method is easy to implement in many model architectures, such as fully connected neural networks, and has a linear computational cost. We apply this regularizer to a real-world fMRI dataset and the Olivetti Faces datasets. Experiments on both datasets demonstrate that the proposed approach produces models that generalize better than those trained with conventional regularizers, and also improves convergence speed.

Motivation & Objective

• Address overfitting in high-dimensional, small-sample datasets common in neuroimaging and medical applications.
• Develop a structured regularizer that leverages feature correlations without requiring custom solvers for complex models.
• Enable efficient regularization in non-convex models like deep neural networks using SGD.
• Improve model robustness to noise and sample size limitations in real-world data such as fMRI and face images.
• Provide a computationally efficient, plug-and-play regularizer compatible with standard deep learning frameworks.

Proposed method

• Use a fast clustering algorithm to group features based on their correlations, capturing underlying data structure.
• Randomly sample these feature groups at each SGD update, introducing stochasticity to the optimization process.
• Apply the grouped features as a form of input-level data augmentation, effectively smoothing the input space.
• The method implicitly enforces a denoising regularizer by averaging over feature clusters, reducing sensitivity to noisy features.
• Integrate the regularizer into the SGD loop with linear computational cost, requiring no architectural changes.
• The approach is compatible with any model architecture, including deep networks, and operates on the input layer where parameter count is highest.

Experimental results

Research questions

• RQ1 Can feature grouping via clustering serve as an effective, general-purpose regularizer in high-dimensional, small-sample settings?
• RQ2 How does the proposed stochastic regularizer compare to standard regularizers like $β$-regularization and dropout in terms of generalization and convergence speed?
• RQ3 Does the method improve robustness to noise and small sample sizes in real-world neuroimaging and vision datasets?
• RQ4 Can this regularizer be efficiently applied to non-convex models like deep neural networks without specialized solvers?
• RQ5 What is the computational cost and memory footprint of the proposed method compared to existing approaches?

Key findings

• On the Olivetti Faces dataset, the feature grouping regularizer achieved higher test accuracy than $β$-regularization, dropout, and CNNs under high noise, with faster convergence.
• For the HCP-small fMRI dataset, the method outperformed all baselines in low-sample regimes, especially when the number of training samples was small.
• The learned weights from the feature grouping approach were visually less noisy than those from other regularizers, explaining its superior performance in noisy settings.
• The method maintained high accuracy even under high Gaussian noise, demonstrating strong robustness to data corruption.
• The computational cost of the regularizer is linear in feature dimension, and it reduces memory usage compared to standard approaches.
• The approach achieved better generalization than CNNs on fMRI data, despite CNNs' inductive bias for translation invariance, which was detrimental for brain images.