[Paper Review] Neural Networks Regularization Through Invariant Features Learning

Soufiane Belharbi, Clément Chatelain+2 more|arXiv (Cornell University)|Sep 6, 2017

Domain Adaptation and Few-Shot Learning3 citations

TL;DR

This paper proposes a regularization method that encourages deep neural networks to learn class-wise invariant features, making them more robust when training data is scarce. By constraining hidden layers to produce identical representations for samples of the same class, the approach improves generalization on few-shot classification tasks, as demonstrated by superior performance on MNIST and its variants.

ABSTRACT

Training deep neural networks is known to require a large number of training samples. However, in many applications only few training samples are available. In this work, we tackle the issue of training neural networks for classification task when few training samples are available. We attempt to solve this issue by proposing a regularization term that constrains the hidden layers of a network to learn class-wise invariant features. In our regularization framework, learning invariant features is generalized to the class membership where samples with the same class should have the same feature representation. Numerical experiments over MNIST and its variants showed that our proposal is more efficient for the case of few training samples. Moreover, we show an intriguing property of representation learning within neural networks. The source code of our framework is freely available this https URL

Motivation & Objective

• Address the challenge of training deep neural networks when only a small number of training samples are available.
• Improve generalization in low-data regimes by enforcing invariance of features across samples of the same class.
• Explore the role of representation learning in enhancing model robustness under data scarcity.
• Develop a regularization framework that generalizes feature invariance to class membership rather than just data augmentation.

Proposed method

• Introduce a novel regularization term that penalizes differences in hidden layer representations between samples of the same class.
• Generalize feature invariance beyond data augmentation to include all samples sharing the same class label.
• Optimize the network with a combined loss function that includes standard classification loss and the proposed invariance regularization.
• Apply the regularization during backpropagation to encourage consistent feature learning across class members.
• Use feature-level constraints to reduce overfitting in low-data scenarios without modifying network architecture.
• Train the model end-to-end using stochastic gradient descent with the regularized objective function.

Experimental results

Research questions

• RQ1 Can enforcing class-wise feature invariance improve generalization in deep neural networks under low-data conditions?
• RQ2 How does the proposed regularization compare to standard data augmentation in few-shot learning settings?
• RQ3 What is the impact of feature invariance on representation learning within deep networks?
• RQ4 Does the regularization method maintain performance across different variants of the MNIST dataset with limited samples?

Key findings

• The proposed regularization method significantly improves classification accuracy on MNIST and its variants when only a few training samples are available.
• The model achieves better generalization than standard approaches, particularly in low-data regimes, due to stronger feature invariance.
• The method demonstrates an intriguing property of representation learning: class-wise invariance enhances feature compactness and discriminability.
• The framework is effective even without explicit data augmentation, showing the robustness of the invariance constraint.
• The source code is publicly available, enabling reproducibility and further research in few-shot representation learning.
• Numerical experiments confirm that the regularization reduces overfitting and improves generalization performance on small datasets.