[Paper Review] Generalization Error of Invariant Classifiers

Jure Sokolić, Raja Giryes+2 more|arXiv (Cornell University)|Oct 14, 2016

Neural Networks and Applications25 references19 citations

TL;DR

This paper establishes a theoretical framework showing that invariant classifiers—designed to be robust to specific input transformations—achieve significantly lower generalization error (GE) than non-invariant counterparts. By factoring the input space into a base space and transformation group, it proves that GE scales with the complexity of the base space, not the full input space, and that invariance can reduce GE by up to a factor of √T, where T is the number of transformations.

ABSTRACT

This paper studies the generalization error of invariant classifiers. In particular, we consider the common scenario where the classification task is invariant to certain transformations of the input, and that the classifier is constructed (or learned) to be invariant to these transformations. Our approach relies on factoring the input space into a product of a base space and a set of transformations. We show that whereas the generalization error of a non-invariant classifier is proportional to the complexity of the input space, the generalization error of an invariant classifier is proportional to the complexity of the base space. We also derive a set of sufficient conditions on the geometry of the base space and the set of transformations that ensure that the complexity of the base space is much smaller than the complexity of the input space. Our analysis applies to general classifiers such as convolutional neural networks. We demonstrate the implications of the developed theory for such classifiers with experiments on the MNIST and CIFAR-10 datasets.

Motivation & Objective

• To develop a theoretical framework linking invariance to reduced generalization error in machine learning.
• To analyze how invariance to transformations affects the complexity and generalization performance of classifiers.
• To derive conditions under which invariant classifiers achieve substantially lower generalization error than non-invariant ones.
• To connect the theory to practical deep learning models like CNNs and scattering transforms.
• To provide bounds on generalization error that explicitly account for invariance and transformation geometry.

Proposed method

• Factor the input space X into a base space B and a transformation group G, such that X ≈ B × G.
• Define stable invariant classifiers using the spectral norm of the Jacobian matrix, ensuring bounded sensitivity to input perturbations.
• Use covering numbers to measure the complexity of the base space B and the full input space X under the Euclidean metric.
• Derive generalization error bounds based on the covering number of the base space, showing that GE scales with N(B; d, ε), not N(X; d, ε).
• Establish sufficient geometric conditions on B and G such that the complexity of B is much smaller than that of X.
• Apply the theory to convolutional neural networks and scattering transforms, validating with experiments on MNIST and CIFAR-10.

Experimental results

Research questions

• RQ1 How does invariance to input transformations affect the generalization error of a learning algorithm?
• RQ2 Can the generalization error of an invariant classifier be bounded in terms of the complexity of the base space rather than the full input space?
• RQ3 What geometric conditions on the base space and transformation group ensure a significant reduction in generalization error?
• RQ4 To what extent can invariance reduce generalization error, and is this reduction quantitatively bounded?
• RQ5 How does the proposed theory explain the empirical success of CNNs and other invariant architectures in reducing generalization error?

Key findings

• The generalization error of an invariant classifier is proportional to the complexity of the base space, not the full input space.
• For a transformation group of size T, the generalization error of an invariant classifier can be reduced by up to a factor of √T compared to a non-invariant classifier.
• Sufficient geometric conditions exist under which the base space complexity is much smaller than the input space complexity, enabling large GE reductions.
• The theory provides a principled explanation for the improved generalization of CNNs and scattering transforms, which are inherently invariant to certain transformations.
• Empirical results on MNIST and CIFAR-10 validate that invariant learning leads to lower generalization error, consistent with theoretical predictions.
• The framework applies broadly to any classifier with invariance to a known transformation group, including deep learning models.