[Paper Review] Dimensionality compression and expansion in Deep Neural Networks

Stefano Recanatesi, Matthew Farrell+4 more|arXiv (Cornell University)|Jun 2, 2019

Generative Adversarial Networks and Image Synthesis47 references42 citations

TL;DR

The paper uses intrinsic dimensionality estimation to show that deep networks create low-dimensional representation manifolds in two phases: expansion in early layers and compression in later layers, with SGD regularization balancing these forces to aid generalization.

ABSTRACT

Datasets such as images, text, or movies are embedded in high-dimensional spaces. However, in important cases such as images of objects, the statistical structure in the data constrains samples to a manifold of dramatically lower dimensionality. Learning to identify and extract task-relevant variables from this embedded manifold is crucial when dealing with high-dimensional problems. We find that neural networks are often very effective at solving this task and investigate why. To this end, we apply state-of-the-art techniques for intrinsic dimensionality estimation to show that neural networks learn low-dimensional manifolds in two phases: first, dimensionality expansion driven by feature generation in initial layers, and second, dimensionality compression driven by the selection of task-relevant features in later layers. We model noise generated by Stochastic Gradient Descent and show how this noise balances the dimensionality of neural representations by inducing an effective regularization term in the loss. We highlight the important relationship between low-dimensional compressed representations and generalization properties of the network. Our work contributes by shedding light on the success of deep neural networks in disentangling data in high-dimensional space while achieving good generalization. Furthermore, it invites new learning strategies focused on optimizing measurable geometric properties of learned representations, beginning with their intrinsic dimensionality.

Motivation & Objective

• Investigate why deep neural networks solve high-dimensional classification tasks effectively.
• Quantify the intrinsic dimensionality of data and learned representations across network layers.
• Understand how training dynamics shape dimensionality and generalization.

Proposed method

• Apply state-of-the-art intrinsic dimensionality estimation techniques to measure local and global manifold dimensionality.
• Train two networks (DeepNet on Fashion-MNIST and ResNet on CIFAR-10/ CIFAR-100) and analyze layer-wise dimensionality.
• Compare before/after training dimensionality to identify expansion and compression phases.
• Model SGD as inducing an effective regularization term that penalizes representation dimensionality.
• Analyze the role of layer types (convolutional, fully connected) and nonlinearities (ReLU) on dimensionality.
• Use linear and nonlinear analyses to interpret how dimensionality relates to task demands and feature selection.

Experimental results

Research questions

• RQ1 What is the intrinsic dimensionality of data manifolds and of learned representations in deep networks across layers?
• RQ2 Do neural networks exhibit distinct phases of dimensionality expansion and compression during learning?
• RQ3 How does stochastic gradient descent influence the dimensionality of representations through an effective regularization?
• RQ4 How does dimensionality relate to generalization and task performance in deep architectures?

Key findings

• Representation manifolds in deep networks are very low-dimensional compared to layer size.
• Dimensionality expands in early layers and compresses in later layers during training.
• ReLU nonlinearities increase dimensionality, while the weight matrix preceding ReLU drives compression of representations.
• SGD introduces an effective regularization that compresses task-irrelevant directions and balances expansion to fit task demands.
• Wider networks do not necessarily increase learned manifold dimensionality, suggesting the dimension is governed by task demands and SGD rather than network size.
• Dimensionality reduction correlates with better generalization and can inform architecture sizing and regularization strategies.