[Paper Review] Verifying the Union of Manifolds Hypothesis for Image Data

Bradley C. A. Brown, Anthony L. Caterini+3 more|arXiv (Cornell University)|Jul 6, 2022

Generative Adversarial Networks and Image Synthesis6 citations

TL;DR

This paper proposes the union of manifolds hypothesis as a more accurate characterization of image data structure than the traditional manifold hypothesis, demonstrating that image data lies on a disjoint union of manifolds with varying intrinsic dimensions. Using pushforward deep generative models and intrinsic dimension estimation, the authors empirically verify disconnected supports across classes in datasets like MNIST and CIFAR-100, showing that higher-intrinsic-dimension classes are harder to classify and that reweighting cross-entropy loss by intrinsic dimension improves classification accuracy.

ABSTRACT

Deep learning has had tremendous success at learning low-dimensional representations of high-dimensional data. This success would be impossible if there was no hidden low-dimensional structure in data of interest; this existence is posited by the manifold hypothesis, which states that the data lies on an unknown manifold of low intrinsic dimension. In this paper, we argue that this hypothesis does not properly capture the low-dimensional structure typically present in image data. Assuming that data lies on a single manifold implies intrinsic dimension is identical across the entire data space, and does not allow for subregions of this space to have a different number of factors of variation. To address this deficiency, we consider the union of manifolds hypothesis, which states that data lies on a disjoint union of manifolds of varying intrinsic dimensions. We empirically verify this hypothesis on commonly-used image datasets, finding that indeed, observed data lies on a disconnected set and that intrinsic dimension is not constant. We also provide insights into the implications of the union of manifolds hypothesis in deep learning, both supervised and unsupervised, showing that designing models with an inductive bias for this structure improves performance across classification and generative modelling tasks. Our code is available at https://github.com/layer6ai-labs/UoMH.

Motivation & Objective

• To challenge the traditional manifold hypothesis, which assumes a single low-dimensional manifold with constant intrinsic dimension across all data.
• To investigate whether image data actually lies on a disconnected union of manifolds with varying intrinsic dimensions.
• To empirically verify the union of manifolds hypothesis using pushforward deep generative models and intrinsic dimension estimation on standard image datasets.
• To explore the implications of the hypothesis for deep learning, particularly in supervised and unsupervised learning tasks.
• To demonstrate that modeling the union of manifolds improves performance in classification and generative modeling.

Proposed method

• Using pushforward deep generative models (DGMs) to test whether disconnected data supports can be modeled, leveraging the theoretical result that DGMs cannot model disconnected supports.
• Training class-specific DGMs on each class of image data and comparing performance to a single global DGM under the same computational budget to infer support disconnectedness.
• Estimating intrinsic dimension per class using the k-nearest neighbors (k-NN) method with the method of Pope et al. (2021), computing local intrinsic dimensionality estimates.
• Applying a reweighted cross-entropy loss in supervised learning, where class weights are proportional to estimated intrinsic dimension to improve classification accuracy on high-dimensional classes.
• Training and evaluating disconnected DGMs (disconnected pushforward models) on image datasets to assess performance in generative modeling.
• Using standard deep learning architectures (VGG-19, ResNet-18, ResNet-34) with no data augmentation to ensure intrinsic dimension estimates remain consistent and unconfounded.

Experimental results

Research questions

• RQ1 Is the support of image data truly disconnected, such that different classes form distinct connected components?
• RQ2 Do different classes in image datasets exhibit significantly varying intrinsic dimensions?
• RQ3 Can reweighting the cross-entropy loss based on intrinsic dimension improve classification performance?
• RQ4 Can disconnected deep generative models outperform standard DGMs in generative modeling tasks?
• RQ5 Does the union of manifolds hypothesis provide a better inductive bias for both supervised and unsupervised deep learning tasks?

Key findings

• The support of image data is empirically disconnected, as evidenced by superior performance of class-specific DGMs compared to a single global DGM under the same computational budget.
• Intrinsic dimension varies significantly across classes in datasets like MNIST and CIFAR-100, with some classes (e.g., '8's and '1's in MNIST) exhibiting higher intrinsic dimension than others.
• Classes with higher intrinsic dimension are more difficult to classify, as shown by lower baseline accuracy on such classes.
• Reweighting the cross-entropy loss by intrinsic dimension leads to improved classification accuracy, particularly on high-dimensional classes, demonstrating the utility of the hypothesis in supervised learning.
• Disconnected DGMs achieve competitive performance compared to standard DGMs across multiple datasets and architectures, validating their potential as a new modeling paradigm.
• The union of manifolds hypothesis provides a more accurate inductive bias than the single manifold assumption, improving performance in both classification and generative modeling tasks.