[Paper Review] On Characterizing the Capacity of Neural Networks using Algebraic Topology

William H. Guss, Ruslan Salakhutdinov|arXiv (Cornell University)|Feb 13, 2018

Topological and Geometric Data Analysis21 references60 citations

TL;DR

The paper uses persistent homology to quantify data complexity, defines the topological capacity of neural networks, and proposes a topology-guided architecture selection approach with empirical phase transitions observed across architectures.

ABSTRACT

The learnability of different neural architectures can be characterized directly by computable measures of data complexity. In this paper, we reframe the problem of architecture selection as understanding how data determines the most expressive and generalizable architectures suited to that data, beyond inductive bias. After suggesting algebraic topology as a measure for data complexity, we show that the power of a network to express the topological complexity of a dataset in its decision region is a strictly limiting factor in its ability to generalize. We then provide the first empirical characterization of the topological capacity of neural networks. Our empirical analysis shows that at every level of dataset complexity, neural networks exhibit topological phase transitions. This observation allowed us to connect existing theory to empirically driven conjectures on the choice of architectures for fully-connected neural networks.

Motivation & Objective

• Formalize geometric complexity of datasets via algebraic topology to inform architecture selection.
• Introduce topological capacity as a measure of a network’s ability to express data topology and generalize.
• Empirically characterize how data homology affects learning across neural architectures.
• Propose topological architecture selection by linking persistent homology of data to minimal expressive networks.
• Demonstrate practical applicability on OpenML and real datasets.

Proposed method

• Adopt persistent homology to quantify the topological complexity of data and decision regions.
• Define support homology H_S(f) and relate it to the positive decision region of a classifier f.
• Establish a homological generalization principle: if a architecture cannot realize a given homology, it will misclassify data subsets (Theorem 3.1).
• Conduct empirical experiments with fully connected networks (ReLU) across depths and first-layer widths to study homological expressivity.
• Compute and analyze E_H^p(f, D) = min{β_p(f)/β_p(D), 1} to assess how well an architecture expresses data homology.
• Use persistence diagrams/filtrations to relate data topology to minimal expressive architectures and estimate h_phase via derived bounds (Eq. 3.1, Eq. 4.1).
• Apply topological architecture selection in practice on OpenML datasets by deriving a lower bound on h_phase from data persistence features.

Experimental results

Research questions

• RQ1 How does the homological (topological) complexity of data constrain the expressivity and generalization of neural networks?
• RQ2 Can persistent homology guide the selection of neural architectures by predicting minimal sufficient capacity (h_phase) for a given dataset?
• RQ3 What are the observed neural topological phase transitions as data homology increases?
• RQ4 To what extent can topology-based predictions transfer to real datasets beyond synthetic constructions (e.g., OpenML, CIFAR-10, UCI)?

Key findings

• Neural networks exhibit topological phase transitions in expressivity as data topology becomes more complex.
• There exists a phase threshold h_phase below which networks fail to converge on more topologically complex data; above it, convergence behavior changes.
• Empirical estimates of h_phase derived from persistent homology features provide a strong starting point for architecture selection (near-zero error at predicted phase in several cases).
• Higher-order homology (β1 and above) is harder to learn with shallow networks, and deeper networks push learnability to higher homology at the cost of increased complexity.
• CIFAR-10 and several UCI/OpenML datasets show non-trivial persistent homology, validating the practicality of topological measures on real data.
• The framework enables data-first architecture search by constraining search with topological capacity estimates derived from persistence diagrams.