[Paper Review] Neural Persistence: A Complexity Measure for Deep Neural Networks Using Algebraic Topology

Bastian Rieck, Matteo Togninalli+5 more|arXiv (Cornell University)|Feb 22, 2019

Topological and Geometric Data AnalysisComputer Science34 citations

TL;DR

This paper introduces neural persistence, a novel complexity measure for deep neural networks derived from algebraic topology applied to weighted stratified graphs. It quantifies architectural complexity, reflects established training practices like dropout and batch normalization, and enables a stopping criterion that reduces training time while maintaining validation accuracy.

ABSTRACT

While many approaches to make neural networks more fathomable have been proposed, they are restricted to interrogating the network with input data. Measures for characterizing and monitoring structural properties, however, have not been developed. In this work, we propose neural persistence, a complexity measure for neural network architectures based on topological data analysis on weighted stratified graphs. To demonstrate the usefulness of our approach, we show that neural persistence reflects best practices developed in the deep learning community such as dropout and batch normalization. Moreover, we derive a neural persistence-based stopping criterion that shortens the training process while achieving comparable accuracies as early stopping based on validation loss.

Motivation & Objective

• To develop a structural complexity measure for deep neural networks independent of input data.
• To identify and quantify architectural properties that correlate with best practices in deep learning.
• To create a training stopping criterion based on neural persistence that reduces training time without sacrificing model accuracy.

Proposed method

• Represent neural networks as weighted stratified graphs, where layers are strata and weights are learned parameters.
• Apply persistent homology from topological data analysis to compute neural persistence as a measure of structural complexity.
• Use the persistence diagram's features—specifically, the lifespans of topological features—to define the neural persistence metric.
• Train models and monitor neural persistence during optimization to detect convergence.
• Derive a stopping rule based on when neural persistence stabilizes, indicating sufficient architectural learning.
• Validate the method against standard early stopping using validation loss as a benchmark.

Experimental results

Research questions

• RQ1 Can neural persistence effectively capture structural complexity in deep neural network architectures?
• RQ2 Does neural persistence correlate with established training techniques such as dropout and batch normalization?
• RQ3 Can a stopping criterion based on neural persistence reduce training time while maintaining model performance?

Key findings

• Neural persistence successfully reflects known best practices in deep learning, such as the use of dropout and batch normalization, indicating its sensitivity to meaningful architectural design choices.
• The proposed neural persistence-based stopping criterion reduces training time compared to validation loss-based early stopping while achieving comparable test accuracy.
• Neural persistence stabilizes earlier in training than validation loss, enabling earlier convergence detection.
• The method provides a data-independent measure of network complexity, offering insights into architecture behavior beyond input-dependent analysis.