[Paper Review] Deep Curvature Suite

Diego Granziol, Xingchen Wan+1 more|arXiv (Cornell University)|Dec 20, 2019

Stochastic Gradient Optimization Techniques49 references4 citations

TL;DR

The Deep Curvature Suite is a PyTorch-based open-source software package that enables efficient, accurate spectral analysis of Hessian and Generalized Gauss-Newton (GGN) matrices in deep neural networks using the Lanczos algorithm with random vectors. It provides high-fidelity curvature estimation—up to an order of magnitude faster than prior tools—while correcting widespread misconceptions about eigenvalue computation and validating spectral estimation via high-dimensional concentration inequalities.

ABSTRACT

We present MLRG Deep Curvature suite, a PyTorch-based, open-source package for analysis and visualisation of neural network curvature and loss landscape. Despite of providing rich information into properties of neural network and useful for a various designed tasks, curvature information is still not made sufficient use for various reasons, and our method aims to bridge this gap. We present a primer, including its main practical desiderata and common misconceptions, of extit{Lanczos algorithm}, the theoretical backbone of our package, and present a series of examples based on synthetic toy examples and realistic modern neural networks tested on CIFAR datasets, and show the superiority of our package against existing competing approaches for the similar purposes.

Motivation & Objective

• To address the lack of accessible, efficient, and correct tools for computing curvature information (Hessian and GGN spectra) in real-world deep learning models.
• To correct common misconceptions in the literature, such as the belief that the Lanczos algorithm computes eigenvalues from the top down.
• To provide a practical, scalable, and well-documented software suite that enables researchers and practitioners to analyze curvature, loss surfaces, and optimization dynamics in modern networks.
• To demonstrate the limitations of diagonal approximations for curvature by showing they can be arbitrarily poor on both synthetic and real datasets.
• To establish theoretical grounding for spectral estimation using high-dimensional concentration inequalities, proving that a single random vector suffices for accurate spectral approximation under certain conditions.

Proposed method

• Employs the Lanczos algorithm with random seed vectors to compute a moment-matched discrete approximation of the Hessian and GGN spectra, leveraging orthogonal iteration and Krylov subspace methods.
• Uses fast Hessian-vector products via automatic differentiation (PyTorch) to avoid explicit Hessian computation, enabling scalability to large models.
• Applies high-dimensional concentration inequalities to theoretically justify that a single random vector can yield accurate spectral estimates for specific matrix classes.
• Implements loss surface visualization by traversing eigenvectors of the Hessian and GGN, enabling analysis of loss and accuracy changes along extreme directions.
• Integrates with GPyTorch’s optimized Lanczos implementation to ensure numerical stability and performance, especially for large-scale models.
• Validates curvature approximations using synthetic matrices with known eigenvalue distributions and real-world models on CIFAR-10 and CIFAR-100.

Experimental results

Research questions

• RQ1 Can a single random vector provide a reliable spectral approximation of the Hessian or GGN in high-dimensional deep learning models?
• RQ2 How do common diagonal approximations of curvature matrices compare to full spectral estimates in terms of accuracy and representativeness?
• RQ3 To what extent do standard assumptions about loss surface geometry (e.g., spin-glass models, random matrix theory) hold for real deep neural networks?
• RQ4 Why do previous studies relying on diagonal approximations or limited eigen-decomposition fail to capture the true curvature landscape?
• RQ5 How do training and test loss surfaces differ along extreme curvature directions, and what does this imply about generalization and optimization?

Key findings

• The Lanczos algorithm with random vectors produces a moment-matched spectral approximation of the Hessian and GGN, with accuracy validated on both synthetic and real deep learning models.
• Diagonal approximations of the Hessian or GGN can be arbitrarily poor, as demonstrated on both synthetic matrices and real networks like VGG-16 and ResNet-18 on CIFAR-10/100.
• The smoothed spectral moments of the Hessian are biased due to kernel smoothing, with the bias increasing for higher-order moments and being positive for positive eigenvalues.
• Training and test loss surfaces are not simple shifts of each other; extremal eigenvectors cause significantly different loss changes on test vs. training sets, challenging prior assumptions.
• The Deep Curvature Suite computes full Hessian and GGN spectra in minutes on a single GPU for modern architectures like VGG-16 and ResNet-18, achieving up to an order of magnitude speedup over competing tools.
• Spectral estimation using a single random vector is theoretically justified via high-dimensional concentration inequalities, proving its validity for specific matrix classes.