[Paper Review] Information-Theoretic Local Minima Characterization and Regularization

Zhiwei Jia, Hao Su|arXiv (Cornell University)|Nov 19, 2019

Advanced Neural Network Applications54 references4 citations

TL;DR

This paper proposes an information-theoretic metric based on the determinant of Fisher information to characterize and regularize local minima in deep neural networks. By deriving a tractable approximation and a practical regularizer, the method improves generalization across CIFAR-10, CIFAR-100, and ImageNet with consistent performance gains, while providing a theoretical generalization bound grounded in PAC-Bayes.

ABSTRACT

Recent advances in deep learning theory have evoked the study of generalizability across different local minima of deep neural networks (DNNs). While current work focused on either discovering properties of good local minima or developing regularization techniques to induce good local minima, no approach exists that can tackle both problems. We achieve these two goals successfully in a unified manner. Specifically, based on the observed Fisher information we propose a metric both strongly indicative of generalizability of local minima and effectively applied as a practical regularizer. We provide theoretical analysis including a generalization bound and empirically demonstrate the success of our approach in both capturing and improving the generalizability of DNNs. Experiments are performed on CIFAR-10, CIFAR-100 and ImageNet for various network architectures.

Motivation & Objective

• To unify the characterization and regularization of generalizable local minima in deep neural networks.
• To address the gap in existing work that treats generalizability characterization and regularization as separate problems.
• To develop a metric grounded in Fisher information that is both theoretically sound and empirically effective.
• To derive a practical regularizer from the metric that improves generalization without sacrificing training efficiency.
• To demonstrate consistent generalization gains across diverse architectures and datasets.

Proposed method

• The paper introduces a metric based on the determinant of Fisher information estimated from training data, which correlates strongly with generalization performance.
• It provides a PAC-Bayesian generalization bound to theoretically justify the metric’s effectiveness.
• An efficient, tractable approximation of the metric is proposed for practical comparison across local minima.
• A novel regularizer is derived from the metric, integrated into optimization via a modified update rule in Algorithm 1.
• The regularizer is applied selectively during later training stages to avoid numerical instability in early optimization.
• The method is evaluated using SGD with momentum on ResNet, Wide-ResNet, and DenseNet across CIFAR-10, CIFAR-100, and ImageNet.

Experimental results

Research questions

• RQ1 What properties of local minima in deep neural networks correlate with better generalization?
• RQ2 Can a single, information-theoretic metric simultaneously characterize and guide optimization toward better generalizable minima?
• RQ3 How can the Fisher information determinant be approximated efficiently for use in modern deep learning training?
• RQ4 Does a regularizer derived from this metric consistently improve generalization across diverse architectures and datasets?
• RQ5 Can the proposed method outperform existing regularization and optimization techniques in terms of test accuracy and robustness?

Key findings

• The proposed metric, based on the determinant of Fisher information, strongly correlates with generalization performance across different local minima.
• On CIFAR-10 and CIFAR-100, the method achieves consistent generalization improvements across ResNet-20, Wide-ResNet-28-2, and DenseNet-k12, with test error reductions observed in multiple runs.
• For ImageNet, the method improves generalization performance across all evaluated architectures, with results reported in mean ± standard deviation across 3 runs.
• The regularizer increases training time by only 1.5× per mini-batch, demonstrating computational efficiency.
• Ablation studies show the regularizer is most effective in the mid-to-late training stages, and applying it only after learning rate drops improves training speed significantly.
• Table 3 confirms that models trained with the regularizer achieve lower metric values (higher flatness), aligning with improved generalization.