[Paper Review] Can We Gain More from Orthogonality Regularizations in Training Deep CNNs?

Nitin Bansal, Xiaohan Chen+1 more|arXiv (Cornell University)|Oct 22, 2018

Advanced Neural Network Applications29 references157 citations

TL;DR

The paper introduces plug-and-play orthogonality regularizations (SO, DSO, MC, SRIP) for CNNs and shows SRIP consistently improves accuracy and convergence across ResNet, WideResNet, and ResNeXt on CIFAR, ImageNet, and SVHN.

ABSTRACT

This paper seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how can we enforce it in more effective and easy-to-use ways? We develop novel orthogonality regularizations on training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and restricted isometry property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on state-of-the-art models: ResNet, WideResNet, and ResNeXt, on several most popular computer vision datasets: CIFAR-10, CIFAR-100, SVHN and ImageNet. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and faster and more stable convergences. We have made our codes and pre-trained models publicly available: https://github.com/nbansal90/Can-we-Gain-More-from-Orthogonality.

Motivation & Objective

• Motivate and study how enforcing near-orthogonality in weight matrices can stabilize training and improve performance of deep CNNs.
• Propose several regularizers that enforce orthogonality in both square and rectangular weight matrices via different formulations.
• Assess the effectiveness of these regularizers on state-of-the-art CNN architectures across standard vision datasets.
• Provide practical guidance on how to integrate these plug-and-play regularizers without altering network architectures.

Proposed method

• Derive four regularizers: Soft Orthogonality (SO), Double Soft Orthogonality (DSO), Mutual Coherence (MC), and Spectral RIP (SRIP).
• SO minimizes ||W^T W - I||_F^2; DSO minimizes ||W^T W - I||_F^2 + ||W W^T - I||_F^2.
• MC minimizes ||W^T W - I||_∞ to suppress column coherence.
• SRIP minimizes the spectral norm of (W^T W - I) to align singular values with one; uses a power-iteration-based approximation for efficiency.
• Adopts a scheme-change training strategy, reducing regularization strength during training to balance early stabilization and late-stage flexibility.
• Evaluates on ResNet, WideResNet, and ResNeXt with CIFAR-10/100, ImageNet, and SVHN; compares SRIP against SR, OMDSM, and Jacobian-based regularization.

Experimental results

Research questions

• RQ1 Can orthogonality regularizations improve training stability and final accuracy of deep CNNs?
• RQ2 Which regularization formulation (SO, DSO, MC, SRIP) provides the best trade-off between performance and computational cost?
• RQ3 Does SRIP outperform existing spectral or hard-constraint orthogonality methods on large-scale datasets like ImageNet?
• RQ4 Is a scheme-change (dynamic regularization strength) beneficial for training with orthogonality regularizations?

Key findings

• SRIP consistently yields the best performance across CIFAR-10/100, ImageNet, and SVHN among the proposed regularizers.
• On CIFAR-10/100, SRIP achieves up to 2.31% top-1 accuracy improvement for Wide ResNet-28-10 and 0.22–0.56% for other models depending on dataset.
• SRIP outperforms spectral regularization, hard orthogonality methods, and Jacobian-norm approaches in reported comparisons.
• SO is a surprisingly robust baseline, while DSO often underperforms.
• Regularizers accelerate early training and smooth convergence without architectural changes; scheme-change (reducing regularization over time) enhances final accuracy.