[Paper Review] Convolutional neural networks with low-rank regularization

Cheng Tai, Tong Xiao+3 more|arXiv (Cornell University)|Jan 1, 2016

Advanced Neural Network ApplicationsComputer Science279 citations

TL;DR

This paper proposes a novel low-rank tensor decomposition algorithm for pruning convolutional neural networks (CNNs), enabling exact global optimization of kernel redundancy. The method trains low-rank constrained CNNs from scratch, achieving up to 50% faster inference (e.g., VGG-16) with improved or comparable accuracy—e.g., 91.31% top-1 accuracy on CIFAR-10 without data augmentation, surpassing state-of-the-art.

ABSTRACT

Abstract: Large CNNs have delivered impressive performance in various computer vision applications. But the storage and computation requirements make it problematic for deploying these models on mobile devices. Recently, tensor decompositions have been used for speeding up CNNs. In this paper, we further develop the tensor decomposition technique. We propose a new algorithm for computing the low-rank tensor decomposition for removing the redundancy in the convolution kernels. The algorithm finds the exact global optimizer of the decomposition and is more effective than iterative methods. Based on the decomposition, we further propose a new method for training low-rank constrained CNNs from scratch. Interestingly, while achieving a significant speedup, sometimes the low-rank constrained CNNs delivers significantly better performance than their non-constrained counterparts. On the CIFAR-10 dataset, the proposed low-rank NIN model achieves $91.31\%$ accuracy (without data augmentation), which also improves upon state-of-the-art result. We evaluated the proposed method on CIFAR-10 and ILSVRC12 datasets for a variety of modern CNNs, including AlexNet, NIN, VGG and GoogleNet with success. For example, the forward time of VGG-16 is reduced by half while the performance is still comparable. Empirical success suggests that low-rank tensor decompositions can be a very useful tool for speeding up large CNNs.

Motivation & Objective

• To reduce computational and storage costs of large CNNs for mobile deployment.
• To address the inefficiency of iterative methods in computing low-rank tensor decompositions.
• To develop a method for training low-rank CNNs from scratch with improved or comparable performance.
• To explore whether low-rank constraints can enhance generalization beyond mere speedup.

Proposed method

• Proposes a new algorithm for computing low-rank tensor decomposition that finds the exact global optimizer, avoiding suboptimal solutions from iterative methods.
• Applies tensor decomposition to convolution kernels to remove redundancy and reduce parameter count.
• Introduces a training procedure for low-rank constrained CNNs starting from random initialization.
• Uses Tucker decomposition or similar low-rank factorization to represent 3D/4D convolutional kernels as smaller, low-rank components.
• Optimizes the low-rank model end-to-end during training while maintaining structural constraints.
• Employs exact optimization to ensure the decomposition is globally optimal, improving stability and performance.

Experimental results

Research questions

• RQ1 Can exact low-rank tensor decomposition outperform iterative methods in kernel compression?
• RQ2 Does training low-rank CNNs from scratch yield better or comparable accuracy compared to full-rank models?
• RQ3 Can low-rank constraints improve generalization, leading to higher accuracy than non-constrained models?
• RQ4 To what extent can inference speed be improved without sacrificing performance?

Key findings

• The proposed low-rank NIN model achieves 91.31% top-1 accuracy on CIFAR-10 without data augmentation, exceeding state-of-the-art results.
• The forward inference time of VGG-16 is reduced by 50% using the low-rank decomposition while maintaining comparable performance.
• The method successfully accelerates multiple architectures, including AlexNet, NIN, VGG, and GoogleNet on both CIFAR-10 and ILSVRC12.
• Surprisingly, low-rank constrained models sometimes outperform their full-rank counterparts, suggesting potential regularization benefits.
• The exact optimization algorithm for low-rank decomposition is more effective than iterative alternatives, yielding better convergence and performance.