[Paper Review] Training CNNs with Low-Rank Filters for Efficient Image Classification

Yani Ioannou, Duncan Robertson+3 more|arXiv (Cornell University)|Nov 20, 2015

Advanced Neural Network ApplicationsComputer Science17 references40 citations

TL;DR

This paper proposes training CNNs from scratch using low-rank, composite convolutional filters—specifically, combining horizontal (1×k) and vertical (k×1) filters as basis functions—enabling significant computational and parameter efficiency. By introducing a novel weight initialization scheme for mixed-shape filter groups, the method achieves comparable or higher accuracy than standard CNNs with up to 55% fewer parameters and 46% less compute, as demonstrated on CIFAR-10, ILSVRC, and MIT Places datasets.

ABSTRACT

We propose a new method for creating computationally efficient convolutional neural networks (CNNs) by using low-rank representations of convolutional filters. Rather than approximating filters in previously-trained networks with more efficient versions, we learn a set of small basis filters from scratch; during training, the network learns to combine these basis filters into more complex filters that are discriminative for image classification. To train such networks, a novel weight initialization scheme is used. This allows effective initialization of connection weights in convolutional layers composed of groups of differently-shaped filters. We validate our approach by applying it to several existing CNN architectures and training these networks from scratch using the CIFAR, ILSVRC and MIT Places datasets. Our results show similar or higher accuracy than conventional CNNs with much less compute. Applying our method to an improved version of VGG-11 network using global max-pooling, we achieve comparable validation accuracy using 41% less compute and only 24% of the original VGG-11 model parameters; another variant of our method gives a 1 percentage point increase in accuracy over our improved VGG-11 model, giving a top-5 center-crop validation accuracy of 89.7% while reducing computation by 16% relative to the original VGG-11 model. Applying our method to the GoogLeNet architecture for ILSVRC, we achieved comparable accuracy with 26% less compute and 41% fewer model parameters. Applying our method to a near state-of-the-art network for CIFAR, we achieved comparable accuracy with 46% less compute and 55% fewer parameters.

Motivation & Objective

• Address the growing computational and model size costs of state-of-the-art CNNs for deployment on low-power devices.
• Reduce computational complexity in convolutional layers without sacrificing classification accuracy.
• Explore whether learning low-rank filters from scratch can improve efficiency and generalization beyond approximating pre-trained models.
• Develop a novel weight initialization method tailored for composite convolutional layers with differently shaped filters (e.g., 1×k, k×1, k×k).

Proposed method

• Represent convolutional filters as linear combinations of small, low-rank basis filters (e.g., 1×k and k×1) instead of full k×k kernels.
• Train networks from scratch using a novel weight initialization scheme that accounts for the structural differences in mixed-shape filter groups.
• Use a basis space of rectangular and square filters to enable efficient, learnable representations of complex spatial patterns.
• Apply the method to existing architectures (VGG-11, GoogLeNet, Network-in-Network) by replacing standard filters with low-rank equivalents in key layers.
• Optimize for both inference efficiency and generalization by restricting filter complexity through basis decomposition.
• Use global max-pooling and architectural modifications to further reduce model size and computation.

Experimental results

Research questions

• RQ1 Can training CNNs from scratch using low-rank, composite filters achieve comparable or better accuracy than standard CNNs with significantly reduced computational cost?
• RQ2 Does learning basis filters from scratch, rather than approximating pre-trained models, lead to improved generalization and efficiency?
• RQ3 How effective is the proposed weight initialization scheme in training networks with heterogeneous filter shapes in the same layer?
• RQ4 To what extent can low-rank filter decompositions (e.g., 1×k and k×1) represent the discriminative patterns of full k×k filters in image classification tasks?
• RQ5 Can the method be generalized across diverse datasets (CIFAR-10, ILSVRC, MIT Places) and architectures (VGG, GoogLeNet, NiN) with consistent gains in efficiency?

Key findings

• Applying the method to an improved VGG-11 with global max-pooling achieved 89.7% top-5 center-crop accuracy—matching or exceeding the original VGG-11—while reducing computation by 41% and model parameters by 76%.
• A variant of the method increased accuracy by 1 percentage point over the improved VGG-11, achieving 89.7% top-5 accuracy with 16% less computation.
• On GoogLeNet, the low-rank version achieved comparable ILSVRC accuracy (88.0% top-5) with 26% less compute and 41% fewer parameters.
• For a near state-of-the-art CIFAR-10 model (NiN), the low-rank variant achieved 91.8% accuracy with 46% less computation and 55% fewer parameters.
• The method outperformed prior approaches in efficiency, with no other network achieving comparable accuracy within an order of magnitude of compute reduction.
• The proposed weight initialization scheme was critical for successful training, enabling stable convergence in networks with mixed-shape filter groups.