[Paper Review] Speeding-up Convolutional Neural Networks Using Fine-tuned CP-Decomposition

Vadim Lebedev, Yaroslav Ganin+3 more|arXiv (Cornell University)|Dec 19, 2014

Tensor decomposition and applications15 references331 citations

TL;DR

This paper proposes a two-stage method to accelerate convolutional neural networks (CNNs) by applying non-linear least squares (NLS)-based CP-decomposition to 4D convolutional kernel tensors, replacing each layer with a sequence of four small-kernel convolutions. The approach achieves up to 8.5× CPU speedup with only a 1% accuracy drop on a 36-class character classification network and 4× speedup on AlexNet with a 1% top-5 error increase, outperforming prior methods in speed-accuracy trade-off through fine-tuning.

ABSTRACT

We propose a simple two-step approach for speeding up convolution layers within large convolutional neural networks based on tensor decomposition and discriminative fine-tuning. Given a layer, we use non-linear least squares to compute a low-rank CP-decomposition of the 4D convolution kernel tensor into a sum of a small number of rank-one tensors. At the second step, this decomposition is used to replace the original convolutional layer with a sequence of four convolutional layers with small kernels. After such replacement, the entire network is fine-tuned on the training data using standard backpropagation process. We evaluate this approach on two CNNs and show that it is competitive with previous approaches, leading to higher obtained CPU speedups at the cost of lower accuracy drops for the smaller of the two networks. Thus, for the 36-class character classification CNN, our approach obtains a 8.5x CPU speedup of the whole network with only minor accuracy drop (1% from 91% to 90%). For the standard ImageNet architecture (AlexNet), the approach speeds up the second convolution layer by a factor of 4x at the cost of $1\%$ increase of the overall top-5 classification error.

Motivation & Objective

• To reduce the computational and memory cost of large convolutional neural networks (CNNs) for deployment on low-end CPUs and mobile devices.
• To address the challenge of maintaining high accuracy while significantly reducing the number of parameters in convolutional layers.
• To improve upon existing tensor decomposition methods for CNN acceleration by using a more accurate decomposition technique and discriminative fine-tuning.
• To demonstrate that low-rank CP-decomposition combined with global fine-tuning yields better speed-accuracy trade-offs than prior approaches.
• To validate the effectiveness of the method on real-world CNN architectures, including a custom character classifier and ImageNet-scale AlexNet.

Proposed method

• Apply non-linear least squares (NLS) to compute a low-rank CP-decomposition of the 4D convolution kernel tensor into a sum of rank-one tensors.
• Replace each original convolutional layer with a sequence of four depthwise-like convolutional layers, each using a small 2D kernel derived from the CP components.
• Use standard backpropagation to fine-tune the entire network after layer replacement to recover and potentially improve accuracy.
• Leverage the tensor algebra structure of CNN kernels to enable efficient decomposition and layer replacement without custom layer implementations.
• Use the rank of the CP-decomposition as a hyperparameter to control the trade-off between speedup and accuracy loss.
• Compare the performance of NLS-based decomposition with greedy algorithms and random initialization to demonstrate the importance of high-quality initialization.

Experimental results

Research questions

• RQ1 Can NLS-based CP-decomposition of 4D convolutional kernels lead to better speed-accuracy trade-offs than greedy decomposition methods in CNNs?
• RQ2 Does discriminative fine-tuning after CP-decomposition replacement significantly improve accuracy compared to non-fine-tuned approximations?
• RQ3 Can the proposed method achieve higher speedups than prior tensor decomposition-based approaches while maintaining or improving accuracy?
• RQ4 How does the quality of the initial CP-decomposition (e.g., NLS vs. greedy) affect the final performance after fine-tuning?
• RQ5 To what extent does the method reduce parameter count and memory footprint in CNNs, especially in layers with spatially-varying kernels?

Key findings

• The NLS-based CP-decomposition method achieved an 8.5× CPU speedup on the 36-class character classification CNN with only a 1% accuracy drop (from 91% to 90%).
• On the AlexNet architecture, the second convolutional layer was accelerated by 4× with a 1% increase in top-5 classification error, demonstrating strong real-world applicability.
• NLS-based decomposition consistently outperformed greedy decomposition, reducing accuracy drop by up to 20 percentage points in some cases (e.g., 24.15% drop with greedy vs. 3.21% with NLS on AlexNet at R=300).
• Fine-tuning significantly improved performance, especially when combined with high-quality initialization via NLS, confirming that better initial approximations lead to better convergence.
• The method reduced the number of parameters in the approximated layers by several times, enabling more compact models suitable for memory-constrained environments.
• The approach demonstrated a regularizing effect, with higher-rank decompositions sometimes improving generalization, and showed potential for use in layers with spatially-varying kernels.