[Paper Review] Automatic Rank Selection for High-Speed Convolutional Neural Network

Hyeji Kim, Chong‐Min Kyung|arXiv (Cornell University)|Jun 28, 2018

Advanced Neural Network Applications12 references4 citations

TL;DR

This paper proposes a model-wise rank selection algorithm for accelerating convolutional neural networks via low-rank decomposition, using a linearly approximated accuracy function and space constraint parameters to efficiently search for optimal rank sets. The method achieves 4.03× speedup on VGG-16 with 90.0% top-5 accuracy, outperforming state-of-the-art methods in inference and training time while maintaining high accuracy.

ABSTRACT

Low-rank decomposition plays a central role in accelerating convolutional neural network (CNN), and the rank of decomposed kernel-tensor is a key parameter that determines the complexity and accuracy of a neural network. In this paper, we define rank selection as a combinatorial optimization problem and propose a methodology to minimize network complexity while maintaining the desired accuracy. Combinatorial optimization is not feasible due to search space limitations. To restrict the search space and obtain the optimal rank, we define the space constraint parameters with a boundary condition. We also propose a linearly-approximated accuracy function to predict the fine-tuned accuracy of the optimized CNN model during the cost reduction. Experimental results on AlexNet and VGG-16 show that the proposed rank selection algorithm satisfies the accuracy constraint. Our method combined with truncated-SVD outperforms state-of-the-art methods in terms of inference and training time at almost the same accuracy.

Motivation & Objective

• To address the inefficiency of layer-wise rank selection in CNN acceleration, which often gets stuck in local optima.
• To reduce the computational search space for optimal rank selection across all layers while maintaining target accuracy.
• To develop a predictive accuracy function that estimates fine-tuned performance without full fine-tuning.
• To enable efficient, high-speed deployment of CNNs on resource-constrained devices through optimal low-rank decomposition.

Proposed method

• Formulates rank selection as a combinatorial optimization problem constrained by accuracy and complexity.
• Introduces space constraint parameters with boundary conditions to limit the search space to feasible rank combinations.
• Employs a linearly-approximated accuracy function that correlates pre-fine-tuning accuracy with post-fine-tuning performance to guide optimization.
• Uses a model-wise search strategy that updates ranks of all layers simultaneously, avoiding local optima common in layer-wise greedy methods.
• Applies truncated SVD for low-rank decomposition and iteratively selects the best rank set based on accuracy and cost constraints.
• Employs a termination condition based on accuracy thresholds derived from fine-tuning experiments on reduced-cost models.

Experimental results

Research questions

• RQ1 Can a model-wise rank selection strategy outperform layer-wise greedy approaches in finding globally optimal rank configurations?
• RQ2 How accurately can pre-fine-tuning test accuracy predict final fine-tuned accuracy in low-rank CNNs?
• RQ3 What constraints are necessary to reduce the search space for rank selection without sacrificing optimality?
• RQ4 Can the proposed method achieve faster inference and training times than state-of-the-art methods while maintaining high accuracy?

Key findings

• The proposed method achieves a 4.03× speedup on VGG-16 with 90.0% top-5 accuracy, outperforming prior methods in both inference and training time.
• The model-wise search strategy successfully avoids local optima, leading to better overall rank configurations than layer-wise approaches.
• The linear approximation of accuracy before and after fine-tuning enables accurate prediction of final performance without full fine-tuning.
• The method reduces the number of kernel layers from 37 to 29 in VGG-16 by optimizing rank selection, contributing to faster inference.
• On GPU, the proposed model achieves 8.39ms forward pass time (2.82× faster than baseline), and 12.11ms backward pass (1.49× faster).
• The method maintains near-original accuracy (89.9% top-5) while significantly reducing computational cost, demonstrating strong trade-off efficiency.