[Paper Review] Ultimate tensorization: compressing convolutional and FC layers alike

Timur Garipov, Dmitry Podoprikhin+2 more|arXiv (Cornell University)|Nov 10, 2016

Tensor decomposition and applications19 references102 citations

TL;DR

The paper extends Tensor Train tensorization to convolutional layers by reshaping kernels into higher-order tensors, achieving large network compression (up to 80x) with minimal accuracy loss, and combines this with prior FC-layer compression.

ABSTRACT

Convolutional neural networks excel in image recognition tasks, but this comes at the cost of high computational and memory complexity. To tackle this problem, [1] developed a tensor factorization framework to compress fully-connected layers. In this paper, we focus on compressing convolutional layers. We show that while the direct application of the tensor framework [1] to the 4-dimensional kernel of convolution does compress the layer, we can do better. We reshape the convolutional kernel into a tensor of higher order and factorize it. We combine the proposed approach with the previous work to compress both convolutional and fully-connected layers of a network and achieve 80x network compression rate with 1.1% accuracy drop on the CIFAR-10 dataset.

Motivation & Objective

• Motivate reducing CNN storage and computation without large accuracy loss.
• Extend tensor factorization to convolutional kernels beyond naive 4D TT-apply.
• Show that higher-order tensor reshaping improves compression effectiveness.
• Demonstrate combined conv and fully-connected layer compression.
• Provide practical training methodology and reproducible results on CIFAR-10.

Proposed method

• Represent convolutional kernels as TT-format via reshaping the 4D kernel into a higher-order tensor.
• Apply matrix TT-format to the reshaped kernel to match FC tensorization benefits.
• Train TT-cores with SGD and momentum using automatic differentiation.
• Compare TT-conv against naive TT-decomposition applied to 4D kernels.
• Combine TT-conv with TT-fc from prior work to compress whole networks.
• Report compression and accuracy trade-offs on CIFAR-10 across architectures.

Experimental results

Research questions

• RQ1 Can TT-decomposition applied directly to convolutional kernels be outperformed by reshaping into a higher-order tensor for TT-convolution?
• RQ2 What compression rates and accuracy trade-offs are achievable when compressing both convolutional and fully-connected layers?
• RQ3 How does TT-conv compare to naive TT application on 4D kernels in CIFAR-10 settings?
• RQ4 Is it feasible to reach substantial network compression (e.g., 80x) with limited accuracy loss on convolution-heavy networks?
• RQ5 How does combining conv and FC compression affect overall performance and storage?

Key findings

• Direct application of TT to 4D conv kernels underperforms; reshaping to higher-order tensors yields better compression.
• Convolving networks with TT-conv achieves up to 4x compression with modest accuracy loss on CIFAR-10 when used alone.
• Combining TT-conv with TT-fc from prior work yields up to 82x network compression with around 1% accuracy drop.
• A conv-dominated network can be compressed by TT-conv to about 4x with roughly 2% accuracy loss (depending on TT-ranks).
• A conv-fc network compressed with TT-fc and TT-conv achieves substantial gains, e.g., 80x total compression with ~1% accuracy loss.
• The approach is validated on CIFAR-10 with reproducible setups and two baseline architectures.