[Paper Review] Tensorizing Neural Networks

Alexander Novikov, Dmitry Podoprikhin+2 more|arXiv (Cornell University)|Sep 22, 2015

Tensor decomposition and applications25 references498 citations

TL;DR

This paper proposes Tensorizing Neural Networks by representing fully-connected layer weight matrices using the Tensor Train (TT) format, reducing parameters by up to 200,000× while preserving model accuracy. The method enables significantly smaller models with faster inference, achieving up to 7× compression of the entire network and allowing much wider, more expressive layers without increasing memory or computation beyond feasible limits.

ABSTRACT

Deep neural networks currently demonstrate state-of-the-art performance in several domains. At the same time, models of this class are very demanding in terms of computational resources. In particular, a large amount of memory is required by commonly used fully-connected layers, making it hard to use the models on low-end devices and stopping the further increase of the model size. In this paper we convert the dense weight matrices of the fully-connected layers to the Tensor Train format such that the number of parameters is reduced by a huge factor and at the same time the expressive power of the layer is preserved. In particular, for the Very Deep VGG networks we report the compression factor of the dense weight matrix of a fully-connected layer up to 200000 times leading to the compression factor of the whole network up to 7 times.

Motivation & Objective

• To reduce the memory and computational cost of fully-connected layers in deep neural networks, which are major bottlenecks in model deployment.
• To preserve model accuracy while drastically compressing the number of parameters in dense weight matrices.
• To enable the use of much wider and more expressive fully-connected layers by replacing standard dense matrices with low-parameter TT-format decompositions.
• To maintain compatibility with standard back-propagation training algorithms through differentiable TT operations.
• To demonstrate that TT-decomposed networks can outperform standard non-convolutional networks on benchmark datasets like CIFAR-10.

Proposed method

• Represent the dense weight matrix of a fully-connected layer as a multi-dimensional tensor and apply the Tensor Train (TT) decomposition to factorize it into a sequence of small core tensors.
• Use the TT-format to parameterize the layer such that the number of parameters scales linearly with the input and output dimensions, rather than quadratically.
• Derive differentiable update rules for back-propagation using the properties of the TT decomposition, ensuring compatibility with standard stochastic gradient descent.
• Train the TT-layer end-to-end using standard optimization techniques, with TT-ranks constrained to control model capacity and compression.
• Apply the TT-layer to fully-connected layers in VGG-style networks, replacing standard dense layers while keeping convolutional layers unchanged.
• Use the TT-Toolbox and MatConvNet extensions to implement and evaluate the TT-layer efficiently in both CPU and GPU environments.

Experimental results

Research questions

• RQ1 Can the Tensor Train decomposition effectively compress the weight matrices of fully-connected layers without significant accuracy loss?
• RQ2 To what extent can TT-decomposition reduce the number of parameters in a fully-connected layer while maintaining model performance?
• RQ3 Does using TT-layers enable the training of much wider and more expressive fully-connected layers than previously feasible?
• RQ4 How does the inference speed and memory usage of TT-layers compare to standard fully-connected layers?
• RQ5 Can TT-decomposed networks achieve state-of-the-art performance on non-convolutional benchmarks like CIFAR-10?

Key findings

• The TT-layer compressed the largest fully-connected layer in a VGG network by a factor of 194,622 (from 102,981,504 to 528 parameters), with only a 0.3% increase in top-5 error.
• The overall network compression factor reached up to 7.4× when both the largest and second-largest fully-connected layers were TT-compressed.
• The TT-layer achieved a 13× speedup in inference time on CPU (1.2ms vs. 16.1ms per image) and a 6.7× speedup on GPU (1.9ms vs. 12.9ms).
• Memory usage for a single forward pass dropped from 392 MB (fully-connected) to 0.766 MB (TT-layer), a reduction of over 500×.
• A very wide TT-layer with 262,144 hidden units on CIFAR-10 outperformed other non-convolutional networks, setting a new state-of-the-art for that class.
• The method outperformed low-rank baseline methods in compression efficiency, achieving similar compression with significantly lower accuracy degradation.