[Paper Review] Compressing Recurrent Neural Networks Using Hierarchical Tucker Tensor Decomposition

Miao Yin, Siyu Liao+3 more|arXiv (Cornell University)|May 9, 2020

Tensor decomposition and applicationsMathematics20 references21 citations

TL;DR

This paper proposes Hierarchical Tucker (HT) tensor decomposition to compress Recurrent Neural Networks (RNNs), enabling stronger hierarchical representation learning and superior model compression. HT-LSTM achieves significantly higher compression ratios (up to 12,945×) and improved test accuracy—up to 88.1% on YouTube Faces and 98.1% on UCF11—outperforming state-of-the-art methods like TT-LSTM, TR-LSTM, and BT-LSTM across multiple benchmarks.

ABSTRACT

Recurrent Neural Networks (RNNs) have been widely used in sequence analysis and modeling. However, when processing high-dimensional data, RNNs typically require very large model sizes, thereby bringing a series of deployment challenges. Although the state-of-the-art tensor decomposition approaches can provide good model compression performance, these existing methods are still suffering some inherent limitations, such as restricted representation capability and insufficient model complexity reduction. To overcome these limitations, in this paper we propose to develop compact RNN models using Hierarchical Tucker (HT) decomposition. HT decomposition brings strong hierarchical structure to the decomposed RNN models, which is very useful and important for enhancing the representation capability. Meanwhile, HT decomposition provides higher storage and computational cost reduction than the existing tensor decomposition approaches for RNN compression. Our experimental results show that, compared with the state-of-the-art compressed RNN models, such as TT-LSTM, TR-LSTM and BT-LSTM, our proposed HT-based LSTM (HT-LSTM), consistently achieves simultaneous and significant increases in both compression ratio and test accuracy on different datasets.

Motivation & Objective

• Address the limitations of existing tensor decomposition methods (e.g., TT, TR, BT) in RNN compression, which suffer from restricted representation capability due to weak hierarchical structure.
• Overcome insufficient model complexity reduction in prior approaches by introducing a decomposition method that enables stronger hierarchical modeling and greater parameter and computation savings.
• Develop a compact RNN architecture using Hierarchical Tucker (HT) decomposition that enhances representation power while minimizing storage and computational costs.
• Demonstrate that HT-based RNNs can achieve higher accuracy and compression ratios than state-of-the-art compressed RNN models on diverse sequence modeling tasks.
• Validate the effectiveness of HT decomposition in both end-to-end training and pre-trained CNN-finetuning settings across multiple video and text datasets.

Proposed method

• Apply Hierarchical Tucker (HT) decomposition to the weight matrices of LSTM networks, particularly the input-to-hidden and output-to-hidden weight tensors.
• Represent the original high-dimensional weight tensor as a core tensor combined with a set of factor matrices, forming a hierarchical structure that captures multi-level dependencies.
• Use a multi-level tensor decomposition framework where each decomposition level is applied recursively to reduce the rank and dimensionality of the weight tensors.
• Optimize the HT-LSTM model using standard backpropagation with ADAM optimizer, incorporating L2 regularization and dropout for improved generalization.
• In the pre-trained CNN setting, use Inception-V3 to extract compact features (2,048-dim), reshape them into 4D tensors (e.g., 8×8×8×4), and apply HT decomposition to the RNN’s weight matrices.
• Set the rank of all HT decomposition nodes to 4 in all experiments to ensure consistent comparison with prior work.

Experimental results

Research questions

• RQ1 Can Hierarchical Tucker decomposition provide better representation capability than existing tensor decomposition methods (e.g., TT, TR, BT) in compressed RNNs?
• RQ2 Does HT-based RNN compression achieve higher model compression ratios while maintaining or improving test accuracy compared to state-of-the-art methods?
• RQ3 How does HT-LSTM perform in comparison to TT-LSTM, TR-LSTM, and BT-LSTM on standard video and text sequence benchmarks?
• RQ4 Can HT-LSTM maintain high performance when used with pre-trained CNNs as feature extractors, especially in low-resource settings?
• RQ5 To what extent does the hierarchical structure of HT decomposition enhance the modeling of sequential dependencies in RNNs?

Key findings

• HT-LSTM achieves a test accuracy of 88.1% on the YouTube Faces dataset, which is 7.3 percentage points higher than the previous state-of-the-art model (80.8%).
• On the UCF11 dataset, HT-LSTM reaches 98.1% accuracy—3.5 percentage points higher than the best prior result (94.6%) and 4.3 points higher than TR-LSTM using the same pre-trained CNN.
• HT-LSTM achieves a compression ratio of 12,945× on UCF11, far exceeding the 25× compression ratio of TR-LSTM when compressing the same vanilla LSTM.
• On the HMDB51 dataset, HT-LSTM achieves 64.2% accuracy, outperforming TR-LSTM (63.8%) and surpassing most prior methods, even without using optical flow data.
• HT-LSTM maintains strong performance with only RGB input, achieving 64.2% accuracy compared to 49.8% for a prior SOTA model using only RGB, demonstrating robustness and efficiency.
• The hierarchical structure of HT decomposition enables better modeling of complex sequential patterns, leading to improved generalization and representation power over non-hierarchical methods.