[Paper Review] Tensor-Train Recurrent Neural Networks for Video Classification

Yinchong Yang, Denis Krompaß+1 more|arXiv (Cornell University)|Jul 6, 2017

Tensor decomposition and applicationsMathematics34 references55 citations

TL;DR

The paper proposes end-to-end Tensor-Train Layers to factorize input-to-hidden weights in RNNs, enabling TT-RNNs to process high-dimensional video data with dramatically fewer parameters while achieving competitive accuracy.

ABSTRACT

The Recurrent Neural Networks and their variants have shown promising performances in sequence modeling tasks such as Natural Language Processing. These models, however, turn out to be impractical and difficult to train when exposed to very high-dimensional inputs due to the large input-to-hidden weight matrix. This may have prevented RNNs' large-scale application in tasks that involve very high input dimensions such as video modeling; current approaches reduce the input dimensions using various feature extractors. To address this challenge, we propose a new, more general and efficient approach by factorizing the input-to-hidden weight matrix using Tensor-Train decomposition which is trained simultaneously with the weights themselves. We test our model on classification tasks using multiple real-world video datasets and achieve competitive performances with state-of-the-art models, even though our model architecture is orders of magnitude less complex. We believe that the proposed approach provides a novel and fundamental building block for modeling high-dimensional sequential data with RNN architectures and opens up many possibilities to transfer the expressive and advanced architectures from other domains such as NLP to modeling high-dimensional sequential data.

Motivation & Objective

• Motivate end-to-end modeling of high-dimensional video data with RNNs without relying on heavy CNN preprocessors.
• Introduce Tensor-Train Factorization to compress input-to-hidden weight mappings in RNNs.
• Develop Tensor-Train RNN variants (TT-SRNN, TT-GRU, TT-LSTM) that are trained jointly with the rest of the network.
• Demonstrate that TT-RNNs can achieve competitive performance with substantially fewer parameters on multiple video benchmarks.

Proposed method

• Explain Tensor-Train Factorization (TTF) and its Core tensors as TT-cores that reconstruct weight matrices via a chain of multiplications.
• Replace the input-to-hidden weight matrix with a Tensor-Train Layer (TTL) that is trained end-to-end.
• Apply TTL to RNN gates in TT-GRU and TT-LSTM (and a vanilla TT-SRNN variant) to model high-dimensional sequences.
• Use double-indexing (i_k, j_k) to enable efficient TT representations of weight tensors.
• Provide a parallelization trick by concatenating gates to reduce parameter count and computation.
• Show compression rates where large FC layers (e.g., 14,745,600 params) are replaced by TTLs with thousands of parameters.

Experimental results

Research questions

• RQ1 Can Tensor-Train factorization be effectively integrated into RNNs to handle high-dimensional video inputs end-to-end?
• RQ2 How does TT-RNN performance compare to plain RNNs (GRU/LSTM) and CNN-based preprocessing pipelines on standard video classification benchmarks?
• RQ3 What are the parameter and training-time trade-offs when using TTLs in LSTM/GRU architectures for video data?

Key findings

• TT-GRU achieves 0.813 accuracy on UCF11 with only 3,232 input-to-hidden parameters (vs. 44,236,800 for plain GRU).
• TT-LSTM achieves 0.796 accuracy on UCF11 with 3,360 input-to-hidden parameters (vs. 58,982,400 for plain LSTM).
• On Hollywood2, TT-GRU achieves 0.537 MAP and TT-LSTM achieves 0.546 MAP with around 3,104–3,304 input-to-hidden parameters, significantly reducing model size.
• TT-RNN variants substantially reduce training time (TT variants around 2 days vs. 8–10 days for plain GRU/LSTM on UCF11).
• On Youtube Celebrities Face data, TT-GRU achieves 0.800 accuracy and TT-LSTM 0.755 with ~3,328–3,392 parameters, outperforming plain LSTM/GRU in parameter efficiency.