[Paper Review] A Tensorized Transformer for Language Modeling

Xindian Ma, Peng Zhang+5 more|arXiv (Cornell University)|Jun 24, 2019

Topic Modeling39 references61 citations

TL;DR

The paper introduces Multi-linear attention, a Block-Term Tensor Decomposition-based compression of Transformer multi-head attention, achieving high parameter reduction with competitive language modeling and translation performance.

ABSTRACT

Latest development of neural models has connected the encoder and decoder through a self-attention mechanism. In particular, Transformer, which is solely based on self-attention, has led to breakthroughs in Natural Language Processing (NLP) tasks. However, the multi-head attention mechanism, as a key component of Transformer, limits the effective deployment of the model to a resource-limited setting. In this paper, based on the ideas of tensor decomposition and parameters sharing, we propose a novel self-attention model (namely Multi-linear attention) with Block-Term Tensor Decomposition (BTD). We test and verify the proposed attention method on three language modeling tasks (i.e., PTB, WikiText-103 and One-billion) and a neural machine translation task (i.e., WMT-2016 English-German). Multi-linear attention can not only largely compress the model parameters but also obtain performance improvements, compared with a number of language modeling approaches, such as Transformer, Transformer-XL, and Transformer with tensor train decomposition.

Motivation & Objective

• Motivate compression of large Transformer models for resource-limited settings in NLP.
• Propose Multi-linear attention using Block-Term Tensor Decomposition to share parameters and employ low-rank structure.
• Integrate the compressed attention into Transformer and demonstrate end-to-end training.
• Analyze compression and compute complexity implications of the proposed approach.
• Empirically validate on language modeling benchmarks and WMT English-German translation.

Proposed method

• Represent self-attention output as a linear combination of orthonormal base vectors via Tucker decomposition (Single-block attention).
• Construct Multi-linear attention using Block-Term Tensor Decomposition to share Q, K, V factor matrices across heads.
• Split-concatenate procedure to form multi-head outputs compatible with Transformer’s encoder/decoder. Demonstrate reconstruction capability of scaled dot-product attention from the Tucker-based representation.
• Analyze compression ratio and time/space complexity, showing significant parameter reduction with comparable performance.

Experimental results

Research questions

• RQ1 Can multi-head attention in Transformer be effectively compressed via Block-Term Tensor Decomposition without sacrificing performance?
• RQ2 Does sharing factor matrices across attention heads yield competitive language modeling and translation results under reduced parameter budgets?
• RQ3 How does the proposed Multi-linear attention compare to Transformer variants (e.g., Transformer-XL, Sparse Transformer) in terms of parameters and perplexity/BLEU?
• RQ4 What is the impact of core tensor size on performance and potential overfitting in language modeling?

Key findings

• Multi-linear attention achieves substantial parameter reduction (e.g., core-1 and core-2 configurations) while maintaining competitive or superior language modeling performance.
• On PTB and WikiText-103, Tensorized Transformer core-1 and core-2 reach perplexities comparable to or better than several strong baselines with far fewer parameters.
• On One-Billion Word, the Tensorized Transformer achieves a strong perplexity with notable parameter efficiency, outperforming vanilla Transformer baselines.
• In NMT (WMT-2016 English-German), Tensorized Transformer cores yield BLEU scores higher than the basic Transformer baseline while using fewer parameters.
• The approach can reconstruct the scaled dot-product attention and generalizes beyond the original Transformer mechanism, enabling end-to-end training within the Transformer framework.