[Paper Review] Factorization tricks for LSTM networks

Oleksii Kuchaiev, Boris Ginsburg|arXiv (Cornell University)|Mar 31, 2017

Neural Networks and Applications13 references88 citations

TL;DR

The paper introduces factorized LSTM (F-LSTM) and group LSTM (G-LSTM) to reduce parameters and accelerate training, achieving near-state perplexity on the One Billion Word Benchmark with fewer RNN parameters.

ABSTRACT

We present two simple ways of reducing the number of parameters and accelerating the training of large Long Short-Term Memory (LSTM) networks: the first one is "matrix factorization by design" of LSTM matrix into the product of two smaller matrices, and the second one is partitioning of LSTM matrix, its inputs and states into the independent groups. Both approaches allow us to train large LSTM networks significantly faster to the near state-of the art perplexity while using significantly less RNN parameters.

Motivation & Objective

• Motivate reducing LSTM parameter count and training time for large language models.
• Propose two modifications to LSTMP (factorization and grouping) to lower compute and memory requirements.
• Evaluate the methods on a large-scale language modeling task and compare to a strong baseline.
• Provide guidance on when and how these approaches improve training efficiency while maintaining performance.

Proposed method

• Factorized LSTM (F-LSTM) replaces the 4n x 2p affine transform with a product of two smaller matrices (W2*W1) where W1 has size 2p x r and W2 has size r x 4n, using rank r < p.
• Group LSTM (G-LSTM) partitions input x and hidden state h into independent groups, with each group computing its own affine transform from R^{2p/k} to R^{4n/k} for k groups.
• Both methods reduce the number of parameters compared to the original LSTMP; F-LSTM reduces parameters to (r*2p + r*4n) versus (2p*4n); G-LSTM distributes computations across groups enabling model parallelism.
• The study uses LSTMP with projection (size p) and tests on the One Billion Word Benchmark for language modeling.
• The authors note projection size is crucial to control embedding and softmax sizes.

Experimental results

Research questions

• RQ1 Can factorizing the LSTM weight matrix maintain performance while reducing parameter count and speeding up training?
• RQ2 Can partitioning the LSTM into independent groups (G-LSTM) provide similar or better efficiency with fewer parameters?
• RQ3 How do F-LSTM and G-LSTM compare to a baseline BIGLSTM on perplexity and training throughput on the One Billion Word Benchmark?
• RQ4 What are the trade-offs in model size, speed (words/sec), and perplexity across different group counts or factorization ranks?

Key findings

• G-LSTM with 2 groups achieves near-baseline perplexity with substantially fewer parameters and faster training.
• F-LSTM with intermediate rank (F512) trains faster and uses fewer parameters than BIGLSTM, achieving competitive perplexity.
• Increasing groups (G-4, G-8) reduces parameter count further but can increase perplexity; G-8 yields 39.4 perplexity with 850.4k parameters.
• Factorized and group-based LSTMs train significantly faster within the same wall-clock time budget, reaching near state-of-the-art perplexity.
• Hierarchical group configurations (e.g., G4-G8) show potential for improved efficiency without large accuracy loss.