[Paper Review] Compressing Language Models using Doped Kronecker Products

Urmish Thakker, Paul N. Whatmough+3 more|arXiv (Cornell University)|Jan 24, 2020

Blind Source Separation Techniques20 references4 citations

TL;DR

This paper proposes Doped Kronecker Product (DKP) compression, which enhances Kronecker Product-based model compression by adding a sparse overlay matrix to recover accuracy lost in large NLP models. By introducing co-matrix row dropout regularization (CMR), DKP mitigates co-matrix adaptation during training, achieving 25× compression of a 25 MB LSTM language model with only 1.2% perplexity increase—surpassing pruning, LMF, and HMD methods.

ABSTRACT

Kronecker Products (KP) have been used to compress IoT RNN Applications by 15-38x compression factors, achieving better results than traditional compression methods. However when KP is applied to large Natural Language Processing tasks, it leads to significant accuracy loss (approx 26%). This paper proposes a way to recover accuracy otherwise lost when applying KP to large NLP tasks, by allowing additional degrees of freedom in the KP matrix. More formally, we propose doping, a process of adding an extremely sparse overlay matrix on top of the pre-defined KP structure. We call this compression method doped kronecker product compression. To train these models, we present a new solution to the phenomenon of co-matrix adaption (CMA), which uses a new regularization scheme called co matrix dropout regularization (CMR). We present experimental results that demonstrate compression of a large language model with LSTM layers of size 25 MB by 25x with 1.4% loss in perplexity score. At 25x compression, an equivalent pruned network leads to 7.9% loss in perplexity score, while HMD and LMF lead to 15% and 27% loss in perplexity score respectively.

Motivation & Objective

• To address the significant accuracy drop (up to 26%) when applying Kronecker Product (KP) compression to large NLP models.
• To restore model capacity lost during KP compression by introducing additional degrees of freedom through a sparse overlay matrix.
• To develop a training methodology that prevents co-matrix adaptation (CMA), a critical issue in DKP training where KP and sparse matrix components co-adapt and degrade performance.
• To achieve high compression ratios (e.g., 25×) while maintaining low perplexity loss compared to existing compression techniques.

Proposed method

• The method expresses a weight matrix as a sum of a Kronecker Product matrix (M_kp = B ⊗ C) and a sparse matrix (M_sp), forming W = M_kp + M_sp.
• The sparse matrix M_sp starts dense and is pruned during training, allowing the model to learn which elements of M_kp require additional degrees of freedom.
• Co-matrix row dropout regularization (CMR) is introduced, applying Bernoulli dropout independently to the outputs of M_kp and M_sp for each row to prevent co-adaptation.
• CMR is dynamically adjusted: its use diminishes as M_sp becomes sparser, reducing regularization pressure over time.
• The training process uses back-propagation to jointly optimize M_kp and M_sp, with sparsity in M_sp controlling the overall compression factor.
• The method is applied to LSTM-based language models, with compression ratios and perplexity evaluated on the PTB benchmark.

Experimental results

Research questions

• RQ1 Can adding a sparse overlay matrix to a Kronecker Product-compressed weight matrix recover accuracy lost in large NLP models?
• RQ2 What training challenges arise when combining KP and sparse matrices, and how can co-matrix adaptation be effectively mitigated?
• RQ3 Can a novel regularization scheme like co-matrix row dropout regularization (CMR) enable stable training of doped Kronecker Product models?
• RQ4 How does DKP-based compression compare to pruning, low-rank factorization, and hybrid methods in terms of compression ratio and perplexity?
• RQ5 What is the maximum achievable compression ratio with minimal perplexity degradation using DKP and CMR?

Key findings

• DKP achieves 25× compression of a 25 MB LSTM language model with only 1.2% increase in test perplexity, compared to the baseline of 82.04.
• At 25× compression, DKP reduces perplexity loss by 7.9 percentage points compared to pruned networks (7.9% loss vs. 1.2% for DKP).
• DKP outperforms HMD (15% perplexity loss) and LMF (27% loss) at 25× compression, demonstrating superior accuracy retention.
• CMR regularization effectively prevents co-matrix adaptation, as evidenced by stable training perplexity even at high sparsity levels in M_sp.
• The method achieves 2.5× higher compression than the best-performing prior technique (Park et al., 2017) while maintaining lower perplexity.
• At 100× compression, DKP achieves a test perplexity of 138.3, a 4-point improvement over the baseline 150.737, showing scalability with controlled sparsity.