[Paper Review] Compressed Deep Networks: Goodbye SVD, Hello Robust Low-Rank Approximation

Murad Tukan, Alaa Maalouf+2 more|arXiv (Cornell University)|Sep 11, 2020

Tensor decomposition and applications58 references4 citations

TL;DR

This paper proposes replacing traditional SVD-based low-rank approximation with robust $ε$-approximate $µ$-norm low-rank approximation for compressing deep neural networks, particularly embedding layers. By using $\ell_p$-norms with $p \in [1,2]$, the method achieves significantly better accuracy retention under compression—e.g., 28% compression of RoBERTa’s embedding layer with only 0.63% accuracy drop versus 11% for SVD—due to improved outlier resistance.

ABSTRACT

A common technique for compressing a neural network is to compute the $k$-rank $\ell_2$ approximation $A_{k,2}$ of the matrix $A\in\mathbb{R}^{n imes d}$ that corresponds to a fully connected layer (or embedding layer). Here, $d$ is the number of the neurons in the layer, $n$ is the number in the next one, and $A_{k,2}$ can be stored in $O((n+d)k)$ memory instead of $O(nd)$. This $\ell_2$-approximation minimizes the sum over every entry to the power of $p=2$ in the matrix $A - A_{k,2}$, among every matrix $A_{k,2}\in\mathbb{R}^{n imes d}$ whose rank is $k$. While it can be computed efficiently via SVD, the $\ell_2$-approximation is known to be very sensitive to outliers ("far-away" rows). Hence, machine learning uses e.g. Lasso Regression, $\ell_1$-regularization, and $\ell_1$-SVM that use the $\ell_1$-norm. This paper suggests to replace the $k$-rank $\ell_2$ approximation by $\ell_p$, for $p\in [1,2]$. We then provide practical and provable approximation algorithms to compute it for any $p\geq1$, based on modern techniques in computational geometry. Extensive experimental results on the GLUE benchmark for compressing BERT, DistilBERT, XLNet, and RoBERTa confirm this theoretical advantage. For example, our approach achieves $28\%$ compression of RoBERTa's embedding layer with only $0.63\%$ additive drop in the accuracy (without fine-tuning) in average over all tasks in GLUE, compared to $11\%$ drop using the existing $\ell_2$-approximation. Open code is provided for reproducing and extending our results.

Motivation & Objective

• To address the sensitivity of SVD-based low-rank approximation to outliers in embedding matrices of NLP models.
• To improve model compression efficiency by replacing $\ell_2$-norm approximation with more robust $\ell_p$-norm approximation for $p \in [1,2]$.
• To develop practical and provably accurate algorithms for computing $\ell_p$-based low-rank approximations in deep network compression.
• To demonstrate that robust low-rank approximation outperforms SVD in accuracy retention across multiple NLP benchmarks without fine-tuning.
• To provide open-source code for reproducibility and extension of the proposed compression framework.

Proposed method

• Replaces the standard $\ell_2$-norm low-rank approximation (via SVD) with $\ell_p$-norm approximation for $p \in [1,2]$ to enhance robustness against outliers.
• Employs modern computational geometry techniques to design practical and provable approximation algorithms for $\ell_p$-based $k$-rank matrix approximation.
• Applies the method to compress embedding layers by factorizing the weight matrix $A \in \mathbb{R}^{n \times d}$ into two smaller matrices $U \in \mathbb{R}^{n \times k}$ and $W \in \mathbb{R}^{k \times d}$, reducing memory from $O(nd)$ to $O(k(n+d))$.
• Uses the $\ell_p$-norm objective $\|A - A_k\|_p^p = \sum_{i=1}^n \|A^{(i)} - A_k^{(i)}\|_p^p$ to minimize error while being less sensitive to extreme values than $\ell_2$.
• Employs a two-stage approach: first, compute a robust low-rank approximation using $\ell_p$-norms; second, apply it to compress the embedding layer without fine-tuning.
• Validates the method on the GLUE benchmark using BERT, RoBERTa, XLNet, and DistilBERT, comparing accuracy and compression rates against SVD.

Experimental results

Research questions

• RQ1 Can $\ell_p$-norm low-rank approximation with $p \in [1,2]$ provide better compression accuracy than $\ell_2$-based SVD in deep neural networks?
• RQ2 How does the robustness of $\ell_p$-norm approximation against outliers affect model performance during compression, especially in large-vocabulary or small models?
• RQ3 To what extent can $\ell_p$-based compression maintain or even improve model accuracy without fine-tuning?
• RQ4 What is the practical and theoretical efficiency of computing $\ell_p$-based $k$-rank approximations for real-world NLP models?
• RQ5 Can the proposed method generalize across different architectures like RoBERTa, XLNet, and DistilBERT, especially when the embedding layer is a bottleneck for accuracy?

Key findings

• The proposed $\ell_p$-based compression achieved 28% compression of RoBERTa’s embedding layer with only 0.63% average accuracy drop across GLUE tasks, compared to 11% drop using SVD.
• For DistilBERT, the method outperformed SVD due to its robustness against outliers and sensitivity to small errors in small models.
• In RoBERTa, the large vocabulary size (50,265 subword units) increased outlier likelihood, making the $\ell_p$-based method particularly effective.
• For XLNet, the method outperformed SVD due to the impact of outliers on relative positional embeddings, which are critical for performance.
• On some tasks like SST-2, accuracy improved slightly at low compression rates, likely due to removal of redundant dimensions.
• The method consistently outperformed SVD across most GLUE tasks and compression rates, demonstrating superior robustness and accuracy retention.