[论文解读] Language model compression with weighted low-rank factorization

Yen-Chang Hsu, Hua Ting+4 more|arXiv (Cornell University)|Jun 30, 2022

Topic Modeling被引用 20

一句话总结

无法从所提供的文本中确定细节，因为所提供的内容是 ICLR 的格式说明，而非论文内容。

ABSTRACT

Factorizing a large matrix into small matrices is a popular strategy for model compression. Singular value decomposition (SVD) plays a vital role in this compression strategy, approximating a learned matrix with fewer parameters. However, SVD minimizes the squared error toward reconstructing the original matrix without gauging the importance of the parameters, potentially giving a larger reconstruction error for those who affect the task accuracy more. In other words, the optimization objective of SVD is not aligned with the trained model's task accuracy. We analyze this previously unexplored problem, make observations, and address it by introducing Fisher information to weigh the importance of parameters affecting the model prediction. This idea leads to our method: Fisher-Weighted SVD (FWSVD). Although the factorized matrices from our approach do not result in smaller reconstruction errors, we find that our resulting task accuracy is much closer to the original model's performance. We perform analysis with the transformer-based language models, showing our weighted SVD largely alleviates the mismatched optimization objectives and can maintain model performance with a higher compression rate. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies requiring expensive model pre-training. Moreover, the evaluation of compressing an already compact model shows our method can further reduce 9% to 30% parameters with an insignificant impact on task accuracy.

研究动机与目标

• 不可用：所提供的文本不包含论文的目标或动机。

提出的方法

• 不可用：在所提供的节选中未提供任何方法细节。

实验结果

研究问题

• RQ1 不可用：在所提供的文本中不存在任何研究问题。

主要发现

• 不可用：在所提供的节选中未给出任何发现。