[論文レビュー] Language model compression with weighted low-rank factorization

Yen-Chang Hsu, Hua Ting+4 more|arXiv (Cornell University)|Jun 30, 2022

Topic Modeling被引用数 20

ひとこと要約

提供されたテキストから詳細を判断できません。提供された内容は論文の内容ではなく、ICLRのフォーマット指示で構成されています。

ABSTRACT

Factorizing a large matrix into small matrices is a popular strategy for model compression. Singular value decomposition (SVD) plays a vital role in this compression strategy, approximating a learned matrix with fewer parameters. However, SVD minimizes the squared error toward reconstructing the original matrix without gauging the importance of the parameters, potentially giving a larger reconstruction error for those who affect the task accuracy more. In other words, the optimization objective of SVD is not aligned with the trained model's task accuracy. We analyze this previously unexplored problem, make observations, and address it by introducing Fisher information to weigh the importance of parameters affecting the model prediction. This idea leads to our method: Fisher-Weighted SVD (FWSVD). Although the factorized matrices from our approach do not result in smaller reconstruction errors, we find that our resulting task accuracy is much closer to the original model's performance. We perform analysis with the transformer-based language models, showing our weighted SVD largely alleviates the mismatched optimization objectives and can maintain model performance with a higher compression rate. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies requiring expensive model pre-training. Moreover, the evaluation of compressing an already compact model shows our method can further reduce 9% to 30% parameters with an insignificant impact on task accuracy.

研究の動機と目的

• 利用不可: 提供されたテキストには論文の目的や動機が含まれていません。

提案手法

• 利用不可: 提供された抜粋には方法論の詳細が含まれていません。

実験結果

リサーチクエスチョン

• RQ1 利用不可: 提供されたテキストには研究質問が含まれていません。

主な発見

• 利用不可: 提供された抜粋には結果が示されていません。