[논문 리뷰] Language model compression with weighted low-rank factorization

Yen-Chang Hsu, Hua Ting외 4명|arXiv (Cornell University)|2022. 06. 30.

Topic Modeling인용 수 20

한 줄 요약

제공된 텍스트에서 세부 정보를 결정할 수 없습니다. 제공된 내용은 논문 내용이 아닌 ICLR 형식 지침으로 구성되어 있습니다.

ABSTRACT

Factorizing a large matrix into small matrices is a popular strategy for model compression. Singular value decomposition (SVD) plays a vital role in this compression strategy, approximating a learned matrix with fewer parameters. However, SVD minimizes the squared error toward reconstructing the original matrix without gauging the importance of the parameters, potentially giving a larger reconstruction error for those who affect the task accuracy more. In other words, the optimization objective of SVD is not aligned with the trained model's task accuracy. We analyze this previously unexplored problem, make observations, and address it by introducing Fisher information to weigh the importance of parameters affecting the model prediction. This idea leads to our method: Fisher-Weighted SVD (FWSVD). Although the factorized matrices from our approach do not result in smaller reconstruction errors, we find that our resulting task accuracy is much closer to the original model's performance. We perform analysis with the transformer-based language models, showing our weighted SVD largely alleviates the mismatched optimization objectives and can maintain model performance with a higher compression rate. Our method can directly compress a task-specific model while achieving better performance than other compact model strategies requiring expensive model pre-training. Moreover, the evaluation of compressing an already compact model shows our method can further reduce 9% to 30% parameters with an insignificant impact on task accuracy.

연구 동기 및 목표

• 사용 불가: 제공된 텍스트에는 논문 목표나 동기가 포함되어 있지 않습니다.

제안 방법

• 사용 불가: 제공된 발췌에 방법론적 세부사항이 제공되지 않습니다.

실험 결과

연구 질문

• RQ1 사용 불가: 제공된 텍스트에 연구 질문이 존재하지 않습니다.

주요 결과

• 사용 불가: 제공된 발췌에 제시된 결과가 없습니다.