[Paper Review] A Short Study on Compressing Decoder-Based Language Models

Tianda Li, Yassir El Mesbahi+7 more|arXiv (Cornell University)|Oct 16, 2021

Topic Modeling30 references4 citations

TL;DR

This paper proposes a time-efficient method for compressing decoder-based language models by pre-training a distilled GPT-2 model from scratch using layer truncation and data cleaning, outperforming DistilGPT-2 on downstream tasks despite significantly reduced training time. It introduces a pseudo-uniform truncation strategy and demonstrates that data cleaning improves both efficiency and performance.

ABSTRACT

Pre-trained Language Models (PLMs) have been successful for a wide range of natural language processing (NLP) tasks. The state-of-the-art of PLMs, however, are extremely large to be used on edge devices. As a result, the topic of model compression has attracted increasing attention in the NLP community. Most of the existing works focus on compressing encoder-based models (tiny-BERT, distilBERT, distilRoBERTa, etc), however, to the best of our knowledge, the compression of decoder-based models (such as GPT-2) has not been investigated much. Our paper aims to fill this gap. Specifically, we explore two directions: 1) we employ current state-of-the-art knowledge distillation techniques to improve fine-tuning of DistilGPT-2. 2) we pre-train a compressed GPT-2 model using layer truncation and compare it against the distillation-based method (DistilGPT2). The training time of our compressed model is significantly less than DistilGPT-2, but it can achieve better performance when fine-tuned on downstream tasks. We also demonstrate the impact of data cleaning on model performance.

Motivation & Objective

• To address the lack of research on compressing decoder-based models like GPT-2, which are less studied than encoder-based models such as BERT.
• To improve fine-tuning performance of DistilGPT-2 using state-of-the-art knowledge distillation and teacher-free techniques.
• To explore layer truncation methods for initializing smaller GPT-2 models from larger teachers, aiming for faster pre-training.
• To evaluate the impact of data cleaning on pre-training efficiency and downstream performance.
• To demonstrate that a compressed GPT-2 model pre-trained without knowledge distillation can outperform DistilGPT-2 on downstream tasks.

Proposed method

• Pre-training a 6-layer GPT-2 student model using pseudo-uniform layer truncation from a 12-layer GPT-2-xl teacher model.
• Applying data cleaning to the OpenWebText dataset by removing HTML, short sentences, non-alphanumeric noise, and duplicates, reducing dataset size by 65.5%.
• Using a pseudo-uniform initialization strategy that evenly distributes parameters from the teacher to the student across layers.
• Benchmarking knowledge distillation techniques including Annealing-KD, MATE-KD, and RAIL-KD for fine-tuning the distilled model.
• Evaluating performance on Wikitext103 and SuperGLUE benchmarks using zero-shot and fine-tuned perplexity and accuracy scores.
• Comparing training time and performance between models pre-trained with and without knowledge distillation.

Experimental results

Research questions

• RQ1 Can knowledge distillation techniques improve fine-tuning performance of DistilGPT-2 on downstream NLP tasks?
• RQ2 Does pre-training a compressed GPT-2 model from scratch using layer truncation outperform DistilGPT-2 in terms of downstream performance and training efficiency?
• RQ3 How does data cleaning of the OpenWebText dataset affect the pre-training time and final model performance?
• RQ4 Which layer truncation strategy yields the best performance for initializing a smaller GPT-2 model from a larger teacher?
• RQ5 Can a distilled GPT-2 model pre-trained without knowledge distillation achieve better performance than DistilGPT-2?

Key findings

• The proposed model pre-trained with pseudo-uniform truncation and cleaned data achieved a test perplexity of 22.42 on Wikitext103 after fine-tuning, outperforming DistilGPT-2’s 21.13 in zero-shot but with significantly less training time.
• The model pre-trained on cleaned OpenWebText data reduced training time to 42 hours (vs. 768 hours for DistilGPT-2), while achieving comparable or better performance on 5 out of 7 downstream tasks.
• The pseudo-uniform truncation strategy outperformed other initialization methods, achieving a 45.93 zero-shot perplexity on Wikitext103, close to DistilGPT-2’s 45.26.
• Data cleaning reduced the OpenWebText dataset size from 332 million to 114 million samples, decreasing pre-training time by 94% while maintaining or improving model performance.
• The model pre-trained on 10% of the cleaned dataset achieved a fine-tuned perplexity of 22.42, comparable to DistilGPT-2’s 21.13, despite having only 1/3 the parameters of the teacher model.
• On the SuperGLUE benchmark, the cleaned-data pre-trained model achieved state-of-the-art performance on 5 out of 7 tasks, including 77.64% accuracy on BoolQ and 73.74% on WSC, outperforming DistilGPT-2 in most cases.