[Paper Review] Knowledge Inheritance for Pre-trained Language Models

Yujia Qin, Yankai Lin+9 more|arXiv (Cornell University)|May 28, 2021

Natural Language Processing Techniques10 citations

TL;DR

This paper proposes Knowledge Inheritance (KI), a framework that leverages knowledge distillation from pre-trained, smaller language models to efficiently train larger ones. By using knowledge distillation as auxiliary supervision during pre-training, KI significantly improves training efficiency and data efficiency, enabling faster convergence and better performance—especially in low-resource settings and domain adaptation scenarios.

ABSTRACT

Recent explorations of large-scale pre-trained language models (PLMs) have revealed the power of PLMs with huge amounts of parameters, setting off a wave of training ever-larger PLMs. However, it requires tremendous computational resources to train a large-scale PLM, which may be practically unaffordable. In addition, existing large-scale PLMs are mainly trained from scratch individually, ignoring that many well-trained PLMs are available. To this end, we explore the question how could existing PLMs benefit training large-scale PLMs in future. Specifically, we introduce a pre-training framework named "knowledge inheritance" (KI) and explore how could knowledge distillation serve as auxiliary supervision during pre-training to efficiently learn larger PLMs. Experimental results demonstrate the superiority of KI in training efficiency. We also conduct empirical analyses to explore the effects of teacher PLMs' pre-training settings, including model architecture, pre-training data, etc. Finally, we show that KI could be applied to domain adaptation and knowledge transfer.

Motivation & Objective

• Address the high computational cost and resource intensity of training large-scale pre-trained language models (PLMs) from scratch.
• Explore how existing, well-trained PLMs can be leveraged to improve the training efficiency of larger PLMs.
• Investigate whether knowledge from smaller PLMs can be systematically inherited across multiple generations of increasingly larger models.
• Examine the impact of various teacher PLM pre-training settings—such as architecture, data, and training objectives—on the effectiveness of knowledge inheritance.
• Extend the KI framework to domain adaptation, enabling large PLMs to efficiently adapt to new domains using smaller, domain-specific teacher models.

Proposed method

• Introduce Knowledge Inheritance (KI), a pre-training framework that uses knowledge distillation from a smaller, pre-trained teacher PLM to guide the training of a larger student PLM.
• Apply knowledge distillation during the self-supervised pre-training phase of the student model, using soft labels from the teacher to provide auxiliary supervision.
• Use a distillation loss that aligns the student’s output logits with those of the teacher model on masked tokens, enhancing knowledge transfer during pre-training.
• Allow flexible architecture matching between teacher and student models, avoiding strict architectural constraints common in parameter recycling methods.
• Apply KI to both general pre-training and domain adaptation, where multiple domain-specific small PLMs serve as teachers to improve a large PLM’s performance on specific domains.
• Implement a strategy for selecting optimal teachers based on pre-training settings, including model architecture and data distribution.

Experimental results

Research questions

• RQ1 Can distilling knowledge from a previously trained, smaller PLM improve the training efficiency of a larger PLM trained from scratch?
• RQ2 Can knowledge inheritance be sequentially applied across a series of PLMs with increasing sizes, enabling cumulative knowledge transfer across generations?
• RQ3 How do different pre-training settings of the teacher PLM—such as model architecture, pre-training data, and training objectives—affect the performance of knowledge inheritance?
• RQ4 Can KI enhance domain adaptation by enabling a large, already-trained PLM to benefit from smaller, domain-specific teacher models?
• RQ5 How does KI compare to self-learning in terms of training efficiency and data efficiency, especially under low-resource conditions?

Key findings

• KI significantly improves training efficiency: the student model achieves lower final perplexity (PPL) and better downstream performance compared to self-learning, even with the same training budget.
• KI enhances data efficiency: the performance gap between KI and self-learning widens when only limited domain-specific data is available, indicating KI is more effective in low-resource settings.
• KI enables multi-domain knowledge transfer: a large PLM can simultaneously absorb knowledge from multiple domain-specific teachers and achieve improved performance on all domains, as shown in experiments with synthetic CS:BIO data.
• Sequential knowledge inheritance is feasible: knowledge can be accumulated and passed down across multiple generations of increasingly larger PLMs, demonstrating cumulative learning potential.
• KI outperforms self-learning in domain adaptation: even with minimal domain-specific data, KI achieves better performance than self-learning, which tends to overfit under low-resource conditions.
• The choice of teacher model matters: empirical analysis shows that teacher models pre-trained on diverse data and with suitable architectures yield better knowledge transfer, suggesting that teacher selection is a key factor in KI success.