[Paper Review] Exploring Versatile Generative Language Model Via Parameter-Efficient Transfer Learning

Zhaojiang Lin, Andrea Madotto+1 more|arXiv (Cornell University)|Apr 8, 2020

Topic Modeling48 references4 citations

TL;DR

This paper proposes a parameter-efficient method to adapt a single pre-trained generative language model (e.g., GPT-2) for multiple downstream language generation tasks simultaneously, using residual adapter layers and task embeddings. By adding only 2–3% task-specific parameters per task, the model achieves performance on par with or better than full fine-tuning, enabling efficient on-device deployment.

ABSTRACT

Fine-tuning pre-trained generative language models to down-stream language generation tasks has shown promising results. However, this comes with the cost of having a single, large model for each task, which is not ideal in low-memory/power scenarios (e.g., mobile). In this paper, we propose an effective way to fine-tune multiple down-stream generation tasks simultaneously using a single, large pre-trained model. The experiments on five diverse language generation tasks show that by just using an additional 2-3% parameters for each task, our model can maintain or even improve the performance of fine-tuning the whole model.

Motivation & Objective

• To address the inefficiency of fine-tuning a separate large model for each downstream language generation task, especially in low-memory environments.
• To enable a single pre-trained language model to dynamically serve multiple diverse generation tasks without retraining the backbone.
• To explore parameter-efficient fine-tuning techniques that maintain or improve performance while minimizing parameter updates.
• To evaluate the trade-offs between multi-task and continual learning settings in parameter-efficient adaptation.

Proposed method

• Introduce residual adapter layers as task-specific adapters inserted after each transformer block, with learnable parameters frozen during fine-tuning of the main model.
• Use task embeddings to condition the adapter layers, allowing the model to dynamically adapt its generation behavior per task.
• Apply knowledge distillation to transfer knowledge from a full fine-tuned model to the adapter-only model, improving performance.
• Train using both multi-task learning (MT) and continual learning (CL) settings to compare generalization and catastrophic forgetting.
• Freeze the pre-trained language model weights and only fine-tune the adapter and task embedding parameters to reduce memory and compute costs.
• Use a shared backbone (e.g., GPT-2) across five diverse tasks: summarization, machine translation, dialogue generation, conversational QA, and task-oriented NLG.

Experimental results

Research questions

• RQ1 Can a single pre-trained language model be effectively adapted to multiple diverse language generation tasks using only a small number of additional parameters?
• RQ2 How does the performance of parameter-efficient adaptation compare to full fine-tuning across multiple generation tasks?
• RQ3 What is the impact of freezing the language model weights versus fine-tuning them in the context of multi-task and continual learning?
• RQ4 How effective is knowledge distillation in improving the performance of the adapter-based model?
• RQ5 Can task embeddings effectively steer the model’s generation output toward specific task-specific behaviors?

Key findings

• The proposed method achieves an average performance of 57.97 BLEU/F1 on five diverse language generation tasks, outperforming full fine-tuning (57.77) and matching or exceeding SOTA results with only 1.13× the parameters of the base model.
• Adding just 2–3% additional parameters per task (via adapters and task embeddings) maintains or improves performance compared to full fine-tuning, demonstrating high parameter efficiency.
• The model with distillation and adapters (VLM) achieved 24.19 ROUGE-2 and 67.1 F1 on summarization, outperforming full fine-tuning (25.45 ROUGE-2, 66.4 F1) and SOTA systems on the same task.
• Removing task embeddings led to a significant drop in performance (e.g., 47.25 average score), showing their critical role in task-specific adaptation.
• The model trained with continual learning (CL) generalizes better than multi-task learning (MT) in some settings, suggesting CL is more suitable for sequential task deployment.
• In dialogue generation, the model achieved 66.2 F1 on CoQA, outperforming the SOTA system (66.2) and matching the reference human performance, despite using fewer parameters.