[Paper Review] A Survey on Transfer Learning in Natural Language Processing

Zaid Alyafeai, Maged S. Al-shaibani+1 more|arXiv (Cornell University)|May 31, 2020

Topic Modeling79 references53 citations

TL;DR

This survey reviews transfer learning advances in NLP, proposes a taxonomy, and summarizes how pretrained language models enable knowledge transfer across tasks and domains.

ABSTRACT

Deep learning models usually require a huge amount of data. However, these large datasets are not always attainable. This is common in many challenging NLP tasks. Consider Neural Machine Translation, for instance, where curating such large datasets may not be possible specially for low resource languages. Another limitation of deep learning models is the demand for huge computing resources. These obstacles motivate research to question the possibility of knowledge transfer using large trained models. The demand for transfer learning is increasing as many large models are emerging. In this survey, we feature the recent transfer learning advances in the field of NLP. We also provide a taxonomy for categorizing different transfer learning approaches from the literature.

Motivation & Objective

• Motivate the study of transfer learning in NLP due to data and compute constraints in deep models.
• Provide a taxonomy for categorizing transfer learning approaches in NLP.
• Summarize recent language model–driven transfer learning advances and their applications.
• Highlight datasets and evaluation settings used in transfer learning for NLP.

Proposed method

• Review and categorize literature following established taxonomies (transductive vs. inductive transfer learning).
• Describe model architectures (RNNs, CNNs, attention-based Transformers) and their roles in transfer learning.
• Present a taxonomy of transfer techniques: sequential fine-tuning, adapter modules, feature-based representations, and zero-shot approaches.
• Summarize prominent pretrained language models and their training objectives (LM, MLM, NSP, seq2seq,XL/encoder–decoder variants).
• Discuss datasets and benchmarks commonly used for transfer learning in NLP (e.g., SQuAD, GLUE, SuperGLUE).
• Synthesize findings from selected studies to map methods to transfer settings.

Experimental results

Research questions

• RQ1 What transfer learning paradigms (transductive vs. inductive) are most effective for NLP tasks?
• RQ2 How do pretrained language models enable knowledge transfer across tasks and domains?
• RQ3 What are the core strategies (fine-tuning, adapters, feature representations, zero-shot) for transferring learning in NLP?
• RQ4 What datasets and benchmarks best capture the progress and limitations of NLP transfer learning?

Key findings

• Language models have evolved to enable substantial knowledge transfer across NLP tasks.
• Transformer-based pretrained models (e.g., BERT, GPT-2, ELMo, XLNet) define diverse transfer capabilities (unidirectional, bidirectional, masked, seq2seq, encoder–decoder).
• Fine-tuning, adapters, and feature-based representations are central transfer techniques with varying data efficiency.
• Domain adaptation and cross-lingual transfer remain active areas with notable gains on low-resource languages.
• Universal fine-tuning and robust pretrained representations achieve state-of-the-art on multiple NLP benchmarks.
• Long-range dependencies and context representation improvements (e.g., Transformer-XL, RoBERTa) enhance transfer performance.