[Paper Review] Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey

Bonan Min, Hayley Ross+7 more|arXiv (Cornell University)|Nov 1, 2021

Topic Modeling182 references161 citations

TL;DR

A comprehensive survey of how large pre-trained language models (PLMs) enable NLP via pre-train/fine-tune, prompting, and text generation, plus data generation for augmentation and future directions.

ABSTRACT

Large, pre-trained transformer-based language models such as BERT have drastically changed the Natural Language Processing (NLP) field. We present a survey of recent work that uses these large language models to solve NLP tasks via pre-training then fine-tuning, prompting, or text generation approaches. We also present approaches that use pre-trained language models to generate data for training augmentation or other purposes. We conclude with discussions on limitations and suggested directions for future research.

Motivation & Objective

• Explain the shift to large pre-trained transformer models in NLP and the three main paradigms (pre-train then fine-tune, prompt-based learning, NLP as text generation).
• Summarize how PLMs are used for tasks like parsing, IE, QA, TE, and sentiment analysis beyond generative tasks.
• Discuss data generation approaches and limitations, and outline future research directions.

Proposed method

• Describe three PLM-based paradigms: pre-train then fine-tune, prompting, and text generation, and a data-generation complement.
• Detail model classes (autoregressive, masked language models, encoder-decoder) and typical pre-training objectives.
• Summarize fine-tuning strategies including full fine-tuning, adapter-based, and efficient fine-tuning methods.
• Discuss data sources, scale effects, and domain-specific pre-training.
• Review prompt design methodologies and their benefits for few-shot learning and task probing.

Figure 1: Three types of pre-trained language models. Model architecture illustrations are from Lewis et al. ( 2020 ) . For the encoder-decoder model, the corruption strategy of document rotation is shown. Alternatives include sentence permutation, text infilling, token deletion/masking, etc.

Experimental results

Research questions

• RQ1 What are the dominant PLM-based paradigms for solving NLP tasks and how do they differ in approach and optimization?
• RQ2 How do model architecture, pre-training data, and fine-tuning strategies affect performance across NLP tasks?
• RQ3 What role does prompting play in leveraging PLMs for few-shot learning and task alignment?
• RQ4 How can PLMs be used to generate data or augment training for NLP tasks, and what are the limitations?
• RQ5 What are the current limitations and future directions for PLMs in NLP?

Key findings

• PLMs enable state-of-the-art performance across diverse NLP tasks through three main paradigms: pre-train then fine-tune, prompt-based learning, and NLP as text generation.
• Autoregressive, masked, and encoder-decoder PLMs differ in training objectives and suitability for different task types.
• Fine-tuning strategies range from full model tuning to adapters and parameter-efficient methods that mitigate forgetting and reduce training costs.
• Data scale and quality significantly influence gains, with model size and dataset size often driving performance improvements; data cleaning is crucial.
• Prompting enables few-shot learning and better alignment with pre-training objectives, while instruction-driven or demonstration-based prompts improve task performance without heavy fine-tuning.
• Data generation via PLMs is a complementary approach for creating silver data or auxiliary context to support target tasks.
• Limitations include domain mismatch, scalability, and potential inaccuracies reflecting training data biases; future directions include improving efficiency, adaptability, and robustness.

Figure 2: Typical “pre-train then fine-tune” strategies. We illustrate strategies that fine-tune the full PLM (left), fine-tune the full PLM in a custom model (center), and fine-tune just a small adapter sub-layer per each Transformer layer (right). We show the Transformer blocks that will be fine-t