[Paper Review] Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing

Pengfei Liu, Weizhe Yuan+4 more|arXiv (Cornell University)|Jul 28, 2021

Topic ModelingComputer Science513 citations

TL;DR

A comprehensive survey organizing prompting-based learning in NLP, detailing prompting basics, model families, design strategies, training approaches, applications, challenges, and meta-analysis, with resources to aid newcomers.

ABSTRACT

This paper surveys and organizes research works in a new paradigm in natural language processing, which we dub "prompt-based learning". Unlike traditional supervised learning, which trains a model to take in an input x and predict an output y as P(y|x), prompt-based learning is based on language models that model the probability of text directly. To use these models to perform prediction tasks, the original input x is modified using a template into a textual string prompt x' that has some unfilled slots, and then the language model is used to probabilistically fill the unfilled information to obtain a final string x, from which the final output y can be derived. This framework is powerful and attractive for a number of reasons: it allows the language model to be pre-trained on massive amounts of raw text, and by defining a new prompting function the model is able to perform few-shot or even zero-shot learning, adapting to new scenarios with few or no labeled data. In this paper we introduce the basics of this promising paradigm, describe a unified set of mathematical notations that can cover a wide variety of existing work, and organize existing work along several dimensions, e.g.the choice of pre-trained models, prompts, and tuning strategies. To make the field more accessible to interested beginners, we not only make a systematic review of existing works and a highly structured typology of prompt-based concepts, but also release other resources, e.g., a website http://pretrain.nlpedia.ai/ including constantly-updated survey, and paperlist.

Motivation & Objective

• Introduce the prompting-based learning paradigm as a shift from traditional supervised learning.
• Provide a formal, unified description of prompting concepts and notation that covers diverse works.
• Organize existing work along dimensions such as pre-trained models, prompts, and tuning strategies.
• Survey training strategies, applications, and prompt-relevant challenges to guide future research.
• Offer curated resources and a structured typology to help newcomers enter prompt-based NLP research.

Proposed method

• Present a formal description of prompting as a three-step process: prompt addition, answer search, and answer mapping.
• Define a unified notation that encompasses cloze and prefix prompts, and Z-valued answer spaces.
• Classify prompting methods by prompt shape (cloze vs. prefix), manual vs. automated prompt learning, and discrete vs. continuous prompts.
• Survey pre-trained language model families by training objective, noising, and directionality to assess suitability for prompting.
• Systematically review design considerations (prompt engineering, answer engineering, multi-prompt learning) and training strategies (promptless fine-tuning, tuning-free prompting, prompt/LM tuning).
• Summarize a broad spectrum of applications (classification, information extraction, QA, generation, multimodal) and link them to prompting choices.
• Provide meta-analytic insights and timelines to highlight trends and transferability across the field.
• Offer resources such as NLPedia–Pretrain and related typologies to facilitate entry for beginners.

Experimental results

Research questions

• RQ1 What are the core concepts and mathematical formalisms that unify prompting-based NLP methods?
• RQ2 How do prompting strategies interact with different pre-trained language models and objectives across tasks?
• RQ3 What are the main design choices (prompts, answer spaces, training strategies) that influence prompting performance?
• RQ4 What are the current applications, challenges, and research directions in prompt-based NLP?
• RQ5 What resources can support researchers and beginners entering the prompting paradigm?

Key findings

• Prompt-based learning enables few-shot or zero-shot adaptation by reformulating tasks as prompting problems for pre-trained LMs.
• Two major shifts underpin prompting: (i) leveraging massive unsupervised pre-training, and (ii) reframing downstream tasks via prompts to elicit model behavior without task-specific supervision.
• Prompts can be cloze or prefix shapes, and can be manual or automatically learned (discrete vs. continuous prompts).
• Different pre-trained LM families (left-to-right, masked, prefix/encoder-decoder) align with different prompt designs and downstream tasks.
• Multi-prompt and prompt-composition techniques further enhance robustness and coverage across tasks.
• The survey outlines training strategies (promptless fine-tuning, tuning-free prompting, prompt/LM tuning) and connects prompting methods to a wide range of NLP tasks and evaluation challenges.