[Paper Review] Pretrained Language Models for Text Generation: A Survey

Junyi Li, Tianyi Tang+3 more|arXiv (Cornell University)|Jan 14, 2022

Topic Modeling66 citations

TL;DR

A comprehensive survey of how pretrained language models (PLMs) are applied to text generation, detailing input encoding, model design, optimization, challenges, resources, and future directions.

ABSTRACT

Text Generation aims to produce plausible and readable text in a human language from input data. The resurgence of deep learning has greatly advanced this field, in particular, with the help of neural generation models based on pre-trained language models (PLMs). Text generation based on PLMs is viewed as a promising approach in both academia and industry. In this paper, we provide a survey on the utilization of PLMs in text generation. We begin with introducing three key aspects of applying PLMs to text generation: 1) how to encode the input into representations preserving input semantics which can be fused into PLMs; 2) how to design an effective PLM to serve as the generation model; and 3) how to effectively optimize PLMs given the reference text and to ensure that the generated texts satisfy special text properties. Then, we show the major challenges arisen in these aspects, as well as possible solutions for them. We also include a summary of various useful resources and typical text generation applications based on PLMs. Finally, we highlight the future research directions which will further improve these PLMs for text generation. This comprehensive survey is intended to help researchers interested in text generation problems to learn the core concepts, the main techniques and the latest developments in this area based on PLMs.

Motivation & Objective

• Introduce the task formulation of PLM-based text generation and the role of input data, model design, and optimization.
• Survey input representation strategies for unstructured, structured, and multimedia data.
• Review PLM architectures (masked LMs, causal LMs, prefix LMs, encoder-decoder) and design tweaks for generation tasks.
• Discuss optimization techniques and how to ensure properties like fidelity, relevance, and coherence in generated text.
• Highlight challenges, resources, applications, and future directions in PLM-based text generation.

Proposed method

• Frame text generation as an input-to-output mapping f_M(x, P) using PLMs.
• Categorize input data into unstructured, structured, and multimedia and review encoding approaches.
• Classify PLMs by pre-training objectives: masked LM, causal LM, prefix LM, encoder-decoder LM, and discuss architectural extensions.
• Review optimization strategies including fine-tuning, multi-task learning, and prompt/adapter-based approaches.
• Discuss representations for cross-lingual and multilingual generation, and methods to preserve structured data information during generation.
• Summarize resources, applications, and future research directions in PLM-based text generation.

Experimental results

Research questions

• RQ1 What input representations and encoding strategies best preserve input semantics for PLM-based generation?
• RQ2 How should PLMs be designed or adapted to serve as effective generation functions across tasks?
• RQ3 What optimization techniques reliably produce fluent, faithful, and task-aligned text?
• RQ4 What are the major challenges (e.g., fidelity, efficiency, multilinguality) and how can they be mitigated?
• RQ5 What resources, datasets, and applications illustrate PLM-based text generation in practice?

Key findings

• PLMs pre-trained on large corpora enable fluent and contextually aware text generation across tasks.
• Hierarchical and graph-based representations help capture long-range and cross-sentence semantics for paragraphs and documents.
• Structured data requires bridging the semantic gap through linearization, representation alignment, and specialized encoding modules.
• Faithfulness and fidelity to input structures can be enhanced with tailored objectives, copying mechanisms, and input-informed prompts.
• Multilingual and cross-lingual PLMs enable generation across languages with shared representations and language-specific adaptations.
• A wide range of resources and application areas exist, along with evolving optimization strategies such as fine-tuning variants and prompt-based methods.