[Paper Review] Few-Shot Text Generation with Pattern-Exploiting Training

Timo Schick, Hinrich Schütze|arXiv (Cornell University)|Dec 22, 2020

Topic Modeling45 references74 citations

TL;DR

The paper introduces genPet, a pattern-exploiting training method that enables data-efficient fine-tuning of generative language models for text generation by using natural language instructions plus a small number of examples. It shows improvements over standard fine-tuning on several summarization and headline generation tasks in zero- and few-shot settings.

ABSTRACT

Providing pretrained language models with simple task descriptions in natural language enables them to solve some tasks in a fully unsupervised fashion. Moreover, when combined with regular learning from examples, this idea yields impressive few-shot results for a wide range of text classification tasks. It is also a promising direction to improve data efficiency in generative settings, but there are several challenges to using a combination of task descriptions and example-based learning for text generation. In particular, it is crucial to find task descriptions that are easy to understand for the pretrained model and to ensure that it actually makes good use of them; furthermore, effective measures against overfitting have to be implemented. In this paper, we show how these challenges can be tackled: We introduce GenPET, a method for text generation that is based on pattern-exploiting training, a recent approach for combining textual instructions with supervised learning that only works for classification tasks. On several summarization and headline generation datasets, GenPET gives consistent improvements over strong baselines in few-shot settings.

Motivation & Objective

• Motivate and enable data-efficient generation by guiding pretrained generators with natural language instructions.
• Extend the pattern-exploiting training paradigm from classification to generation tasks.
• Evaluate genPet on diverse English summarization and headline-generation tasks in zero-shot and few-shot regimes.
• Analyze components contributing to genPet’s performance and identify factors driving improvements.

Proposed method

• Introduce genPet, a finetuning procedure for generative language models that uses textual instructions via patterns to steer generation.
• Adapt the Pet framework to encoder-decoder models by applying a single or split pattern P to x, enabling the model to produce y conditioned on P(x).
• Use a pattern P and, when appropriate, selectively process parts of P(x) with encoder vs. decoder to influence how instructions affect generation.
• Formulate the probability p(y|x) via the model’s token probabilities at the verbalized instruction positions within P(x).
• Address three challenges: optimal instruction integration, instruction understanding and robustness to pattern changes, and overfitting mitigation in few-shot settings.
• Describe the encoder-decoder generation mechanics and the decomposition p(y|z) into token-wise conditionals for generation.

Experimental results

Research questions

• RQ1 Can genPet enable data-efficient fine-tuning for text generation tasks using instructions and a small labeled set?
• RQ2 How should instructions be integrated into encoder-decoder models to maximize their effect on generated outputs?
• RQ3 How can we ensure the model understands instructions and remains robust to pattern variations?
• RQ4 What strategies mitigate overfitting in few-shot generation with instruction-based prompts?

Key findings

• genPet enables significant data-efficient fine-tuning of Pegasus for summarization and headline generation with as few as 10 or 100 training examples.
• GenPet consistently outperforms regular finetuning on multiple summarization and headline-generation tasks in few-shot settings.
• Processing of instructions by the decoder tends to have a stronger impact on generation than encoding-only processing.
• The paper analyzes factors contributing to genPet’s performance and quantifies the impact of its components.