[Paper Review] Neural Text Generation with Unlikelihood Training

Sean Welleck, Ilia Kulikov+4 more|arXiv (Cornell University)|Aug 12, 2019

Topic ModelingComputer Science31 references241 citations

TL;DR

The paper introduces unlikelihood training to reduce degeneration in neural text generation, improving repetition and token distribution while maintaining perplexity, and demonstrates superiority over standard likelihood training across decoding strategies including beam search.

ABSTRACT

Neural text generation is a key tool in natural language applications, but it is well known there are major problems at its core. In particular, standard likelihood training and decoding leads to dull and repetitive outputs. While some post-hoc fixes have been proposed, in particular top-$k$ and nucleus sampling, they do not address the fact that the token-level probabilities predicted by the model are poor. In this paper we show that the likelihood objective itself is at fault, resulting in a model that assigns too much probability to sequences containing repeats and frequent words, unlike those from the human training distribution. We propose a new objective, unlikelihood training, which forces unlikely generations to be assigned lower probability by the model. We show that both token and sequence level unlikelihood training give less repetitive, less dull text while maintaining perplexity, giving superior generations using standard greedy or beam search. According to human evaluations, our approach with standard beam search also outperforms the currently popular decoding methods of nucleus sampling or beam blocking, thus providing a strong alternative to existing techniques.

Motivation & Objective

• Motivate and analyze degeneration in neural text generation trained with maximum likelihood.
• Propose a training objective (unlikelihood) to penalize unlikely or repetitive tokens.
• Develop token-level and sequence-level unlikelihood losses to improve generation quality.
• Demonstrate improvements in repetition, token distribution, and human judgments across decoding methods.
• Show applicability to fine-tuning large models like GPT-2 with practical training updates.

Proposed method

• Define unlikelihood loss that penalizes negative token candidates during next-token prediction.
• Combine likelihood update with unlikelihood update at token level (L_UL-token).
• Use previous-context tokens as negative candidates to discourage repetitions and overuse of frequent tokens.
• Introduce sequence-level unlikelihood (L_ULS) by applying penalties on model-generated continuations to address train–test distribution mismatch.
• Fine-tune with a mix of L_UL-token and L_UL-seq losses for efficient improvements.
• Demonstrate efficiency: sequence-level fine-tuning can be effective within about 1,500 updates.

Experimental results

Research questions

• RQ1 Can unlikelihood training reduce repetition and dullness in generated text compared to maximum likelihood training?
• RQ2 Does unlikelihood training preserve perplexity and token accuracy while improving generation quality?
• RQ3 Do token-level and sequence-level unlikelihood objectives offer complementary benefits, and can they outperform existing decoding fixes (top-k, nucleus, beam blocking) in human judgments?
• RQ4 Is unlikelihood training effective when applied to large Transformer-based language models and pre-trained systems (e.g., GPT-2) through fine-tuning?
• RQ5 How do different decoding strategies interact with unlikelihood-trained models in terms of generation quality?

Key findings

• Unlikelihood training reduces both token-level repetition and sequence-level repetition compared toMLE baselines.
• Token-level unlikelihood increases the number of unique next tokens and lowers repetition (wrep) without significant loss in perplexity or accuracy.
• Sequence-level unlikelihood further reduces repetition dramatically (e.g., seq-rep-4 substantially lower) and yields more unique token continuations.
• Fine-tuning with sequence-level unlikelihood (or combined token+sequence) significantly outperforms likelihood-trained models using nucleus sampling or beam blocking in human evaluations.
• The approach is effective across deterministic (greedy/beam) and stochastic decoding, and can improve existing pre-trained models with relatively few updates (≈1,500).
• GPT-2 fine-tuning with unlikelihood objectives yields comparable gains in repetition reduction and maintains language modeling quality.