[Paper Review] Plug and Play Language Models: A Simple Approach to Controlled Text Generation

Sumanth Dathathri, Andrea Madotto+6 more|arXiv (Cornell University)|Dec 4, 2019

Speech and dialogue systems407 citations

TL;DR

PPLM combines a pretrained language model with lightweight attribute models to steer generation without retraining, using gradient updates in latent space to control topics and sentiment while preserving fluency.

ABSTRACT

Large transformer-based language models (LMs) trained on huge text corpora have shown unparalleled generation capabilities. However, controlling attributes of the generated language (e.g. switching topic or sentiment) is difficult without modifying the model architecture or fine-tuning on attribute-specific data and entailing the significant cost of retraining. We propose a simple alternative: the Plug and Play Language Model (PPLM) for controllable language generation, which combines a pretrained LM with one or more simple attribute classifiers that guide text generation without any further training of the LM. In the canonical scenario we present, the attribute models are simple classifiers consisting of a user-specified bag of words or a single learned layer with 100,000 times fewer parameters than the LM. Sampling entails a forward and backward pass in which gradients from the attribute model push the LM's hidden activations and thus guide the generation. Model samples demonstrate control over a range of topics and sentiment styles, and extensive automated and human annotated evaluations show attribute alignment and fluency. PPLMs are flexible in that any combination of differentiable attribute models may be used to steer text generation, which will allow for diverse and creative applications beyond the examples given in this paper.

Motivation & Objective

• Motivate the need for controllable text generation without retraining large LMs.
• Propose a flexible framework to combine a base LM with simple attribute models at inference.
• Demonstrate topic and sentiment control and evaluate attribute alignment and fluency.
• Compare PPLM to existing baselines and show potential for detoxification and constrained storytelling.

Proposed method

• Use a pretrained transformer-based LM p(x) (e.g., GPT-2 345M) without modifying parameters.
• Attach one or more differentiable attribute models p(a|x) (e.g., bag-of-words or a single-layer discriminator).
• Perform gradient-based updates in the LM latent space Ht to maximize log p(a|x) while keeping log p(x) via a KL-divergence and post-norm fusion constraints.
• Update latent representations ΔHt and resample from the updated distribution to generate tokens, with a controllable strength knob.
• Combine multiple attribute models and apply a post-hoc fusion with the original LM distribution to preserve fluency.
• Evaluate with both automated metrics (perplexity, Dist-1/2/3) and human judgments on fluency and attribute relevance.

Experimental results

Research questions

• RQ1 Can PPLM steer text generation toward predefined attributes (topics or sentiments) without retraining the LM?
• RQ2 How does latent-space manipulation compare to output-distribution weighting for attribute control?
• RQ3 Does combining KL regularization and post-norm fusion preserve fluency while enforcing attribute alignment?
• RQ4 Is PPLM effective across different attribute models (BoW, discriminators) and domains?

Key findings

• PPLM achieves attribute control over topics and sentiment while maintaining fluency comparable to the baseline LM.
• Latent-space manipulation (BC/BCR) yields substantially higher topic control than reranking or weighting methods alone.
• PPLM-Discrim with latent updates plus ranking (BCR) attains strong sentiment control with competitive fluency and outperforms several baselines.
• Compared to CTRL and GPT-2 finetuned for positivity, PPLM often matches or exceeds authors’ notion of attribute relevance and fluency.
• PPLM can be used for detoxification by following gradients from a toxicity detector, and supports structured story writing.
• Code for experiments is released, demonstrating practical accessibility and reproducibility.