[Paper Review] Text Generation with Exemplar-based Adaptive Decoding

Hao Peng, Ankur P. Parikh+3 more|arXiv (Cornell University)|Apr 9, 2019

Topic Modeling67 references4 citations

TL;DR

This paper proposes exemplar-based adaptive decoding, a method that customizes the decoder for each input by reparameterizing its parameters using retrieved training exemplars, improving control and fluency in abstractive text generation. The approach outperforms baselines on summarization and data-to-text generation, achieving state-of-the-art results with minimal computational overhead.

ABSTRACT

We propose a novel conditioned text generation model. It draws inspiration from traditional template-based text generation techniques, where the source provides the content (i.e., what to say), and the template influences how to say it. Building on the successful encoder-decoder paradigm, it first encodes the content representation from the given input text; to produce the output, it retrieves exemplar text from the training data as "soft templates," which are then used to construct an exemplar-specific decoder. We evaluate the proposed model on abstractive text summarization and data-to-text generation. Empirical results show that this model achieves strong performance and outperforms comparable baselines.

Motivation & Objective

• To address the lack of control and reliability in neural sequence-to-sequence text generation, which often produces generic or degenerate outputs.
• To combine the precision of template-based methods with the fluency of neural generation by using retrieved training exemplars as soft templates.
• To improve structural and stylistic consistency in generated text without increasing model complexity or computational cost.
• To develop a method that adapts the decoder dynamically per input using exemplar information, rather than treating exemplars as static input context.

Proposed method

• The model retrieves relevant training examples (exemplars) from the training set based on similarity to the input source.
• It reparameterizes the decoder parameters using a context-aware transformation network that conditions on the exemplar, creating an exemplar-specific decoder.
• The adaptive decoder is integrated into a standard encoder-decoder framework, replacing the standard decoder with a dynamically configured one.
• The reparameterization mechanism allows the decoder to inherit structural and stylistic patterns from the exemplar without requiring explicit slot-filling.
• The method supports end-to-end training and can be applied to any sequence-to-sequence model architecture.
• Exemplars are used not just for attention or copying, but to directly shape the decoder's behavior through parameter adaptation.

Experimental results

Research questions

• RQ1 Can exemplars be used to improve structural and stylistic control in neural text generation beyond simple attention or copying?
• RQ2 Does dynamically adapting the decoder based on exemplars lead to more accurate and fluent outputs compared to fixed decoders?
• RQ3 Can exemplar-informed decoding reduce the tendency toward generic or degenerate outputs in abstractive generation?
• RQ4 How does exemplar-based adaptive decoding compare to existing exemplar-augmented methods in terms of performance and efficiency?

Key findings

• The proposed model achieves state-of-the-art performance on three benchmark datasets for abstractive summarization and data-to-text generation.
• On the Gigaword dataset, the model outperforms strong baselines in ROUGE-4 and BLEU scores, demonstrating improved fluency and factual consistency.
• Qualitative analysis shows that the model effectively adopts structural patterns from exemplars, such as noun adjuncts or prepositional phrases, even when word overlap is minimal.
• The model is sensitive to exemplar style and tense, correctly following stylistic cues in exemplars while avoiding hallucinations when exemplars are consistent.
• When exemplars contain distractions (e.g., incorrect locations or numbers), the model sometimes inherits these errors, indicating dependency on exemplar quality.