[Paper Review] Exploring Prompt-based Few-shot Learning for Grounded Dialog Generation

Chujie Zheng, Minlie Huang|arXiv (Cornell University)|Sep 14, 2021

Topic Modeling32 references27 citations

TL;DR

The paper proposes a prompt-based few-shot framework for grounded dialog generation (GDG) and shows prompted language models outperform conversational models across three GDG tasks, with discrete prompts offering robustness and strong grounding performance.

ABSTRACT

Dialog models can be greatly strengthened through grounding on various external information, but grounded dialog corpora are usually not naturally accessible. In this work, we focus on the few-shot learning for grounded dialog generation (GDG). We first propose a simple prompting method for GDG tasks, where different constructs of model input, such as the grounding source and the conversation context, are distinguished through continuous or discrete prompts. On three typical GDG tasks, we empirically demonstrate and analyze in-depth the effectiveness of our method. We then conduct extensive experiments to thoroughly investigate how our prompting method works with different pre-trained models. We show that prompted language models perform superiorly to conversational models, and further analyze various factors that influence the effects of prompting. Overall, our work introduces a prompt-based perspective to the few-shot learning for GDG tasks, and provides valuable findings and insights for future research.

Motivation & Objective

• Address the data scarcity problem in grounded dialog generation by leveraging few-shot learning.
• Propose a simple prompting method to distinguish complex input constructs (grounding source vs. conversation context) in GDG tasks.
• Evaluate how prompting interacts with a range of pre-trained models across multiple GDG datasets.

Proposed method

• Introduce continuous (ind indicative tokens) and discrete (textual descriptions with task instructions) prompts to separate grounding source and conversation context in inputs.
• Use GPT2-medium as backbone to test prompting effectiveness on three GDG tasks (Wizard-of-Wikipedia, PersonaChat, ESConv).
• Train all model parameters with cross-entropy on target responses; employ beam search and nucleus sampling during inference depending on model task.
• Compare prompting against No Prompts and analyze sensitivity to prompt initialization and perturbations.
• Assess prompts with multiple backbones (GPT2, T5, BART) to study cross-model applicability.
• Report evaluation via perplexity, BLEU-2, unigram F1, and grounding metrics (Wiki/PSN F1 or Match Ratio).

Experimental results

Research questions

• RQ1 Does prompt-based few-shot learning improve GDG performance when grounding sources are present?
• RQ2 How do continuous versus discrete prompts compare in few-shot and full-data regimes for GDG?
• RQ3 How do prompting methods interact with different pre-trained model architectures (autoregressive vs. encoder-decoder) and sizes?
• RQ4 Are prompted language models better at grounding than conversational models across GDG tasks?

Key findings

• Distinguishing input constructs with prompts boosts GDG few-shot performance, but poorly initialized prompts can hurt learning.
• Discrete prompts generally outperform continuous prompts under both few-shot and full-data settings, showing robustness to minor prompt changes.
• Prompted language models outperform conversational models on the three GDG tasks in many settings.
• Prompting effectiveness depends on backbone model size and pre-training objectives; language models like GPT2 and T5-Large benefit more from prompting than BART in some tasks.
• Prompted language models like GPT2 and T5 can better utilize grounding information than unprompted counterparts, while conversational models often do not benefit from prompting.