[Paper Review] Few-shot Natural Language Generation for Task-Oriented Dialog

Baolin Peng, Chenguang Zhu+5 more|arXiv (Cornell University)|Feb 27, 2020

Topic Modeling27 references31 citations

TL;DR

The paper introduces FewShotWOZ, a few-shot NLG benchmark for task-oriented Dialog, and SC-GPT, a semantically conditioned GPT-2-based model that excels at generating domain-specific, fluent responses with limited labeled data.

ABSTRACT

As a crucial component in task-oriented dialog systems, the Natural Language Generation (NLG) module converts a dialog act represented in a semantic form into a response in natural language. The success of traditional template-based or statistical models typically relies on heavily annotated data, which is infeasible for new domains. Therefore, it is pivotal for an NLG system to generalize well with limited labelled data in real applications. To this end, we present FewShotWoz, the first NLG benchmark to simulate the few-shot learning setting in task-oriented dialog systems. Further, we develop the SC-GPT model. It is pre-trained on a large set of annotated NLG corpus to acquire the controllable generation ability, and fine-tuned with only a few domain-specific labels to adapt to new domains. Experiments on FewShotWoz and the large Multi-Domain-WOZ datasets show that the proposed SC-GPT significantly outperforms existing methods, measured by various automatic metrics and human evaluations.

Motivation & Objective

• Motivate NLG in task-oriented dialog to perform well with scarce labeled data in new domains.
• Propose FewShotWOZ to simulate few-shot adaptation across multiple domains.
• Develop SC-GPT, a semantically conditioned pre-trained model for controllable, fluent NLG.
• Demonstrate that SC-GPT outperforms baselines in both automatic metrics and human evaluation on FewShotWOZ and MultiWOZ.

Proposed method

• Use a three-stage training recipe: (i) large-scale plain-text pre-training (GPT-2 backbone); (ii) dialog-act conditioned pre-training on annotated dialog acts and responses; (iii) domain-specific fine-tuning with limited domain labels.
• Represent dialog acts as sequential control codes accompanying the target utterance during pre-training and fine-tuning.
• Employ an auto-regressive Transformer to model p_theta(x|A) with generation conditioned on the dialog act A.
• Fine-tune SC-GPT on new domains with only a few dozens of domain-specific labels to adapt to novel acts and slots.
• Preserve flexibility by avoiding delexicalization and enabling controllable generation grounded in semantic acts.

Experimental results

Research questions

• RQ1 Can SC-GPT generalize well to new domains with minimal in-domain labeled data?
• RQ2 Does pre-training on large dialog-act annotated corpora improve controllability and fluency over baselines in few-shot NLG?
• RQ3 How does FewShotWOZ reflect few-shot adaptation challenges in real-world task-oriented dialog systems?

Key findings

• SC-GPT achieves the best overall performance in FewShotWOZ and MultiWOZ on automatic metrics and human evaluation.
• SC-GPT significantly reduces slot error rate (ERR) and improves BLEU compared to SC-LSTM and GPT-2 in few-shot settings.
• Human evaluation shows SC-GPT outperforms baselines in informativeness and naturalness across domains.
• SC-GPT generalizes to unseen dialog acts better than baselines, with notably lower ERR on unseen acts.
• In MultiWOZ, SC-GPT yields the best BLEU while GPT-2 is competitive when training data is abundant.
• The FewShotWOZ benchmark reveals strong generalization and controllability benefits when using SC-GPT.