[Paper Review] Language Models as Few-Shot Learner for Task-Oriented Dialogue Systems

Andrea Madotto, Zihan Liu+2 more|arXiv (Cornell University)|Aug 14, 2020

Topic Modeling22 references36 citations

TL;DR

The paper investigates how priming large language models (GPT-2 variants) with few-shot examples enables solving NLU, DST, Dialogue Policy, and NLG tasks in task-oriented dialogue without parameter updates, comparing with finetuning baselines and outlining limitations.

ABSTRACT

Task-oriented dialogue systems use four connected modules, namely, Natural Language Understanding (NLU), a Dialogue State Tracking (DST), Dialogue Policy (DP) and Natural Language Generation (NLG). A research challenge is to learn each module with the least amount of samples (i.e., few-shots) given the high cost related to the data collection. The most common and effective technique to solve this problem is transfer learning, where large language models, either pre-trained on text or task-specific data, are fine-tuned on the few samples. These methods require fine-tuning steps and a set of parameters for each task. Differently, language models, such as GPT-2 (Radford et al., 2019) and GPT-3 (Brown et al., 2020), allow few-shot learning by priming the model with few examples. In this paper, we evaluate the priming few-shot ability of language models in the NLU, DST, DP and NLG tasks. Importantly, we highlight the current limitations of this approach, and we discuss the possible implication for future work.

Motivation & Objective

• Motivate reducing data collection for modular task-oriented dialogue systems (NLU, DST, DP, NLG).
• Evaluate language-model priming as a no-finetuning few-shot approach across core tasks.
• Compare LM-primed few-shot results with finetuning-based baselines under limited data.
• Identify practical limitations and suggest directions for longer-context models and future work.

Proposed method

• Use three prefix styles (binary, value-based, generative) to prime LMs for few-shot learning without updating parameters.
• Apply prefixes to map inputs to outputs for NLU (slot filling and intent), DST, ACT, and NLG tasks.
• Evaluate across standard datasets (SNIPS for NLU slot filling and intent; MultiWOZ for DST and ACT; FewShotWOZ for NLG).
• Compare LM-primed few-shot results against selected finetuning baselines (e.g., TOD-BERT, BERT, SC-GPT variants).
• Experiment with different GPT-2 model sizes (SMALL, LARGE, XL) under context window constraints.

Experimental results

Research questions

• RQ1 Can LM priming enable competitive few-shot performance on NLU, DST, ACT, and NLG compared to finetuned baselines?
• RQ2 How does model size influence few-shot performance in each task?
• RQ3 What are the practical limitations of LM priming in terms of prefix design, shot count, and input length?
• RQ4 What future improvements could enhance few-shot capability for task-oriented dialogue systems?

Key findings

• LM priming with larger GPT-2 models generally yields better performance on NLU and NLG tasks.
• For DST and ACT, the larger XL model does not consistently outperform the LARGE model, suggesting prefix design or context effects matter.
• In NLU, ACT, and NLG, LM priming can achieve similar or better results than the weakest finetuning baselines under limited shots.
• Two main limitations are identified: (i) binary/value-based prefixes require multiple forward passes per class/slot, and (ii) GPT-2’s 1024-token input limit restricts the number of shots.