[Paper Review] A Simple Language Model for Task-Oriented Dialogue

Ehsan Hosseini-Asl, Bryan McCann+3 more|arXiv (Cornell University)|May 2, 2020

Topic ModelingComputer Science62 references159 citations

TL;DR

SimpleTOD uses a single causal language model to handle all task-oriented dialogue sub-tasks end-to-end, achieving state-of-the-art results on MultiWOZ for dialogue state tracking and end-to-end metrics.

ABSTRACT

Task-oriented dialogue is often decomposed into three tasks: understanding user input, deciding actions, and generating a response. While such decomposition might suggest a dedicated model for each sub-task, we find a simple, unified approach leads to state-of-the-art performance on the MultiWOZ dataset. SimpleTOD is a simple approach to task-oriented dialogue that uses a single, causal language model trained on all sub-tasks recast as a single sequence prediction problem. This allows SimpleTOD to fully leverage transfer learning from pre-trained, open domain, causal language models such as GPT-2. SimpleTOD improves over the prior state-of-the-art in joint goal accuracy for dialogue state tracking, and our analysis reveals robustness to noisy annotations in this setting. SimpleTOD also improves the main metrics used to evaluate action decisions and response generation in an end-to-end setting: inform rate by 8.1 points, success rate by 9.7 points, and combined score by 7.2 points.

Motivation & Objective

• Reframe task-oriented dialogue as a single sequence prediction problem.
• Leverage pre-trained open-domain causal language models (e.g., GPT-2) for TOD.
• Reduce error propagation by training a unified model end-to-end across sub-tasks.
• Demonstrate robustness to noisy annotations and provide code/data for replication.
• Provide analysis of token design and pre-training effects on TOD performance.

Proposed method

• Train a single Transformer-based causal language model on a concatenated TOD sequence x^t = [C_t; B_t; D_t; A_t; S_t].
• Represent dialogue context, belief state, database results, actions, and delexicalized response as a single generation task.
• Initialize from pre-trained weights (DistilGPT2/GPT-2) and tokenize with pretrained BPE; truncate sequences >1024 tokens.
• Use special tokens to delineate user/system segments and end-of-segment markers to guide generation.
• Evaluate in an end-to-end setting on MultiWOZ 2.0/2.1, reporting joint DST accuracy and end-to-end metrics (Inform, Success, BLEU, Combined).
• Demonstrate that a unidirectional decoder with minimal supervision can outperform prior modular/state-tracking models.

Experimental results

Research questions

• RQ1 Can task-oriented dialogue be effectively solved with a single causal language model rather than a modular pipeline?
• RQ2 How do pre-training and token-segmentation choices affect end-to-end TOD performance on MultiWOZ?
• RQ3 What is the impact of including or excluding database search results during training and inference?
• RQ4 How robust is end-to-end TOD to noisy annotations in practical datasets?

Key findings

• SimpleTOD achieves state-of-the-art joint goal accuracy for dialogue state tracking on MultiWOZ 2.1 (55.76 without test-cleaning; 57.47 with cleaning).
• In end-to-end evaluation, SimpleTOD surpasses prior work on inform rate, success rate, and combined score (e.g., inform 84.4, success 70.1, BLEU 15.01, combined 92.26 with no DB inputs).
• Using oracle DB search or dynamic DB search can yield higher per-metric scores but the best end-to-end performance occurs without DB search guidance.
• End-to-end, single-model TOD can outperform specialized, multi-component baselines without additional supervision.
• Ablations show the importance of end-of-segment tokens and pre-training; larger SimpleTOD models are not always better for MultiWOZ end-to-end performance.