[Paper Review] DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialogue

Shikib Mehri, Mihail Eric+1 more|arXiv (Cornell University)|Sep 28, 2020

Topic Modeling27 references97 citations

TL;DR

DialoGLUE introduces a public benchmark of 7 task-oriented dialogue datasets across 4 NLU tasks, with strong baselines showing improvements via ConvBERT and task-adaptive training.

ABSTRACT

A long-standing goal of task-oriented dialogue research is the ability to flexibly adapt dialogue models to new domains. To progress research in this direction, we introduce DialoGLUE (Dialogue Language Understanding Evaluation), a public benchmark consisting of 7 task-oriented dialogue datasets covering 4 distinct natural language understanding tasks, designed to encourage dialogue research in representation-based transfer, domain adaptation, and sample-efficient task learning. We release several strong baseline models, demonstrating performance improvements over a vanilla BERT architecture and state-of-the-art results on 5 out of 7 tasks, by pre-training on a large open-domain dialogue corpus and task-adaptive self-supervised training. Through the DialoGLUE benchmark, the baseline methods, and our evaluation scripts, we hope to facilitate progress towards the goal of developing more general task-oriented dialogue models.

Motivation & Objective

• Motivate and enable generalization of task-oriented dialogue models across multiple domains and datasets.
• Provide a standardized evaluation benchmark and publicly available baselines and leaderboard resources.
• Demonstrate the efficacy of open-domain dialogue pre-training and task-adaptive self-supervised training for transfer to task-oriented tasks.

Proposed method

• Construct a public benchmark (DialoGLUE) of 7 datasets across 4 NLU tasks in task-oriented dialogue.
• Implement architectures built on a BERT-like encoder for intent prediction, slot filling, semantic parsing, and dialogue state tracking.
• Create ConvBERT by fine-tuning BERT on a large open-domain dialogue corpus (≈700M conversations).
• Apply task-adaptive training with masked language modeling on target datasets, including pre-training and multi-tasking.
• Evaluate four model variants (BERT, ConvBERT, BERT-DG, ConvBERT-DG) under four setups (direct fine-tuning, MLM pre-training, MLM multi-tasking, and both).
• Provide public evaluation scripts and a leaderboard on EvalAI, with open-sourced code and baselines.

Experimental results

Research questions

• RQ1 Can a standardized, multi-task benchmark like DialoGLUE drive progress toward generic, transfer-friendly task-oriented dialogue models?
• RQ2 Do large-scale open-domain dialogue pre-training and task-adaptive self-supervised training improve performance across diverse dialogue understanding tasks?
• RQ3 How do different pre-training and fine-tuning strategies (pre-training, multi-tasking, or both) affect performance on intent, slot filling, semantic parsing, and dialogue state tracking?
• RQ4 Is there measurable cross-task transfer when training on multiple DialoGLUE datasets?
• RQ5 What is the impact of few-shot data settings on the effectiveness of self-supervised and transfer learning approaches?

Key findings

• DialoGLUE provides a challenging, standardized benchmark across 7 datasets and 4 tasks with publicly available baselines.
• ConvBERT with task-adaptive training matches or exceeds state-of-the-art results on 5 of 7 tasks and yields a notable +2.98 joint-goal accuracy on MultiWOZ compared to prior best models.
• Task-adaptive self-supervised training combined with open-domain dialogue pre-training yields aggregate gains across the DialoGLUE tasks, though gains are task-dependent.
• Self-supervised training on a broad DialoGLUE data mix can help in few-shot settings, with notable improvements on MultiWOZ and dialogue-state-tracking tasks.
• In some tasks (TOP and DSTC8), specialized architectures outperform general encoders, indicating room for task-specific adaptations.
• Overall, the combination of ConvBERT and task-adaptive training provides the strongest aggregate performance across the benchmark.