[Paper Review] DIET: Lightweight Language Understanding for Dialogue Systems

Tanja Bunk, Daksh Varshneya+2 more|arXiv (Cornell University)|Apr 21, 2020

Topic ModelingComputer Science33 references113 citations

TL;DR

DIET introduces a multi-task Dual Intent and Entity Transformer architecture for joint intent classification and entity recognition in dialogue systems, showing strong results even without pre-trained embeddings and offering faster training than large models like BERT.

ABSTRACT

Large-scale pre-trained language models have shown impressive results on language understanding benchmarks like GLUE and SuperGLUE, improving considerably over other pre-training methods like distributed representations (GloVe) and purely supervised approaches. We introduce the Dual Intent and Entity Transformer (DIET) architecture, and study the effectiveness of different pre-trained representations on intent and entity prediction, two common dialogue language understanding tasks. DIET advances the state of the art on a complex multi-domain NLU dataset and achieves similarly high performance on other simpler datasets. Surprisingly, we show that there is no clear benefit to using large pre-trained models for this task, and in fact DIET improves upon the current state of the art even in a purely supervised setup without any pre-trained embeddings. Our best performing model outperforms fine-tuning BERT and is about six times faster to train.

Motivation & Objective

• Motivate the need for fast, multilingual, and trainable NLU for dialogue systems in real-world software ecosystems.
• Propose a modular multi-task architecture that jointly handles intent classification and entity recognition.
• Explore the impact of sparse (one-hot, character n-grams) and dense (pre-trained embeddings) features in DIET.
• Investigate the benefits of incorporating masked reconstruction objectives as regularizers.
• Evaluate DIET on multi-domain NLU benchmarks and compare against state-of-the-art baselines.

Proposed method

• DIET featurizes input as a sequence of tokens augmented with sparse features (token-level one-hot, character n-grams up to length 5) and dense features from pre-trained embeddings (ConveRT, BERT, GloVe).
• A two-layer Transformer with relative position attention encodes context, operating on the concatenated dense+sparse features.
• A CRF layer on top of the Transformer outputs performs named entity recognition.
• Intent classification uses a dot-product loss over a semantic space for the CLS representation of the sequence and intent labels, with negative sampling for ranking.
• A masked-token reconstruction objective is added to the Transformer outputs to regularize and learn general features.
• Total loss is a weighted sum of intent loss, entity (CRF) loss, and masked reconstruction loss, enabling flexible ablation.

Experimental results

Research questions

• RQ1 Can DIET jointly model intent classification and entity recognition effectively in a multi-domain setting?
• RQ2 What is the impact of combining sparse features with various pre-trained dense embeddings on NLU performance?
• RQ3 Does a masked reconstruction objective improve DIET’s generalization and accuracy?
• RQ4 How does DIET compare to state-of-the-art approaches like HERMIT and fine-tuned BERT across standard NLU benchmarks?
• RQ5 Is a purely supervised DIET model competitive with models leveraging large pre-trained language models, and how fast is training?

Key findings

• On the challenging NLU-Benchmark, DIET with sparse features plus ConveRT embeddings achieves strong intent and entity F1 scores, outperforming the HERMIT baseline on intents and achieving higher entity recall.
• A model using sparse features with ConveRT (no mask loss) yields top performance for intents and competitive results for entities, surpassing state-of-the-art by about 3 percentage points in F1 on both tasks.
• In ablations, using only sparse features with a mask loss improves both intents and entities by about 1 percentage point; GloVe embeddings with sparse features are competitive, and BERT embeddings without task-specific fine-tuning may underperform compared to ConveRT or GloVe in this setup.
• DIET with frozen ConveRT embeddings and sparse features outperforms fine-tuned BERT in entity recognition, while matching intent accuracy, and is significantly faster to train (10 hours vs. 60 hours on NLU-Benchmark).
• On ATIS and SNIPS, DIET with sparse features and ConveRT or GloVe achieves competitive results close to Joint BERT, even with no fine-tuning of embeddings.