[Paper Review] Simple, Fast, Accurate Intent Classification and Slot Labeling.

Arshit Gupta, John Hewitt+1 more|arXiv (Cornell University)|Mar 19, 2019

Speech and dialogue systems6 citations

TL;DR

This paper proposes a family of simple, non-recurrent neural architectures for joint intent classification and slot labeling, leveraging label-recurrent, dilated convolutional networks to achieve state-of-the-art accuracy—30% error reduction in slot labeling on the Snips dataset—while being twice as fast at inference and requiring 30–50% less training time than comparable recurrent models.

ABSTRACT

In real-time dialogue systems running at scale, there is a tradeoff between system performance, time taken for training to converge, and time taken to perform inference. In this work, we study modeling tradeoffs intent classification (IC) and slot labeling (SL), focusing on non-recurrent models. We propose a simple, modular family of neural architectures for joint IC+SL. Using this framework, we explore a number of self-attention, convolutional, and recurrent models, contributing a large-scale analysis of modeling paradigms for IC+SL across two datasets. At the same time, we discuss a class of 'label-recurrent' models, proposing that otherwise non-recurrent models with a 10-dimensional representation of the label history provide multi-point SL improvements. As a result of our analysis, we propose a class of label-recurrent, dilated, convolutional IC+SL systems that are accurate, achieving a 30% error reduction in SL over the state-of-the-art performance on the Snips dataset, as well as fast, at 2x the inference and 2/3 to 1/2 the training time of comparable recurrent models.

Motivation & Objective

• To address the tradeoff between model accuracy, training speed, and inference latency in large-scale real-time dialogue systems.
• To explore non-recurrent architectures for joint intent classification and slot labeling, focusing on efficiency and performance.
• To investigate the impact of incorporating label history representations in otherwise non-recurrent models for improved slot labeling.
• To develop a modular, scalable framework for comparing self-attention, convolutional, and recurrent modeling paradigms in IC+SL.
• To achieve state-of-the-art performance in slot labeling while significantly reducing inference and training time.

Proposed method

• Proposes a modular, non-recurrent neural architecture family for joint intent classification and slot labeling using dilated convolutions and self-attention mechanisms.
• Introduces a 'label-recurrent' mechanism that maintains a 10-dimensional representation of previous slot labels to improve multi-point slot labeling.
• Employs a joint training objective for intent classification and slot labeling within a unified encoder-decoder framework.
• Uses dilated convolutions to capture long-range dependencies without increasing model depth or computational cost.
• Applies residual connections and layer normalization to stabilize training and improve convergence speed.
• Evaluates multiple modeling paradigms—including self-attention, convolutional, and recurrent—on two benchmark datasets.

Experimental results

Research questions

• RQ1 How do non-recurrent models compare to recurrent models in terms of accuracy, training time, and inference speed for joint intent classification and slot labeling?
• RQ2 Can a 10-dimensional label history representation improve slot labeling performance in otherwise non-recurrent models?
• RQ3 What is the impact of dilated convolutions and self-attention on slot labeling accuracy and inference efficiency?
• RQ4 How does the proposed label-recurrent mechanism affect multi-point slot labeling performance?
• RQ5 Can a unified, modular architecture achieve state-of-the-art performance while maintaining low latency and fast training?

Key findings

• The proposed label-recurrent, dilated convolutional model achieves a 30% error reduction in slot labeling compared to the state of the art on the Snips dataset.
• Inference speed is twice as fast as comparable recurrent models, making it suitable for real-time deployment.
• Training time is reduced by 30–50% compared to recurrent baselines, improving model iteration speed.
• The model maintains high accuracy while eliminating the need for recurrent units, simplifying the architecture.
• The label-recurrent mechanism significantly improves multi-point slot labeling without increasing model complexity.
• The framework enables efficient comparison of self-attention, convolutional, and recurrent paradigms across two datasets.