[Paper Review] A Simple but Effective BERT Model for Dialog State Tracking on Resource-Limited Systems

Tuan Lai, Quan Hung Tran+2 more|arXiv (Cornell University)|Oct 28, 2019

Topic Modeling24 references4 citations

TL;DR

This paper proposes a simple, parameter-efficient BERT-based model for dialog state tracking (DST) that treats each slot-value pair as a sequence input, enabling zero-shot generalization to new ontology entries. By applying knowledge distillation, the model achieves state-of-the-art performance on the WoZ 2.0 dataset while being 8x smaller and 7x faster than the original BERT model, with minimal accuracy drop.

ABSTRACT

In a task-oriented dialog system, the goal of dialog state tracking (DST) is to monitor the state of the conversation from the dialog history. Recently, many deep learning based methods have been proposed for the task. Despite their impressive performance, current neural architectures for DST are typically heavily-engineered and conceptually complex, making it difficult to implement, debug, and maintain them in a production setting. In this work, we propose a simple but effective DST model based on BERT. In addition to its simplicity, our approach also has a number of other advantages: (a) the number of parameters does not grow with the ontology size (b) the model can operate in situations where the domain ontology may change dynamically. Experimental results demonstrate that our BERT-based model outperforms previous methods by a large margin, achieving new state-of-the-art results on the standard WoZ 2.0 dataset. Finally, to make the model small and fast enough for resource-restricted systems, we apply the knowledge distillation method to compress our model. The final compressed model achieves comparable results with the original model while being 8x smaller and 7x faster.

Motivation & Objective

• To address the complexity and scalability issues of existing neural DST models, which often require separate models per slot type and are hard to maintain in production.
• To enable dynamic adaptation to changing domain ontologies without retraining, by decoupling slot-value prediction from fixed ontology constraints.
• To develop a lightweight, efficient DST model suitable for resource-limited systems such as mobile devices.
• To demonstrate that a simple fine-tuned BERT model can outperform complex, heavily engineered prior approaches on standard DST benchmarks.

Proposed method

• The model uses a standard BERT base encoder to encode a concatenation of the dialog context and a candidate slot-value pair as a single input sequence.
• Each candidate slot-value pair is processed independently via BERT, with the [CLS] token representation used to predict the relevance score of the pair.
• The model shares parameters across all slot types, so the number of parameters does not scale with ontology size.
• Knowledge distillation is applied to compress the full BERT model into a smaller student model with fewer layers and hidden dimensions.
• The distilled model is fine-tuned using soft labels from the full model, preserving performance while reducing size and inference time.
• The final compressed model is evaluated on CPU and GPU to demonstrate efficiency gains in real-world deployment scenarios.

Experimental results

Research questions

• RQ1 Can a simple, unified BERT-based architecture outperform complex, slot-specific DST models on standard benchmarks like WoZ 2.0?
• RQ2 Does a BERT-based DST model trained in a sequence-to-sequence manner generalize to unseen slot-value pairs without retraining?
• RQ3 Can knowledge distillation effectively compress a large BERT model for DST while maintaining high accuracy and significantly improving inference speed and model size?
• RQ4 How does the performance of the distilled model compare to other state-of-the-art models in terms of accuracy, size, and speed?

Key findings

• The full BERT-based model achieves a joint goal accuracy of 90.5% and turn request accuracy of 97.6% on the WoZ 2.0 test set, outperforming all prior methods.
• The distilled student model achieves 90.4% joint goal accuracy and 97.7% turn request accuracy, matching the full model’s performance while being 8x smaller and 7x faster on CPU.
• The distilled model is smaller (55MB) and faster than several prior BERT-based models, including DistilBERT (270MB) and Apple Core ML BERT (220MB).
• On CPU, the distilled model runs in 0.205 seconds per turn, which is 7x faster than the full model (1.465 seconds) and 3x faster than DistilBERT (0.579 seconds).
• The distilled model has fewer parameters (14M) than GLAD (17M), a non-BERT-based model, demonstrating the effectiveness of knowledge distillation for DST.
• The model generalizes to new slot-value pairs not seen during training, as it does not require retraining when the ontology changes.