[Paper Review] SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling

Di Wu, Liang Ding+2 more|arXiv (Cornell University)|Oct 6, 2020

Topic Modeling22 references7 citations

TL;DR

SlotRefine proposes a fast, non-autoregressive model for joint intent detection and slot filling using a two-pass refinement mechanism to address uncoordinated slot predictions. It achieves state-of-the-art performance on ATIS and Snips datasets while accelerating inference by up to 10.77× compared to autoregressive baselines.

ABSTRACT

Slot filling and intent detection are two main tasks in spoken language understanding (SLU) system. In this paper, we propose a novel non-autoregressive model named SlotRefine for joint intent detection and slot filling. Besides, we design a novel two-pass iteration mechanism to handle the uncoordinated slots problem caused by conditional independence of non-autoregressive model. Experiments demonstrate that our model significantly outperforms previous models in slot filling task, while considerably speeding up the decoding (up to X 10.77). In-depth analyses show that 1) pretraining schemes could further enhance our model; 2) two-pass mechanism indeed remedy the uncoordinated slots.

Motivation & Objective

• Address the inefficiency and high latency of autoregressive models in joint intent detection and slot filling.
• Overcome the uncoordinated slots problem—such as I-song following B-singer—caused by conditional independence in non-autoregressive models.
• Develop a fast inference mechanism that maintains high accuracy by replacing CRF with a two-pass refinement strategy.
• Evaluate the effectiveness of pretraining schemes (GloVe, BERT) on the proposed model for improved robustness and OOV handling.

Proposed method

• Adopt a non-autoregressive Transformer-based encoder to jointly predict intent and slot labels in parallel, eliminating sequential dependency.
• Introduce a two-pass mechanism: the first pass generates initial predictions, and the second pass refines them using B-tags (beginning tags) as additional input.
• Inject initial tag embeddings (including 'O' for outside) at each token position to guide the refinement process.
• Use a learnable refinement head to refine slot tags in the second pass, explicitly modeling slot boundary consistency.
• Apply a multi-head self-attention mechanism in the Transformer encoder to capture both sentence-level and word-level representations.
• Integrate pretraining via GloVe or BERT embeddings to improve generalization, especially for OOV words, with fine-tuning using the two-pass mechanism.

Experimental results

Research questions

• RQ1 Can a non-autoregressive model achieve state-of-the-art performance in joint intent detection and slot filling while significantly improving inference speed?
• RQ2 Does the proposed two-pass refinement mechanism effectively reduce uncoordinated slot predictions (e.g., I-tag following B-tag) compared to standard non-autoregressive models?
• RQ3 How does the two-pass mechanism compare to CRF in terms of performance and efficiency for slot filling?
• RQ4 To what extent do pretraining schemes like GloVe and BERT enhance the performance of the non-autoregressive model?
• RQ5 Does the model show particular gains on datasets with high OOV rates, such as Snips?

Key findings

• SlotRefine achieves 96.22% slot F1 on the ATIS dataset and 93.72% on the Snips dataset, significantly outperforming previous models including those with CRF.
• The model reduces uncoordinated slot errors more effectively than both one-pass non-autoregressive models and CRF-augmented models, as shown by faster convergence in error reduction during training.
• Inference speed is accelerated by up to 10.77× compared to autoregressive baselines, with a 2.8× speedup over CRF-augmented models.
• Pretraining with BERT boosts performance to 97.74% intent accuracy and 88.64% slot F1 on ATIS, outperforming existing BERT-based models on the same benchmark.
• The model with GloVe pretraining achieves 97.35% intent accuracy and 87.57% slot F1 on ATIS, showing that even simple pretraining improves performance, especially on OOV-heavy datasets like Snips.
• The two-pass mechanism is more effective than CRF in balancing performance and efficiency, with a 0.89 F1 improvement over CRF-augmented models while maintaining faster inference.