[Paper Review] Modeling Token-level Uncertainty to Learn Unknown Concepts in SLU via Calibrated Dirichlet Prior RNN

Yilin Shen, Wenhu Chen+1 more|arXiv (Cornell University)|Oct 16, 2020

Topic Modeling36 references4 citations

TL;DR

This paper proposes a Calibrated Dirichlet Prior RNN that models token-level uncertainty in slot filling without requiring unknown concept annotations. By integrating Dirichlet priors into softmax-based RNNs and calibrating concentration parameters via multi-task learning, the method achieves state-of-the-art performance, improving F1 by up to 8.18% on OOD concept detection across Snips, ATIS, and a Concept Learning benchmark.

ABSTRACT

One major task of spoken language understanding (SLU) in modern personal assistants is to extract semantic concepts from an utterance, called slot filling. Although existing slot filling models attempted to improve extracting new concepts that are not seen in training data, the performance in practice is still not satisfied. Recent research collected question and answer annotated data to learn what is unknown and should be asked, yet not practically scalable due to the heavy data collection effort. In this paper, we incorporate softmax-based slot filling neural architectures to model the sequence uncertainty without question supervision. We design a Dirichlet Prior RNN to model high-order uncertainty by degenerating as softmax layer for RNN model training. To further enhance the uncertainty modeling robustness, we propose a novel multi-task training to calibrate the Dirichlet concentration parameters. We collect unseen concepts to create two test datasets from SLU benchmark datasets Snips and ATIS. On these two and another existing Concept Learning benchmark datasets, we show that our approach significantly outperforms state-of-the-art approaches by up to 8.18%. Our method is generic and can be applied to any RNN or Transformer based slot filling models with a softmax layer.

Motivation & Objective

• To address the challenge of detecting unknown concepts in spoken language understanding (SLU) without requiring annotated unknown concept data.
• To model token-level uncertainty in RNN-based slot filling models using Dirichlet priors, enabling detection of out-of-distribution (OOD) concepts.
• To improve uncertainty estimation robustness by calibrating Dirichlet concentration parameters through multi-task learning.
• To develop a generic method applicable to any softmax-based model, including Transformers and BERT.
• To evaluate the approach on benchmark datasets with realistic OOD concept detection scenarios.

Proposed method

• Introduces a Dirichlet Prior RNN by degenerating the Dirichlet prior into the softmax layer during RNN training, preserving model architecture while enabling uncertainty modeling.
• Proposes a multi-task learning objective to calibrate Dirichlet concentration parameters, enhancing uncertainty estimation robustness.
• Uses utterance syntax (e.g., dependency trees) to guide the extraction of high-uncertainty phrases as potential unknown concepts.
• Employs Dirichlet entropy as a confidence score to identify tokens likely to be unknown concepts.
• Applies uncertainty calibration to reduce overconfidence in unseen OOD words, especially effective when OOV words are present.
• Trains the model end-to-end on standard slot filling data, with no need for explicit unknown concept annotations.

Experimental results

Research questions

• RQ1 Can Dirichlet prior integration into RNN-based slot filling models effectively model token-level uncertainty for unknown concept detection?
• RQ2 Does multi-task calibration of Dirichlet concentration parameters improve uncertainty estimation robustness?
• RQ3 How does the proposed method perform on real-world OOD concept detection compared to SOTA baselines?
• RQ4 To what extent does incorporating utterance syntax improve unknown concept extraction?
• RQ5 Can the method be generalized to other models, including Transformers and BERT?

Key findings

• The Calibrated Dirichlet Prior RNN achieves a 5.04% absolute F1 improvement over SOTA on the Concept Learning benchmark when combining Dirichlet entropy with utterance syntax.
• On the ATIS dataset, the method improves F1 by 3.62% over SOTA, with a 5.41% gain on Snips.
• The method achieves a maximum 8.18% F1 improvement on the Concept Learning dataset, particularly benefiting from OOV words in new concepts.
• Dirichlet entropy outperforms baseline uncertainty metrics such as dropout perturbation, Gaussian noise, and variance of top candidates.
• The addition of syntax information significantly boosts performance on the Concept Learning dataset, where new concepts are often noun phrases, but has limited impact on Snips and ATIS due to single-word or non-phrase concepts.
• Statistical significance tests confirm that the best-performing method (Calibrated Prior RNN + Dirichlet Entropy + syntax) significantly outperforms all SOTA competitors (p-value < 0.01) across all datasets.