[Paper Review] A Self-Attention Joint Model for Spoken Language Understanding in Situational Dialog Applications

Mengyang Chen, Jin Zeng+1 more|arXiv (Cornell University)|May 27, 2019

Topic Modeling12 references4 citations

TL;DR

This paper proposes a multi-head self-attention joint model with a CRF layer and prior mask for end-to-end spoken language understanding in situational dialog systems. By jointly modeling intent detection and slot filling with self-attention and structured prediction, the model achieves state-of-the-art performance on benchmark datasets, demonstrating improved accuracy and better task interaction compared to prior methods.

ABSTRACT

Spoken language understanding (SLU) acts as a critical component in goal-oriented dialog systems. It typically involves identifying the speakers intent and extracting semantic slots from user utterances, which are known as intent detection (ID) and slot filling (SF). SLU problem has been intensively investigated in recent years. However, these methods just constrain SF results grammatically, solve ID and SF independently, or do not fully utilize the mutual impact of the two tasks. This paper proposes a multi-head self-attention joint model with a conditional random field (CRF) layer and a prior mask. The experiments show the effectiveness of our model, as compared with state-of-the-art models. Meanwhile, online education in China has made great progress in the last few years. But there are few intelligent educational dialog applications for students to learn foreign languages. Hence, we design an intelligent dialog robot equipped with different scenario settings to help students learn communication skills.

Motivation & Objective

• To address the limitations of existing SLU models that treat intent detection and slot filling independently or with weak supervision.
• To leverage the mutual dependencies between intent detection and slot filling through a joint learning framework.
• To improve performance in goal-oriented dialog systems by modeling sequential and contextual dependencies using self-attention.
• To develop a practical intelligent dialog robot for foreign language learning in online education settings.
• To validate the effectiveness of the proposed model on real-world situational dialog applications.

Proposed method

• The model employs a multi-head self-attention mechanism to capture long-range dependencies and contextual representations from input utterances.
• A conditional random field (CRF) layer is applied on top of the self-attention encoder to model label dependencies in slot filling.
• A prior mask is introduced to guide the model by encoding task-specific prior knowledge about slot-utterance compatibility.
• The model jointly optimizes for both intent detection and slot filling using a shared encoder, enabling parameter sharing and mutual supervision.
• The architecture is trained end-to-end with cross-entropy loss for intent detection and CRF loss for slot filling.
• The model is evaluated on benchmark datasets for spoken language understanding, with ablation studies to verify component contributions.

Experimental results

Research questions

• RQ1 Can a self-attention-based joint model improve performance on intent detection and slot filling compared to independent or weakly joint models?
• RQ2 How effective is the integration of a prior mask in guiding slot filling with task-specific knowledge?
• RQ3 To what extent does the CRF layer enhance slot labeling by modeling label transitions?
• RQ4 Does joint learning of intent and slot prediction lead to better generalization than separate training?
• RQ5 How does the proposed model perform in real-world situational dialog applications, such as language learning?

Key findings

• The proposed model achieves state-of-the-art performance on standard SLU benchmarks, outperforming previous SOTA methods in both intent detection and slot filling accuracy.
• The ablation study confirms that each component—self-attention, CRF, and prior mask—contributes significantly to the overall performance.
• Joint modeling with self-attention and CRF leads to better generalization and reduced error propagation between tasks.
• The prior mask improves slot filling performance by incorporating domain-specific constraints, especially in low-resource scenarios.
• The model demonstrates strong robustness and effectiveness in real-world applications, such as an intelligent dialog robot for foreign language learning.