[Paper Review] PIN: A Novel Parallel Interactive Network for Spoken Language Understanding

Peilin Zhou, Zhiqi Huang+2 more|arXiv (Cornell University)|Sep 28, 2020

Topic Modeling29 references4 citations

TL;DR

This paper proposes a Parallel Interactive Network (PIN) for joint intent detection and slot filling in spoken language understanding, leveraging a Gaussian self-attentive encoder to capture local context and two bidirectional modules—Intent2Slot and Slot2Intent—for explicit, reciprocal guidance between tasks. The model achieves state-of-the-art performance when combined with BERT, particularly on the SNIPS dataset, demonstrating improved slot F1-score and overall accuracy through enhanced mutual supervision and a cooperation mechanism that fuses intuitive and rational inference pathways.

ABSTRACT

Spoken Language Understanding (SLU) is an essential part of the spoken dialogue system, which typically consists of intent detection (ID) and slot filling (SF) tasks. Recently, recurrent neural networks (RNNs) based methods achieved the state-of-the-art for SLU. It is noted that, in the existing RNN-based approaches, ID and SF tasks are often jointly modeled to utilize the correlation information between them. However, we noted that, so far, the efforts to obtain better performance by supporting bidirectional and explicit information exchange between ID and SF are not well studied.In addition, few studies attempt to capture the local context information to enhance the performance of SF. Motivated by these findings, in this paper, Parallel Interactive Network (PIN) is proposed to model the mutual guidance between ID and SF. Specifically, given an utterance, a Gaussian self-attentive encoder is introduced to generate the context-aware feature embedding of the utterance which is able to capture local context information. Taking the feature embedding of the utterance, Slot2Intent module and Intent2Slot module are developed to capture the bidirectional information flow for ID and SF tasks. Finally, a cooperation mechanism is constructed to fuse the information obtained from Slot2Intent and Intent2Slot modules to further reduce the prediction bias.The experiments on two benchmark datasets, i.e., SNIPS and ATIS, demonstrate the effectiveness of our approach, which achieves a competitive result with state-of-the-art models. More encouragingly, by using the feature embedding of the utterance generated by the pre-trained language model BERT, our method achieves the state-of-the-art among all comparison approaches.

Motivation & Objective

• To address the limited modeling of bidirectional, explicit information flow between intent detection and slot filling in existing RNN-based SLU approaches.
• To improve slot filling performance by capturing local context information through a Gaussian self-attentive encoder.
• To reduce prediction bias by fusing intuitive (implicit) and rational (explicit) reasoning pathways in a cooperation mechanism.
• To investigate the effectiveness of integrating pre-trained language models like BERT into the proposed PIN framework for enhanced performance.
• To validate the model's superiority on benchmark SLU datasets, particularly SNIPS, where intent-slot co-occurrence patterns are more pronounced.

Proposed method

• A Gaussian self-attentive encoder is used to generate context-aware, local-aware utterance representations that capture both local and global dependencies for each token.
• The Intent2Slot module explicitly uses intent predictions to guide slot filling by injecting intent-aware context into the slot decoder.
• The Slot2Intent module performs the reverse: it uses slot label distributions to refine intent classification through attention-based fusion.
• An implicit interaction stage leverages shared parameters in the encoder and intuitive decoders to capture subconscious, parameterized relationships between intent and slot.
• An explicit interaction stage uses rational decoders to process and refine predictions from the implicit stage, enabling conscious, rule-based refinement of outputs.
• A cooperation mechanism adaptively fuses features from both implicit and explicit stages to reduce prediction bias and improve robustness.

Experimental results

Research questions

• RQ1 Can bidirectional, explicit information exchange between intent detection and slot filling improve joint SLU performance?
• RQ2 Does incorporating local context information via a Gaussian self-attentive encoder enhance slot filling accuracy?
• RQ3 Can a dual-pathway architecture—combining implicit (intuitive) and explicit (rational) reasoning—reduce prediction bias and improve model robustness?
• RQ4 To what extent does integrating BERT embeddings further boost performance in the PIN framework?
• RQ5 How do co-occurrence patterns between intent and slot labels influence model performance, especially on datasets like SNIPS?

Key findings

• The PIN model achieves competitive performance on both ATIS and SNIPS, with notable improvements in slot F1-score and sentence-level semantic frame accuracy compared to baseline models.
• On the SNIPS dataset, the model outperforms previous state-of-the-art approaches, particularly benefiting from the strong co-occurrence patterns between intents and slots.
• The ablation study confirms that removing either the Intent2Slot or Slot2Intent module leads to performance degradation, validating the importance of bidirectional interaction.
• Eliminating the Gaussian self-attentive encoder results in a significant drop in slot F1-score, confirming its role in capturing local context for slot filling.
• The cooperation mechanism consistently improves performance across both datasets, demonstrating its effectiveness in balancing intuitive and rational inference pathways.
• When combined with BERT, the PIN model achieves state-of-the-art results on both slot F1-score and overall accuracy, establishing a new SOTA on the benchmark datasets.