[Paper Review] Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling

Jixuan Wang, Kai Wei+3 more|arXiv (Cornell University)|Dec 21, 2020

Topic Modeling41 references4 citations

TL;DR

This paper proposes a novel Transformer encoder architecture that jointly learns intent detection and slot filling while encoding syntactic knowledge through multi-task training on dependency parsing ancestors and part-of-speech tagging. By training one attention head to predict syntactic ancestors and sharing embeddings with POS tagging, the model achieves state-of-the-art performance on SNIPS and ATIS datasets with only two encoder layers, improving F1 by 1.59% and accuracy by 0.85% on SNIPS over prior non-pretrained models.

ABSTRACT

We propose a novel Transformer encoder-based architecture with syntactical knowledge encoded for intent detection and slot filling. Specifically, we encode syntactic knowledge into the Transformer encoder by jointly training it to predict syntactic parse ancestors and part-of-speech of each token via multi-task learning. Our model is based on self-attention and feed-forward layers and does not require external syntactic information to be available at inference time. Experiments show that on two benchmark datasets, our models with only two Transformer encoder layers achieve state-of-the-art results. Compared to the previously best performed model without pre-training, our models achieve absolute F1 score and accuracy improvement of 1.59% and 0.85% for slot filling and intent detection on the SNIPS dataset, respectively. Our models also achieve absolute F1 score and accuracy improvement of 0.1% and 0.34% for slot filling and intent detection on the ATIS dataset, respectively, over the previously best performed model. Furthermore, the visualization of the self-attention weights illustrates the benefits of incorporating syntactic information during training.

Motivation & Objective

• To improve intent detection and slot filling performance by integrating syntactic knowledge directly into the Transformer encoder architecture.
• To eliminate the need for external syntactic information at inference time by learning syntactic supervision during training.
• To investigate whether joint training on dependency parsing ancestors and POS tagging enhances attention learning and model generalization.
• To demonstrate that syntactic supervision improves attention patterns and model performance without requiring pre-training.

Proposed method

• The model uses a multi-task learning framework to jointly train for intent detection, slot filling, dependency parsing ancestor prediction, and part-of-speech tagging.
• A dedicated attention head in a specialized Transformer encoder layer is trained to predict the full syntactic ancestry of each token using dependency parsing trees.
• Part-of-speech tagging is performed via a shared encoder branch with the NLU task, enhancing syntactic representation learning.
• The model employs standard Transformer encoder layers with multi-head self-attention, feed-forward networks, layer normalization, and residual connections.
• Syntactic supervision is injected during training but not required at inference, enabling end-to-end deployment.
• The architecture is evaluated on the SNIPS and ATIS benchmark datasets using F1 and accuracy as metrics.

Experimental results

Research questions

• RQ1 Can encoding syntactic knowledge via multi-task learning improve performance on intent detection and slot filling without requiring external syntactic information at inference?
• RQ2 Does training a dedicated attention head to predict syntactic ancestors lead to better attention patterns and model performance?
• RQ3 How does syntactic supervision affect the attention weight distributions in the Transformer encoder?
• RQ4 Can a minimal architecture with only two Transformer encoder layers achieve state-of-the-art results when syntactic knowledge is integrated?

Key findings

• On the SNIPS dataset, the model achieves an absolute F1 improvement of 1.59% and accuracy improvement of 0.85% over the previously best-performing non-pretrained model.
• On the ATIS dataset, the model achieves an absolute F1 improvement of 0.1% and accuracy improvement of 0.34% over the previous state-of-the-art model.
• Visualization of attention weights shows that models with syntactic supervision develop more structured and informative attention patterns, especially over long-distance dependencies.
• The attention heads trained for dependency prediction learn to attend more strongly to syntactically relevant ancestors, improving contextual representation.
• The ablation study confirms that both dependency parsing and POS tagging supervision contribute to performance gains, with dependency supervision having a stronger impact.
• The model achieves SOTA results with only two Transformer encoder layers, demonstrating the efficiency and effectiveness of syntactic inductive bias.