[Paper Review] Joint model for intent and entity recognition

Petr Lorenc|arXiv (Cornell University)|Sep 7, 2021

Topic Modeling7 references4 citations

TL;DR

This paper proposes a joint deep learning model for intent classification and named entity recognition (NER) using word embeddings (GloVe, FastText, BERT) and convolutional neural networks over character-level representations. The model achieves state-of-the-art F1 scores of 98.44 on NER and 95.74 on intent classification using GloVe embeddings, outperforming single-task baselines while reducing model complexity and training time through joint learning.

ABSTRACT

The semantic understanding of natural dialogues composes of several parts. Some of them, like intent classification and entity detection, have a crucial role in deciding the next steps in handling user input. Handling each task as an individual problem can be wasting of training resources, and also each problem can benefit from each other. This paper tackles these problems as one. Our new model, which combine intent and entity recognition into one system, is achieving better metrics in both tasks with lower training requirements than solving each task separately. We also optimize the model based on the inputs.

Motivation & Objective

• To develop a joint model that simultaneously performs intent classification and named entity recognition to improve efficiency and performance.
• To evaluate the impact of different word embeddings (GloVe, FastText, BERT) on joint intent and entity recognition tasks.
• To compare the performance of two model architectures—one using bidirectional LSTM and another using time-distributed neural networks—on the ATIS dataset.
• To demonstrate that joint modeling reduces model complexity and training time while maintaining or improving performance over separate models.
• To investigate whether BERT embeddings benefit from additional context-aware networks like LSTM, or if they are redundant due to their inherent contextualization.

Proposed method

• The model uses character-level embeddings processed through a 1D convolutional neural network (CNN) to capture subword features.
• It incorporates word-level features such as case type, numeric status, and capitalization for each token.
• The model is architecture-agnostic regarding word embeddings, allowing plug-in of GloVe, FastText, or BERT embeddings without code changes.
• Two model variants are implemented: one using bidirectional LSTM for sequence modeling and another using time-distributed feedforward networks.
• Dropout and early stopping are applied to prevent overfitting, with training halted if validation loss does not improve for two epochs.
• The model is trained and evaluated on the public ATIS dataset, which includes 4,978 training and 893 test examples for intent classification, and 20,426 and 3,665 examples for NER, respectively.

Experimental results

Research questions

• RQ1 Does a joint model for intent classification and NER outperform separate models in terms of accuracy and F1 score?
• RQ2 How do different word embeddings (GloVe, FastText, BERT) affect the performance of the joint model on intent and entity recognition tasks?
• RQ3 Does the use of an additional context-aware network like LSTM improve performance when using BERT embeddings, or does it introduce noise?
• RQ4 Can the joint model achieve competitive performance with fewer parameters and faster training compared to single-task models?
• RQ5 What is the impact of architectural choices (LSTM vs. time-distributed networks) on model efficiency and performance?

Key findings

• The joint model with GloVe embeddings achieved the highest NER F1 score of 98.44, outperforming BERT (95.78) and FastText (96.83).
• The same model achieved 95.74% accuracy on intent classification, surpassing BERT (92.45%) and FastText (90.70%).
• When using BERT embeddings, the model with time-distributed networks (Model 2) outperformed the LSTM-based version, suggesting that LSTM adds no benefit and may introduce noise.
• Model 2 achieved a 65% reduction in training time (8s per epoch) compared to Model 1 (23s per epoch), with only a minor drop in performance.
• The joint model achieved state-of-the-art performance on NER (98.44 F1) and high intent accuracy (95.74%), outperforming prior single-task models in the comparison.
• The number of parameters was significantly reduced in Model 2 (724,882) compared to Model 1 (2.5M), demonstrating improved efficiency.