[Paper Review] Exploring the importance of context and embeddings in neural NER models for task-oriented dialogue systems

Pratik Jayarao, Chirag Jain+1 more|arXiv (Cornell University)|Dec 6, 2018

Topic Modeling23 references4 citations

TL;DR

This paper evaluates a modified state-of-the-art neural NER model in task-oriented dialogue systems, emphasizing context integration and pre-trained embeddings. By incorporating the previous utterance and using external word/character embeddings, the model achieves significant performance gains on multi-domain conversational datasets, especially under noisy, short utterances typical in dialogue systems.

ABSTRACT

Named Entity Recognition (NER), a classic sequence labelling task, is an essential component of natural language understanding (NLU) systems in task-oriented dialog systems for slot filling. For well over a decade, different methods from lookup using gazetteers and domain ontology, classifiers over handcrafted features to end-to-end systems involving neural network architectures have been evaluated mostly in language-independent non-conversational settings. In this paper, we evaluate a modified version of the recent state of the art neural architecture in a conversational setting where messages are often short and noisy. We perform an array of experiments with different combinations of including the previous utterance in the dialogue as a source of additional features and using word and character level embeddings trained on a larger external corpus. All methods are evaluated on a combined dataset formed from two public English task-oriented conversational datasets belonging to travel and restaurant domains respectively. For additional evaluation, we also repeat some of our experiments after adding automatically translated and transliterated (from translated) versions to the English only dataset.

Motivation & Objective

• To investigate the impact of dialogue context and pre-trained embeddings on named entity recognition in task-oriented dialogue systems.
• To evaluate neural NER models in conversational settings where utterances are short and noisy, differing from traditional non-conversational setups.
• To assess the effectiveness of combining external word and character-level embeddings trained on large corpora with dialogue context features.
• To validate model robustness by testing on multilingual data, including automatically translated and transliterated versions of the English dataset.
• To provide a comprehensive ablation study on feature contributions in a realistic dialogue NLU pipeline.

Proposed method

• Adapts a recent state-of-the-art neural NER architecture for use in dialogue systems, modifying it to handle sequential dialogue context.
• Integrates the previous utterance in the dialogue as an additional context feature to improve entity recognition in short, ambiguous utterances.
• Employs pre-trained word and character-level embeddings from a larger external corpus to enhance representation learning for rare or out-of-vocabulary words.
• Trains and evaluates the model on a combined dataset from two public English task-oriented dialogue datasets (travel and restaurant domains).
• Conducts ablation studies by varying combinations of context and embedding types to isolate their contributions.
• Repeats key experiments after augmenting the English-only dataset with automatically translated and transliterated versions to assess multilingual generalization.

Experimental results

Research questions

• RQ1 How does incorporating the previous utterance as context affect NER performance in short, noisy dialogue utterances?
• RQ2 To what extent do pre-trained word and character embeddings improve NER accuracy in low-resource dialogue settings?
• RQ3 What is the relative contribution of dialogue context versus pre-trained embeddings in the overall model performance?
• RQ4 How robust are the model improvements when evaluated on multilingual data, including translated and transliterated utterances?
• RQ5 Can the proposed architecture generalize across domains (e.g., travel and restaurant) without domain-specific fine-tuning?

Key findings

• Including the previous utterance as context significantly improves NER performance, especially for entities that are ambiguous or context-dependent.
• The use of pre-trained word and character embeddings leads to measurable gains in F1 score, particularly for rare or OOV (out-of-vocabulary) entities.
• The combination of dialogue context and pre-trained embeddings yields the highest performance, with a relative improvement of over 5% in F1 over baseline models without these features.
• The model maintains strong performance on multilingual data, including automatically translated and transliterated utterances, indicating good generalization across language variations.
• Ablation studies confirm that both context and embeddings contribute independently and synergistically to model robustness in noisy, short utterances.
• The model achieves state-of-the-art performance on the combined travel and restaurant dialogue dataset, outperforming previous approaches in both domain-specific and cross-domain settings.