[Paper Review] Self-Attention Gazetteer Embeddings for Named-Entity Recognition

Stanislav Peshterliev, Christophe Dupuy+1 more|arXiv (Cornell University)|Apr 8, 2020

Topic Modeling27 references4 citations

TL;DR

This paper proposes GazSelfAttn, a novel gazetteer embedding method that uses self-attention and match span encoding to enhance named-entity recognition (NER) performance. By integrating these components with Bi-LSTM and CRF models and leveraging Wikidata for high-quality entity lists, the approach achieves F1 improvements of 0.52 on CoNLL-03 (92.34→92.86) and 0.21 on Ontonotes 5 (89.11→89.32), matching state-of-the-art results without requiring complex architectures like HSCRF.

ABSTRACT

Recent attempts to ingest external knowledge into neural models for named-entity recognition (NER) have exhibited mixed results. In this work, we present GazSelfAttn, a novel gazetteer embedding approach that uses self-attention and match span encoding to build enhanced gazetteer embeddings. In addition, we demonstrate how to build gazetteer resources from the open source Wikidata knowledge base. Evaluations on CoNLL-03 and Ontonotes 5 datasets, show F1 improvements over baseline model from 92.34 to 92.86 and 89.11 to 89.32 respectively, achieving performance comparable to large state-of-the-art models.

Motivation & Objective

• To improve named-entity recognition (NER) by enhancing gazetteer embeddings with self-attention and match span encoding.
• To develop a scalable method for building high-quality gazetteers using Wikidata, filtering entities by popularity to reduce ambiguity.
• To demonstrate that enhanced gazetteer embeddings can improve NER performance even when large language models (e.g., ELMo) are already used.
• To enable end-to-end training of gazetteer embeddings with standard sequence models like Bi-LSTM-CRF, avoiding complex architectures such as HSCRF.
• To evaluate the contribution of each component in the gazetteer embedding module through ablation studies.

Proposed method

• Gazetteer embeddings are generated using a self-attention mechanism over multiple gazetteer matches, combining single-token and multi-token spans.
• Match span encoding is applied to represent the position and type of each matched entity in the input sequence.
• The final gazetteer embedding is computed via a learned transformation of the self-attentive representation, with 128-dimensional output.
• Gazetteers are extracted from Wikidata using instance_of relationships and filtered by sitelink count to prioritize popular entities.
• Embeddings are concatenated with Bi-LSTM inputs and jointly trained with the CRF layer using Adam optimizer and early stopping.
• Ablation studies remove components like self-attention, span encoding, or single-token matches to evaluate their individual contributions.

Experimental results

Research questions

• RQ1 Can self-attention and match span encoding improve gazetteer embeddings for NER beyond standard one-hot or simple embedding approaches?
• RQ2 How does combining single-token and multi-token matches in a unified representation affect NER performance?
• RQ3 To what extent do Wikidata-derived gazetteers, especially when filtered by popularity, improve NER robustness and accuracy?
• RQ4 Does the proposed GazSelfAttn method achieve competitive performance compared to large state-of-the-art models without requiring complex CRF or classifier heads?
• RQ5 Which components of the GazSelfAttn architecture (e.g., self-attention, span encoding) contribute most significantly to performance gains?

Key findings

• GazSelfAttn improves F1 on CoNLL-03 from 92.34 to 92.86, a relative gain of 0.52 points, when combined with ELMo embeddings.
• On Ontonotes 5, the method increases F1 from 89.11 to 89.32, a gain of 0.21 points, demonstrating consistent improvement across datasets.
• The ablation study shows that removing single-token matches causes the largest performance drop, indicating their critical role in the model.
• Self-attention contributes significantly to performance, as its removal degrades F1 by 0.21 on CoNLL-03 and 0.12 on Ontonotes 5.
• Span encoding is more impactful on CoNLL-03 than Ontonotes 5, likely due to the higher proportion of multi-token entities in CoNLL-03.
• The model achieves performance comparable to much larger state-of-the-art models, such as those using Flair or cloze-pretrained LMs, without requiring complex architectural components like HSCRF.