[Paper Review] Computationally Efficient NER Taggers with Combined Embeddings and Constrained Decoding

Brian Lester, Daniel Pressel+2 more|arXiv (Cornell University)|Jan 4, 2020

Topic Modeling4 citations

TL;DR

This paper proposes a computationally efficient NER tagger that combines multiple pre-trained embeddings via concatenation and applies constrained decoding to a cross-entropy trained model, achieving a 786% speed-up over contextual embedding baselines while maintaining strong performance across multiple datasets and tasks.

ABSTRACT

Current State-of-the-Art models in Named Entity Recognition (NER) are neural models with a Conditional Random Field (CRF) as the final network layer, and pre-trained "contextual embeddings". The CRF layer is used to facilitate global coherence between labels, and the contextual embeddings provide a better representation of words in context. However, both of these improvements come at a high computational cost. In this work, we explore two simple techniques that substantially improve NER performance over a strong baseline with negligible cost. First, we use multiple pre-trained embeddings as word representations via concatenation. Second, we constrain the tagger, trained using a cross-entropy loss, during decoding to eliminate illegal transitions. While training a tagger on CoNLL 2003 we find a $786$\\% speed-up over a contextual embeddings-based tagger without sacrificing strong performance. We also show that the concatenation technique works across multiple tasks and datasets. We analyze aspects of similarity and coverage between pre-trained embeddings and the dynamics of tag co-occurrence to explain why these techniques work. We provide an open source implementation of our tagger using these techniques in three popular deep learning frameworks --- TensorFlow, Pytorch, and DyNet.

Motivation & Objective

• To reduce the computational cost of state-of-the-art NER models that rely on contextual embeddings and CRF layers.
• To improve NER performance using simple, efficient techniques without increasing training complexity.
• To explore whether combining multiple pre-trained embeddings enhances representation quality and generalization.
• To evaluate the effectiveness of constrained decoding in enforcing label consistency without CRF.
• To provide a publicly available, framework-agnostic implementation for broad adoption.

Proposed method

• Concatenating multiple pre-trained contextual embeddings (e.g., BERT, ELMo) as input representations for each token.
• Training a sequence tagger with cross-entropy loss instead of a CRF layer to reduce computational overhead.
• Applying a transition constraint matrix during decoding to eliminate illegal label transitions, ensuring global label coherence.
• Using dynamic programming with constrained transitions to decode predictions efficiently.
• Evaluating the model on CoNLL 2003 and other datasets to assess performance and speed.
• Providing open-source implementations in TensorFlow, PyTorch, and DyNet for reproducibility and integration.

Experimental results

Research questions

• RQ1 Can combining multiple pre-trained embeddings improve NER performance with minimal computational cost?
• RQ2 Does constrained decoding on a cross-entropy trained tagger match the label coherence of a CRF layer?
• RQ3 How does the combination of multiple embeddings affect label coverage and representation similarity?
• RQ4 To what extent can speed improvements be achieved without sacrificing performance?
• RQ5 Does the proposed method generalize across different NER datasets and tasks?

Key findings

• The model achieves a 786% speed-up over a contextual embedding-based tagger with CRF on the CoNLL 2003 dataset.
• The performance of the proposed model matches or exceeds that of strong baselines, including CRF-based models, on CoNLL 2003 and other datasets.
• Concatenating multiple pre-trained embeddings improves representation quality and generalization across diverse NER tasks.
• Constrained decoding effectively enforces label consistency, replacing the need for a CRF layer without performance loss.
• The method demonstrates robustness and transferability across multiple datasets and tasks.
• The open-source implementation in TensorFlow, PyTorch, and DyNet enables broad adoption and integration.