[Paper Review] Bidirectional LSTM-CRF Models for Sequence Tagging

Zhiheng Huang, Wei Xu|arXiv (Cornell University)|Aug 9, 2015

Natural Language Processing Techniques23 references3,279 citations

TL;DR

This paper proposes a bidirectional LSTM-CRF model for sequence tagging, combining bidirectional long short-term memory networks to capture both past and future context with a conditional random field layer to model label dependencies. The model achieves state-of-the-art or near-state-of-the-art performance on POS, chunking, and NER tasks, and demonstrates robustness with reduced reliance on word embeddings compared to prior methods.

ABSTRACT

At the moment, the vast majority of Portuguese archives with an online presence use a software solution to manage their finding aids: e.g. Digitarq or Archeevo. Most of these finding aids are written in natural language without any annotation that would enable a machine to identify named entities, geographical locations or even some dates. That would allow the machine to create smart browsing tools on top of those record contents like entity linking and record linking. In this work we have created a set of datasets to train Machine Learning algorithms to find those named entities and geographical locations. After training several algorithms we tested them in several datasets and registered their precision and accuracy. These results enabled us to achieve some conclusions about what kind of precision we can achieve with this approach in this context and what to do with the results: do we have enough precision and accuracy to create toponymic and anthroponomic indexes for archival finding aids? Is this approach suitable in this context? These are some of the questions we intend to answer along this paper.

Motivation & Objective

• To develop and evaluate deep neural network models for sequence tagging tasks such as POS, chunking, and NER.
• To investigate the effectiveness of bidirectional LSTM and CRF components in improving tagging accuracy.
• To reduce dependency on pre-trained word embeddings by leveraging contextual and sequential modeling.
• To establish a robust sequence tagging framework that performs well even without external linguistic features.

Proposed method

• Proposes a bidirectional LSTM-CRF (BI-LSTM-CRF) model that integrates bidirectional long short-term memory networks with a conditional random field layer.
• Uses bidirectional LSTMs to encode both past and future context for each token by processing sequences in forward and backward directions.
• Applies a CRF layer on top of the LSTM outputs to model label dependencies and ensure globally optimal tag sequences.
• Employs a joint training procedure using backpropagation through time (BPTT) for the RNN components and CRF decoding for sequence-level optimization.
• Uses word embeddings as input features, but demonstrates strong performance even with random or non-optimized embeddings.
• Evaluates multiple variants: standard LSTM, bidirectional LSTM, LSTM-CRF, and BI-LSTM-CRF, across multiple benchmark datasets.

Experimental results

Research questions

• RQ1 Can a bidirectional LSTM-CRF model outperform standard CRF and LSTM-based models on sequence tagging benchmarks?
• RQ2 To what extent does the BI-LSTM-CRF model reduce dependence on pre-trained word embeddings?
• RQ3 How does the integration of bidirectional context and CRF decoding improve tagging accuracy across POS, chunking, and NER tasks?
• RQ4 Is the BI-LSTM-CRF model robust when key linguistic features (e.g., case, prefix, suffix) are removed?

Key findings

• The BI-LSTM-CRF model achieves an F1 score of 94.46 on the CoNLL2000 chunking dataset, outperforming previous state-of-the-art systems.
• On the CoNLL2003 NER dataset, the model reaches an F1 score of 90.10 with Senna embeddings and gazetteer features, surpassing Conv-CRF and other prior models.
• With only word features and no external embeddings, the BI-LSTM-CRF model achieves 84.74 F1 on CoNLL2003 NER, demonstrating strong robustness.
• The model reduces dependency on word embeddings: it maintains high accuracy even when using random embeddings, unlike earlier models such as Conv-CRF.
• The BI-LSTM-CRF model achieves 97.55% accuracy on the POS task without external data, outperforming all prior systems in the same setting.
• The model consistently outperforms all baseline variants (LSTM, BI-LSTM, LSTM-CRF) across all three tasks, confirming the effectiveness of combining bidirectional context and CRF decoding.