[Paper Review] Hybrid semi-Markov CRF for Neural Sequence Labeling

Zhi-Xiu Ye, Zhen-Hua Ling|arXiv (Cornell University)|May 10, 2018

Natural Language Processing Techniques3 citations

TL;DR

This paper proposes a hybrid semi-Markov CRF (HSCRF) that jointly models word-level and segment-level labels for neural sequence labeling, using word-level labels to inform segment scores and training CRF and HSCRF output layers jointly. The method achieves state-of-the-art F1 score of 91.53% on the CoNLL 2003 NER benchmark without external knowledge, outperforming prior models by better capturing both short and long entities through dual supervision.

ABSTRACT

This paper proposes hybrid semi-Markov conditional random fields (SCRFs) for neural sequence labeling in natural language processing. Based on conventional conditional random fields (CRFs), SCRFs have been designed for the tasks of assigning labels to segments by extracting features from and describing transitions between segments instead of words. In this paper, we improve the existing SCRF methods by employing word-level and segment-level information simultaneously. First, word-level labels are utilized to derive the segment scores in SCRFs. Second, a CRF output layer and an SCRF output layer are integrated into an unified neural network and trained jointly. Experimental results on CoNLL 2003 named entity recognition (NER) shared task show that our model achieves state-of-the-art performance when no external knowledge is used.

Motivation & Objective

• To improve neural sequence labeling by integrating word-level and segment-level label supervision in semi-Markov CRFs.
• To address the limitation of existing SCRF models that rely solely on segment-level labels, which may underfit short entities.
• To design a joint training and decoding framework that leverages both CRF and HSCRF outputs for improved sequence labeling.
• To achieve state-of-the-art performance on the CoNLL 2003 NER benchmark using only standard training data and pre-trained embeddings.

Proposed method

• The HSCRF model computes segment scores using the sum of word-level label scores within each segment, where each word's score is derived from a learned weight vector and its feature representation.
• Word-level features include word embeddings, segment-relative position embeddings, and the difference between the first and last word representations in a segment.
• The model integrates a standard CRF output layer and an HSCRF output layer into a unified neural network, enabling joint training.
• A naive joint decoding algorithm is proposed to decode both word-level and segment-level labels simultaneously during inference.
• The segment score function is defined as the sum of individual word scores within a segment, with the overall segmentation score computed via exponential of transition and segment scores.
• The final probability distribution over segmentations is normalized across all possible segmentations using the standard CRF normalization factor.

Experimental results

Research questions

• RQ1 Can word-level labels improve segment score estimation in semi-Markov CRFs for sequence labeling?
• RQ2 Does jointly training CRF and HSCRF output layers lead to better performance than using either model alone?
• RQ3 How does the joint modeling of word-level and segment-level labels affect the recognition of long versus short named entities?
• RQ4 Can the proposed HSCRF model achieve state-of-the-art performance on CoNLL 2003 NER without relying on external knowledge?

Key findings

• The HSCRF model achieved a state-of-the-art F1 score of 91.53% on the CoNLL 2003 NER benchmark, outperforming all comparable models without external knowledge.
• The joint training and decoding framework (LM-BLSTM-JNT) achieved the highest F1 score of 91.38% across all entity lengths, demonstrating the benefit of joint optimization.
• For long entities (≥6 words), the HSCRF model achieved an F1 score of 74.55%, outperforming CRF (72.73%) and GSCRF (73.64%), indicating better long-sequence modeling.
• For short entities (≤2 words), the HSCRF model maintained comparable performance to CRF (F1: 91.84 vs. 91.68), showing robustness to short spans.
• The ablation study showed that word-level labels improved short entity recognition, while segment-level labels enhanced long entity detection, validating the complementary nature of the dual supervision.
• The model with joint training and decoding (LM-BLSTM-JNT) achieved the best overall performance across all entity length categories, confirming the effectiveness of the joint framework.