[Paper Review] Design Challenges and Misconceptions in Neural Sequence Labeling

Jie Yang, Shuailong Liang+1 more|arXiv (Cornell University)|Jun 12, 2018

Neural Networks and ApplicationsComputer Science88 citations

TL;DR

The authors reproduce and systematically compare twelve neural sequence labeling models across NER, chunking, and POS tasks within a unified framework to reveal design challenges and practical guidance for practitioners.

ABSTRACT

We investigate the design challenges of constructing effective and efficient neural sequence labeling systems, by reproducing twelve neural sequence labeling models, which include most of the state-of-the-art structures, and conduct a systematic model comparison on three benchmarks (i.e. NER, Chunking, and POS tagging). Misconceptions and inconsistent conclusions in existing literature are examined and clarified under statistical experiments. In the comparison and analysis process, we reach several practical conclusions which can be useful to practitioners.

Motivation & Objective

• Assess reproducibility and fair comparison of neural sequence labeling models under unified settings.
• Identify how architectural choices (character representation, word representation, inference layer) influence performance.
• Clarify misconceptions in reported results and provide practical guidelines for practitioners.

Proposed method

• Reproduce twelve neural sequence labeling architectures using a unified PyTorch-based framework (NCRF++).
• Explore three design dimensions: character sequence representations (CNN vs LSTM), word sequence representations (CNN vs LSTM), and inference layer (softmax vs CRF).
• Evaluate on three benchmarks: CoNLL 2003 NER, CoNLL 2000 chunking, and PTB POS tagging, with controlled hyperparameters and data splits.
• Conduct statistical analyses (mean, std, max over multiple seeds) and ablation studies to assess robustness.
• Analyze external factors such as pretrained embeddings, tag schemes, hardware environment, and optimizers.

Experimental results

Research questions

• RQ1 How do character-level representations (CNN vs LSTM) affect sequence labeling performance and speed?
• RQ2 How do word-level representations (CNN vs LSTM) compare in effectiveness and decoding time across tasks?
• RQ3 Does a CRF inference layer consistently outperform softmax across NER, chunking, and POS?
• RQ4 What is the impact of pretrained embeddings, tag scheme (BIO vs BIOES), and hardware on model performance?
• RQ5 What practical guidelines can be derived for fair comparison and deployment of neural sequence labeling models?

Key findings

• Character information yields significant accuracy gains across tasks, with character LSTM and character CNN offering improvements, the difference between them being task-dependent.
• Word-based LSTM encoders generally outperform CNN encoders in most settings, indicating the importance of global word context for sequence labeling.
• CRF inference improves performance on NER and chunking, but provides no clear advantage on POS tagging.
• Word-level encoders with CRF plus appropriate character representations achieve competitive results, with CNN-based word encoders offering speed advantages in some configurations.
• BIOES tagging outperforms BIO tagging on NER, while differences for other tasks may be less pronounced; pretrained GloVe embeddings and SGD optimizer yield strong gains.
• Decoding speed is impacted by CRF (slower) and character-level components (slower for LSTM-based character models), while CNN-based word encoders offer faster decoding.