[Paper Review] Empower Sequence Labeling with Task-Aware Neural Language Model

Liyuan Liu, Jingbo Shang+5 more|arXiv (Cornell University)|Sep 13, 2017

Natural Language Processing Techniques151 citations

TL;DR

LM-LSTM-CRF integrates a character-level neural language model with word-level BiLSTM-CRF, using highway layers to mediate task-specific knowledge, achieving state-of-the-art results on NER, POS, and chunking without extra annotations.

ABSTRACT

Linguistic sequence labeling is a general modeling approach that encompasses a variety of problems, such as part-of-speech tagging and named entity recognition. Recent advances in neural networks (NNs) make it possible to build reliable models without handcrafted features. However, in many cases, it is hard to obtain sufficient annotations to train these models. In this study, we develop a novel neural framework to extract abundant knowledge hidden in raw texts to empower the sequence labeling task. Besides word-level knowledge contained in pre-trained word embeddings, character-aware neural language models are incorporated to extract character-level knowledge. Transfer learning techniques are further adopted to mediate different components and guide the language model towards the key knowledge. Comparing to previous methods, these task-specific knowledge allows us to adopt a more concise model and conduct more efficient training. Different from most transfer learning methods, the proposed framework does not rely on any additional supervision. It extracts knowledge from self-contained order information of training sequences. Extensive experiments on benchmark datasets demonstrate the effectiveness of leveraging character-level knowledge and the efficiency of co-training. For example, on the CoNLL03 NER task, model training completes in about 6 hours on a single GPU, reaching F1 score of 91.71$\pm$0.10 without using any extra annotation.

Motivation & Objective

• Motivate improving sequence labeling by leveraging knowledge extracted from raw text without extra annotations.
• Propose a concise neural framework that integrates word-level and character-level knowledge for sequence labeling.
• Mitigate task discordance between language modeling and sequence labeling via highway layers and targeted feature transformation.
• Demonstrate efficiency and effectiveness on standard benchmarks compared to prior state-of-the-art methods.

Proposed method

• Use a character-level LSTM language model to capture character-level knowledge from unannotated text.
• Introduce highway layers to transform character-level outputs into two task-specific spaces: one for the language model and one for sequence labeling.
• Employ a word-level BiLSTM with a CRF layer for sequence labeling, and fine-tune pre-trained word embeddings for word-level knowledge.
• Joint objective combining CRF likelihood for labels and forward/backward language model likelihood, with a mediation strategy to handle task discordance.
• Train end-to-end with stochastic optimization, dropout, and gradient clipping on GPU to achieve efficiency.

Experimental results

Research questions

• RQ1 Can task-specific knowledge from a character-level language model improve sequence labeling without extra supervision?
• RQ2 Does a highway-based mediation effectively align character-level representations with the needs of language modeling and sequence labeling?
• RQ3 How does the proposed LM-LSTM-CRF perform on standard benchmarks compared to state-of-the-art methods, in terms of accuracy/F1 and training efficiency?
• RQ4 Is finetuning word embeddings sufficient for leveraging word-level knowledge without heavy co-training?

Key findings

• LM-LSTM-CRF achieves state-of-the-art F1 on CoNLL03 NER compared to non-external-resource baselines (and rivals several with extra resources).
• The model also yields strong results on WSJ POS tagging and CoNLL00 chunking, with competitive or superior performance to baselines.
• Co-training the language model with sequence labeling without mediation can hurt performance; using highway layers to transform character-level features mitigates this issue.
• Fine-tuning pre-trained word embeddings for the word-level component enables efficient use of word-level knowledge without large-scale co-training.
• The approach demonstrates strong efficiency, training on a single GPU in competitive time and with a concise model, compared to large external-resource language models.
• Empirical analyses show that larger hidden state sizes in LM-LSTM-CRF continue to improve performance, while other baselines may peak earlier, highlighting the benefit of task-specific knowledge extraction.