[Paper Review] Multi-Task Cross-Lingual Sequence Tagging from Scratch

Zhilin Yang, Ruslan Salakhutdinov+1 more|arXiv (Cornell University)|Mar 20, 2016

Natural Language Processing Techniques33 references197 citations

TL;DR

The paper proposes a deep hierarchical GRU with a CRF layer for sequence tagging, enabling multi-task and cross-lingual joint training without feature engineering, achieving state-of-the-art results across English, Dutch, and Spanish NER, English POS tagging, and chunking.

ABSTRACT

We present a deep hierarchical recurrent neural network for sequence tagging. Given a sequence of words, our model employs deep gated recurrent units on both character and word levels to encode morphology and context information, and applies a conditional random field layer to predict the tags. Our model is task independent, language independent, and feature engineering free. We further extend our model to multi-task and cross-lingual joint training by sharing the architecture and parameters. Our model achieves state-of-the-art results in multiple languages on several benchmark tasks including POS tagging, chunking, and NER. We also demonstrate that multi-task and cross-lingual joint training can improve the performance in various cases.

Motivation & Objective

• Motivate a task- and language-agnostic sequence tagging model that avoids feature engineering.
• Investigate multi-task training within a language and cross-lingual training across languages.
• Demonstrate effectiveness on diverse tasks (POS, chunking, NER) and languages (English, Dutch, Spanish).
• Show that sharing architecture and parameters can improve performance without parallel data.

Proposed method

• Use deep hierarchical gated recurrent units (GRUs) on character and word levels to encode morphology and context.
• Apply a bidirectional multi-layer GRU at both character and word levels to learn representations without handcrafted features.
• Combine word-level representations with a CRF layer to predict tag sequences with first-order dependencies.
• Train with a max-margin augmented CRF objective that incorporates a cost (Hamming loss) between predicted and true tag sequences.
• Share network architecture and parameters across tasks (multi-task) and languages (cross-lingual) to enable joint training.
• Fine-tune pre-trained word embeddings (SENNA for English; Polyglot for Dutch/Spanish) during training.

Experimental results

Research questions

• RQ1 Can a task- and language-agnostic neural sequence tagging model achieve competitive or state-of-the-art results across POS tagging, chunking, and NER?
• RQ2 Does multi-task joint training within a language improve performance on related tagging tasks?
• RQ3 Does cross-lingual joint training leveraging morphological similarities across languages improve performance in multilingual settings without parallel corpora?
• RQ4 What is the contribution of character-level morphology versus word-level semantics in the overall model performance?

Key findings

• Achieved state-of-the-art results on CoNLL 2000 English chunking (95.41% with joint training).
• Achieved state-of-the-art results on CoNLL 2002 Dutch NER (85.19% with joint training).
• Achieved state-of-the-art results on CoNLL 2002 Spanish NER (85.77% with joint training).
• Achieved state-of-the-art results on CoNLL 2003 English NER (91.20% with gazetteer features).
• Achieved strong results on Penn Treebank POS tagging (97.55% accuracy; second-best in literature).
• Joint training improvements include Span NER +1.08 and English chunking +0.75, with various gains depending on labeling rates; cross-lingual joint training shows morphological sharing benefits (e.g., country-name morphology visualization).
• Both word embeddings and character-level GRUs contribute to performance, with word embeddings providing larger gains.