[Paper Review] Transfer Learning for Sequence Tagging with Hierarchical Recurrent Networks

Zhilin Yang, Ruslan Salakhutdinov+1 more|arXiv (Cornell University)|Mar 18, 2017

Topic Modeling20 references230 citations

TL;DR

The paper studies transfer learning for neural sequence taggers using hierarchical recurrent networks and proposes three parameter-sharing architectures to enable cross-domain, cross-application, and cross-lingual transfer, achieving improvements and new state-of-the-art results on several benchmarks.

ABSTRACT

Recent papers have shown that neural networks obtain state-of-the-art performance on several different sequence tagging tasks. One appealing property of such systems is their generality, as excellent performance can be achieved with a unified architecture and without task-specific feature engineering. However, it is unclear if such systems can be used for tasks without large amounts of training data. In this paper we explore the problem of transfer learning for neural sequence taggers, where a source task with plentiful annotations (e.g., POS tagging on Penn Treebank) is used to improve performance on a target task with fewer available annotations (e.g., POS tagging for microblogs). We examine the effects of transfer learning for deep hierarchical recurrent networks across domains, applications, and languages, and show that significant improvement can often be obtained. These improvements lead to improvements over the current state-of-the-art on several well-studied tasks.

Motivation & Objective

• Investigate whether neural sequence taggers can benefit from transfer learning across related tasks.
• Propose a unified, parameter-sharing framework for cross-domain, cross-application, and cross-lingual transfer.
• Evaluate how sharing different layers/parameters impacts transfer effectiveness across languages, domains, and applications.
• Demonstrate improvements over baselines and state-of-the-art on multiple benchmark datasets.

Proposed method

• A base neural sequence tagging model with a character-level GRU and a word-level GRU feeding a CRF output layer.
• Three transfer architectures (T-A, T-B, T-C) that share different subsets of network parameters across source and target tasks.
• Joint training with a two-task objective using shared and task-specific parameters; gradient-based optimization with AdaGrad.
• Three task transfer settings: cross-domain (with possible label mapping), cross-domain with disparate labels (T-B), cross-application, and cross-lingual transfer by sharing morphologically similar character representations (T-C).
• Training procedure alternates between source and target tasks, updating both shared and task-specific parameters; early stopping on the target task.

Experimental results

Research questions

• RQ1 Can neural sequence taggers trained on a high-resource source task improve performance on a low-resource target task via transfer learning?
• RQ2 How do cross-domain, cross-application, and cross-lingual transfer compare in terms of performance and parameter sharing?
• RQ3 What is the impact of sharing more vs fewer model components on transfer effectiveness?
• RQ4 Do transfer-learning models achieve state-of-the-art results on standard sequence tagging benchmarks?

Key findings

• Transfer learning consistently improves target-task performance over non-transfer baselines, especially at low labeling rates.
• Cross-domain transfer from PTB POS or CoNLL NER to Genia and Twitter yields substantial gains, including high accuracy with minimal labeled data.
• Cross-application and cross-lingual transfers also provide meaningful improvements under low-resource conditions.
• The three architectures follow a performance order: T-A > T-B > T-C, reflecting the amount of shared parameters.
• The approach achieves new state-of-the-art results on several benchmarks, and the base model is competitive even without transfer.