[Paper Review] Transfer Learning for Sequence Labeling Using Source Model and Target Data

Lingzhen Chen, Alessandro Moschitti|arXiv (Cornell University)|Feb 14, 2019

Topic Modeling20 references16 citations

TL;DR

This paper proposes a transfer learning approach for sequence labeling that adapts a pre-trained source model to a target domain with new entity categories, without access to source data. It introduces a neural adapter to align source and target label distributions, significantly improving performance on new categories—especially with limited target data—achieving up to 10.30 F1-point gains on I-CAB and 1.83 on CONLL-2003.

ABSTRACT

In this paper, we propose an approach for transferring the knowledge of a neural model for sequence labeling, learned from the source domain, to a new model trained on a target domain, where new label categories appear. Our transfer learning (TL) techniques enable to adapt the source model using the target data and new categories, without accessing to the source data. Our solution consists in adding new neurons in the output layer of the target model and transferring parameters from the source model, which are then fine-tuned with the target data. Additionally, we propose a neural adapter to learn the difference between the source and the target label distribution, which provides additional important information to the target model. Our experiments on Named Entity Recognition show that (i) the learned knowledge in the source model can be effectively transferred when the target data contains new categories and (ii) our neural adapter further improves such transfer.

Motivation & Objective

• Address the challenge of adapting pre-trained sequence labeling models to new domains where novel entity categories emerge.
• Enable knowledge transfer from a source model to a target model without access to the source training data.
• Mitigate catastrophic forgetting and annotation disagreement when new categories are introduced.
• Improve performance on new entity types, especially when labeled target data is scarce.
• Demonstrate the generalizability of the method beyond NER to other sequence labeling tasks.

Proposed method

• Fine-tune a pre-trained BLSTM-CRF model on target data by adding new output neurons for unseen categories and initializing them with transferred source model weights.
• Introduce a neural adapter that connects the source model’s hidden states to the target model’s input, enabling feature-level knowledge transfer.
• Use a bidirectional LSTM in the adapter to encode contextual differences between source and target label distributions.
• Train the target model end-to-end with the adapter, keeping the source model frozen to prevent parameter updates.
• Initialize new output layer neurons with source model parameters and fine-tune them on target data, including both seen and unseen categories.
• Apply CRF decoding to maintain sequence consistency during inference on the target task.

Experimental results

Research questions

• RQ1 Can a pre-trained sequence labeling model be effectively adapted to a new domain with novel entity categories when source data is unavailable?
• RQ2 How does parameter initialization from the source model affect performance on new categories in the target domain?
• RQ3 To what extent does the neural adapter improve transfer performance, especially with limited labeled target data?
• RQ4 Does the adapter help reduce forgetting of previously learned categories during adaptation to new labels?
• RQ5 How generalizable is the proposed method across different sequence labeling tasks and datasets?

Key findings

• The proposed transfer learning method significantly improves F1 scores on new entity categories, achieving an average gain of 1.83 points on CONLL-2003 and 10.30 points on I-CAB compared to the baseline.
• The neural adapter consistently improves performance on new categories, particularly when only 10–25% of the target data is available for training.
• Models with the adapter converge faster during fine-tuning, showing smoother and more stable learning curves.
• The adapter mitigates the impact of annotation disagreement, such as when surface forms of new categories were previously labeled differently or not at all.
• The method effectively prevents catastrophic forgetting of previously learned categories while adapting to new ones.
• The approach generalizes well across different entity types (LOC, PER, ORG, MISC, GPE), with consistent improvements across all categories on both datasets.