[Paper Review] Neural Adaptation Layers for Cross-domain Named Entity Recognition

Bill Yuchen Lin, Wei Lu|arXiv (Cornell University)|Oct 15, 2018

Topic Modeling36 references4 citations

TL;DR

This paper proposes lightweight neural adaptation layers—word, sentence, and output adaptation layers—that enable effective cross-domain named entity recognition without retraining on source-domain data. By introducing a single learnable hyper-parameter to control knowledge transfer, the method significantly outperforms existing transfer learning approaches on benchmark NER datasets, demonstrating strong generalization across domains with minimal computational overhead.

ABSTRACT

Recent research efforts have shown that neural architectures can be effective in conventional information extraction tasks such as named entity recognition, yielding state-of-the-art results on standard newswire datasets. However, despite significant resources required for training such models, the performance of a model trained on one domain typically degrades dramatically when applied to a different domain, yet extracting entities from new emerging domains such as social media can be of significant interest. In this paper, we empirically investigate effective methods for conveniently adapting an existing, well-trained neural NER model for a new domain. Unlike existing approaches, we propose lightweight yet effective methods for performing domain adaptation for neural models. Specifically, we introduce adaptation layers on top of existing neural architectures, where no re-training using the source domain data is required. We conduct extensive empirical studies and show that our approach significantly outperforms state-of-the-art methods.

Motivation & Objective

• To address the performance drop of neural NER models when applied to new domains such as social media.
• To overcome limitations of existing transfer learning methods, including rigid assumptions about shared input/output spaces and the need for retraining on source data.
• To develop a lightweight, efficient adaptation mechanism that preserves knowledge from a pre-trained source model while adapting to a target domain.
• To enable domain adaptation at both input and output levels by introducing dedicated adaptation layers that bridge domain shifts.
• To provide a generalizable framework applicable to other cross-domain structured prediction tasks in NLP.

Proposed method

• Introduces a word adaptation layer that learns a linear transformation between source and target domain word embeddings, enabling domain-invariant representations.
• Employs a sentence adaptation layer using a bidirectional LSTM to model contextual domain shifts at the sentence level, improving alignment between source and target input sequences.
• Adds an output adaptation layer between the LSTM and CRF layer to model structural differences in label distributions across domains.
• Uses a single learnable hyper-parameter ψ to control the extent of knowledge transfer from the source model, allowing flexible adaptation.
• Applies the adaptation layers on top of a pre-trained BLSTM-CRF model without fine-tuning the original model weights, ensuring efficiency and reusability.
• Learns the adaptation layers using only labeled target-domain data, avoiding the need for source-domain data during adaptation.

Experimental results

Research questions

• RQ1 Can we improve cross-domain NER performance without retraining the source-domain model?
• RQ2 How effective is a modular, layer-wise adaptation strategy in bridging domain shifts at the input, hidden, and output levels?
• RQ3 Can we achieve better performance than existing transfer learning baselines like INIT and MULT with a lightweight, plug-and-play mechanism?
• RQ4 How does the hyper-parameter ψ influence the trade-off between preserving source knowledge and adapting to the target domain?
• RQ5 Can the proposed method generalize to other structured prediction tasks beyond NER?

Key findings

• The proposed method significantly outperforms state-of-the-art transfer learning baselines, including INIT and MULT, across multiple cross-domain NER benchmarks.
• The ablation study confirms that each adaptation layer (word, sentence, output) contributes independently and cumulatively to performance gains.
• The method achieves strong performance even when source and target domains exhibit substantial domain shifts in vocabulary and label distributions.
• The model maintains high efficiency, requiring no retraining on source data and only fine-tuning on target data, making it practical for real-world deployment.
• The hyper-parameter ψ enables effective control of knowledge transfer, and its optimal value is robust across different domain pairs.
• The approach generalizes well, as it can be applied to any pre-trained neural NER model without architectural changes.