[Paper Review] LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting

Xiang Chen, Lei Li+7 more|arXiv (Cornell University)|Aug 31, 2021

Topic Modeling39 citations

TL;DR

LightNER reframes NER as a generative task with a unified verbalizer and introduces a pluggable guidance module to enable lightweight, cross-domain and cross-task transfer without updating the PLM parameters.

ABSTRACT

Most NER methods rely on extensive labeled data for model training, which struggles in the low-resource scenarios with limited training data. Existing dominant approaches usually suffer from the challenge that the target domain has different label sets compared with a resource-rich source domain, which can be concluded as class transfer and domain transfer. In this paper, we propose a lightweight tuning paradigm for low-resource NER via pluggable prompting (LightNER). Specifically, we construct the unified learnable verbalizer of entity categories to generate the entity span sequence and entity categories without any label-specific classifiers, thus addressing the class transfer issue. We further propose a pluggable guidance module by incorporating learnable parameters into the self-attention layer as guidance, which can re-modulate the attention and adapt pre-trained weights. Note that we only tune those inserted module with the whole parameter of the pre-trained language model fixed, thus, making our approach lightweight and flexible for low-resource scenarios and can better transfer knowledge across domains. Experimental results show that LightNER can obtain comparable performance in the standard supervised setting and outperform strong baselines in low-resource settings. Code is in https://github.com/zjunlp/DeepKE/tree/main/example/ner/few-shot.

Motivation & Objective

• Motivate robust NER in low-resource settings where labeled data is scarce and label sets differ across domains (class transfer).
• Propose a generative framing of NER with a unified verbalizer to handle varying entity categories without label-specific classifiers.
• Introduce a pluggable guidance module that modulates pre-trained LM attention for cross-domain knowledge transfer while keeping the LM fixed.
• Demonstrate that LightNER achieves competitive performance in rich-resource settings and superior results in low-resource scenarios across multiple datasets.

Proposed method

• Convert NER from sequence labeling to a generative framework using a unified learnable verbalizer to map entity categories to descriptive words.
• Use a seq2seq with a pointer mechanism to generate entity spans and categories from input X.
• Construct a unified verbalizer that maps each entity type to a set of label words and learns weights for combining them into p_tag.
• Incorporate a pluggable guidance module by inserting trainable parameters into self-attention layers that re-modulate attention while keeping the PLM fixed.
• Apply a lightweight tuning regime by training only the pluggable guidance and verbalizer weights (beta) while the backbone LM stays frozen.
• Enable cross-domain transfer and cross-task adaptability through decoupled class transfer and domain knowledge guidance.

Experimental results

Research questions

• RQ1 Can a unified verbalizer enable class transfer in low-resource NER without changing the model architecture?
• RQ2 Does a pluggable guidance module on top of frozen PLMs effectively support cross-domain and cross-task knowledge transfer for NER?
• RQ3 How does LightNER perform in standard supervised and low-resource cross-domain settings compared to baselines?
• RQ4 Is it feasible to transfer knowledge across domains and tasks with minimal trainable parameters in a generative NER framework?

Key findings

• LightNER achieves competitive performance with strong baselines in standard supervised settings.
• In low-resource, cross-domain settings LightNER consistently outperforms several baselines, including LC-BERT, template-based BART, and BERT-MRC.
• Ablation shows the pluggable guidance module substantially contributes to cross-domain improvements, while the unified verbalizer handles class transfer.
• Full-parameter tuning offers limited gains and reduces cross-domain transfer ability, highlighting the importance of the lightweight, pluggable approach.
• Zero-shot cross-domain results with mixed guidance prompts indicate meaningful transfer capability compared to other methods.