[Paper Review] Plug-Tagger: A Pluggable Sequence Labeling Framework Using Language Models

Xin Zhou, Ruotian Ma+4 more|arXiv (Cornell University)|Oct 14, 2021

Topic Modeling38 references4 citations

TL;DR

Plug-Tagger is a plug-and-play framework for sequence labeling that uses label word prediction instead of classification, enabling a frozen pre-trained language model to perform multiple tasks by switching lightweight input vectors. It achieves performance comparable to full fine-tuning with only 0.1% task-specific parameters and up to 70x faster task switching in resource-constrained settings.

ABSTRACT

Plug-and-play functionality allows deep learning models to adapt well to different tasks without requiring any parameters modified. Recently, prefix-tuning was shown to be a plug-and-play method on various text generation tasks by simply inserting corresponding continuous vectors into the inputs. However, sequence labeling tasks invalidate existing plug-and-play methods since different label sets demand changes to the architecture of the model classifier. In this work, we propose the use of label word prediction instead of classification to totally reuse the architecture of pre-trained models for sequence labeling tasks. Specifically, for each task, a label word set is first constructed by selecting a high-frequency word for each class respectively, and then, task-specific vectors are inserted into the inputs and optimized to manipulate the model predictions towards the corresponding label words. As a result, by simply switching the plugin vectors on the input, a frozen pre-trained language model is allowed to perform different tasks. Experimental results on three sequence labeling tasks show that the performance of the proposed method can achieve comparable performance with standard fine-tuning with only 0.1\% task-specific parameters. In addition, our method is up to 70 times faster than non-plug-and-play methods while switching different tasks under the resource-constrained scenario.

Motivation & Objective

• To enable plug-and-play adaptation of pre-trained language models (PLMs) for sequence labeling tasks without modifying model architecture or parameters.
• To overcome the limitation of existing plug-and-play methods like prefix-tuning, which fail on sequence labeling due to task-specific classifier requirements.
• To enable rapid task switching in industrial settings where model redeployment is costly and time-consuming.
• To reuse the PLM's language model head by reformulating sequence labeling as label word prediction, avoiding task-specific classifiers.
• To achieve high performance with minimal trainable parameters and low inference latency in dynamic, resource-constrained environments.

Proposed method

• Reformulate sequence labeling as a label word prediction task by selecting high-frequency words as representative label words for each class.
• Insert task-specific continuous vectors (plugins) into the input sequence to guide the frozen PLM to predict the corresponding label words.
• Use the PLM’s pre-trained language model head for prediction, eliminating the need for a task-specific classifier.
• Map predicted label words back to actual labels via a predefined label word mapping table.
• Optimize only the input plugin vectors during training, keeping all PLM parameters frozen.
• Enable task switching by simply replacing the plugin vectors at inference time, without retraining or reconfiguring the model.

Experimental results

Research questions

• RQ1 Can sequence labeling be made fully plug-and-play using only input-level modifications, without changing the model architecture or parameters?
• RQ2 Can label word prediction effectively replace traditional classification heads in sequence labeling tasks while maintaining performance?
• RQ3 How does the performance of Plug-Tagger compare to standard fine-tuning in terms of accuracy and parameter efficiency?
• RQ4 To what extent does Plug-Tagger improve inference speed and deployment efficiency in dynamic, multi-task environments?
• RQ5 What is the impact of label word selection strategy on the robustness and generalization of the plug-and-play framework?

Key findings

• Plug-Tagger achieves performance comparable to standard fine-tuning on three sequence labeling tasks, using only 0.1% of the parameters for task-specific adaptation.
• The method is up to 70 times faster than non-plug-and-play methods when switching tasks under resource-constrained conditions.
• Label word prediction via high-frequency word selection enables effective reuse of the PLM’s language model head across diverse label sets.
• The framework maintains strong performance across NER, POS tagging, and chunking, with consistent gains on CoNLL2003 and ACE2005, though performance degradation was observed in chunking due to adjacent same-label phrases being incorrectly grouped.
• ExactMatch, a label word selection strategy based on exact matching, outperformed GreedyMatch in most scenarios, especially where label consistency and word frequency were critical.
• The plug-and-play mechanism allows full model reuse and eliminates the need for model retraining or redeployment when switching tasks.