[論文レビュー] Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification

Shengding Hu, Ning Ding+6 more|arXiv (Cornell University)|Aug 4, 2021

Topic Modeling被引用数 70

ひとこと要約

本論文は Knowledgeable Prompt-tuning (KPT) を提案し、外部知識ベースを用いて verbalizers を拡張・洗練させることで、プロンプトベースのテキスト分類を改善し、特に zero- and few-shot settings で効果を発揮します。KPT は diverse label words と refinement techniques を活用することで、誤り率を低減し、予測をより安定させることを示します。

ABSTRACT

Tuning pre-trained language models (PLMs) with task-specific prompts has been a promising approach for text classification. Particularly, previous studies suggest that prompt-tuning has remarkable superiority in the low-data scenario over the generic fine-tuning methods with extra classifiers. The core idea of prompt-tuning is to insert text pieces, i.e., template, to the input and transform a classification problem into a masked language modeling problem, where a crucial step is to construct a projection, i.e., verbalizer, between a label space and a label word space. A verbalizer is usually handcrafted or searched by gradient descent, which may lack coverage and bring considerable bias and high variances to the results. In this work, we focus on incorporating external knowledge into the verbalizer, forming a knowledgeable prompt-tuning (KPT), to improve and stabilize prompt-tuning. Specifically, we expand the label word space of the verbalizer using external knowledge bases (KBs) and refine the expanded label word space with the PLM itself before predicting with the expanded label word space. Extensive experiments on zero and few-shot text classification tasks demonstrate the effectiveness of knowledgeable prompt-tuning.

研究の動機と目的

• Motivate improving prompt-tuning by addressing limited coverage and bias in manual/verbalizers.
• Leverage external knowledge bases to expand the label word set for each class.
• Develop refinement mechanisms to filter noise in expanded verbalizers.
• Evaluate KPT in zero-shot and few-shot text classification across multiple datasets.

提案手法

• Wrap inputs with a prompt template to convert classification into masked language modeling.
• Construct a knowledgeable verbalizer by expanding each class with related label words from external KBs.
• Refine the expanded verbalizers using frequency, relevance, contextualized calibration, and learnable refinement.
• Utilize refined verbalizers via average or weighted average scoring to map label-word scores to class probabilities.

実験結果

リサーチクエスチョン

• RQ1 Can external knowledge bases broaden the coverage of verbalizers for text classification?
• RQ2 Do refinement steps reduce noise and improve stability in zero- and few-shot settings?
• RQ3 What gains do KPT variants achieve over standard PT and other baselines across datasets?
• RQ4 How does contextualized calibration interact with knowledge-expanded verbalizers in zero-shot scenarios?

主な発見

• KPT variants outperform standard prompt-tuning and simple verbalizers in zero-shot and few-shot settings.
• Contextualized calibration (CC) provides substantial gains, especially in zero-shot, with large average improvements over PT baselines.
• Refinement methods (frequency, relevance, shared-class calibration, and learnable refinement) reduce label-noise and improve performance, notably in topic classification tasks.
• KPT reduces prediction variance and yields more stable performance across shots.
• In zero-shot, KPT achieves reductions in error rates and demonstrates diverse, multi-granularity label words beyond class names.