[論文レビュー] Prompt-Learning for Fine-Grained Entity Typing

Ning Ding, Yulin Chen+7 more|arXiv (Cornell University)|Aug 24, 2021

Topic Modeling被引用数 44

ひとこと要約

この論文は、プロンプト学習を用いて完全監視、少数-shot、ゼロ-shotの設定で粒度の高い固有表現の型付けを行う方法を研究し、データが限られている場合に vanilla fine-tuning より改善を示す。

ABSTRACT

As an effective approach to tune pre-trained language models (PLMs) for specific tasks, prompt-learning has recently attracted much attention from researchers. By using \textit{cloze}-style language prompts to stimulate the versatile knowledge of PLMs, prompt-learning can achieve promising results on a series of NLP tasks, such as natural language inference, sentiment classification, and knowledge probing. In this work, we investigate the application of prompt-learning on fine-grained entity typing in fully supervised, few-shot and zero-shot scenarios. We first develop a simple and effective prompt-learning pipeline by constructing entity-oriented verbalizers and templates and conducting masked language modeling. Further, to tackle the zero-shot regime, we propose a self-supervised strategy that carries out distribution-level optimization in prompt-learning to automatically summarize the information of entity types. Extensive experiments on three fine-grained entity typing benchmarks (with up to 86 classes) under fully supervised, few-shot and zero-shot settings show that prompt-learning methods significantly outperform fine-tuning baselines, especially when the training data is insufficient.

研究の動機と目的

• Motivate the use of prompt-learning to stimulate PLMs for fine-grained entity typing.
• Develop a simple prompt-learning pipeline with entity-oriented labels and templates.
• Investigate both supervised and self-supervised prompt-learning to handle data-scarce regimes.
• Evaluate performance on three benchmarks under fully supervised, few-shot, and zero-shot settings.

提案手法

• Formulate entity typing as a cloze-style task using prompt templates and label words.
• Construct entity-oriented label word sets V* and compute P(y|x) as an average over P([MASK] = w|T(x)) for w in V_y.
• Explore hard-encoding T1–T3 and soft-encoding T4 prompts to instantiate prompts for PLMs.
• Train M with cross-entropy loss L = -log P(y|x; θ, φ) and optimize prompt parameters φ jointly with M.
• Propose a self-supervised prompt-learning method for zero-shot typing by contrasting distributions over V* using positive and negative sentence pairs.
• Use Jensen-Shannon divergence to measure distribution similarity and optimize with a contrastive-like objective.
• Employ a dataset of about 1M positive and negative pairs for self-supervised learning from an entity-linked corpus.

実験結果

リサーチクエスチョン

• RQ1 Can prompt-learning outperform vanilla fine-tuning for fine-grained entity typing in fully supervised regimes?
• RQ2 How do hard and soft prompt encodings compare in effectiveness across datasets?
• RQ3 Does prompt-learning provide advantages in few-shot settings compared to traditional fine-tuning?
• RQ4 Is zero-shot entity typing feasible with self-supervised prompt-learning using unlabeled data?
• RQ5 What are the characteristics and limitations of prompt-learning for entity attribute detection on hierarchical type sets?

主な発見

データセット	Acc (FT)	Acc (Plet H)	Acc (Plet S)	MiF (FT)	MiF (Plet H)	MiF (Plet S)	MaF (FT)	MaF (Plet H)	MaF (Plet S)
Few-NERD	79.75	79.90	79.86	85.74	85.84	85.76	85.74	85.84	85.76
OntoNotes	59.71	60.37	65.68	70.47	70.78	74.53	76.57	76.42	79.77
BBN	62.39	65.92	63.11	68.88	71.55	68.68	67.37	70.82	67.81

• Prompt-based methods outperform vanilla fine-tuning on the three benchmarks in many settings, with varying gains depending on data availability.
• Hard-encoding templates (T3) and soft-encoding templates (T4) both improve results over fine-tuning, with dataset-dependent preferences.
• In fully supervised settings, Plet (hard) and Plet (soft) yield higher accuracies and macro/micro F1 scores across Few-NERD, OntoNotes, and BBN.
• Zero-shot and few-shot scenarios benefit notably from prompt-learning, particularly OntoNotes and Few-NERD where gains are substantial.
• Self-supervised prompt-learning enables improvement in zero-shot typing by aligning distributions over label words for similar entities without explicit labels.