[Paper Review] RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extraction

Yew Ken Chia, Lidong Bing+2 more|arXiv (Cornell University)|Mar 17, 2022

Topic Modeling4 citations

TL;DR

This paper introduces Zero-Shot Relation Triplet Extraction (ZeroRTE), a novel task requiring models to extract full (head, relation, tail) triplets for unseen relations. It proposes RelationPrompt, a method that uses large language models to generate synthetic training data via structured prompt templates conditioned on relation labels, and Triplet Search Decoding to extract multiple triplets per sentence, achieving state-of-the-art performance on FewRel and Wiki-ZSL.

ABSTRACT

Despite the importance of relation extraction in building and representing knowledge, less research is focused on generalizing to unseen relations types. We introduce the task setting of Zero-Shot Relation Triplet Extraction (ZeroRTE) to encourage further research in low-resource relation extraction methods. Given an input sentence, each extracted triplet consists of the head entity, relation label, and tail entity where the relation label is not seen at the training stage. To solve ZeroRTE, we propose to synthesize relation examples by prompting language models to generate structured texts. Concretely, we unify language model prompts and structured text approaches to design a structured prompt template for generating synthetic relation samples when conditioning on relation label prompts (RelationPrompt). To overcome the limitation for extracting multiple relation triplets in a sentence, we design a novel Triplet Search Decoding method. Experiments on FewRel and Wiki-ZSL datasets show the efficacy of RelationPrompt for the ZeroRTE task and zero-shot relation classification. Our code and data are available at github.com/declare-lab/RelationPrompt.

Motivation & Objective

• To address the lack of research in zero-shot relation extraction for full triplet prediction, where no training examples exist for test relation types.
• To overcome data scarcity in low-resource relation extraction by generating synthetic training data for unseen relations.
• To develop a method that generalizes to unseen relations without requiring annotated training examples for those relations.
• To enable effective and interpretable extraction of multiple relation triplets from a single sentence in the zero-shot setting.

Proposed method

• Design a structured prompt template that combines relation label prompts with language model generation to produce synthetic relation triplet examples.
• Condition the language model on relation labels to generate contextually grounded, structured sentences containing head and tail entities.
• Introduce Triplet Search Decoding, a method that iteratively identifies and extracts multiple relation triplets from a single input sentence using the generated synthetic data.
• Fine-tune a relation extractor on the synthetic data to perform zero-shot inference on unseen relations.
• Use large pre-trained language models to leverage their semantic and factual knowledge for generating diverse and realistic synthetic samples.
• Optimize the number of synthetic samples (e.g., 1000) to balance performance and avoid overfitting due to noisy data.

Experimental results

Research questions

• RQ1 Can language models generate high-quality, structured synthetic data for unseen relation types in the context of relation triplet extraction?
• RQ2 How effective is the proposed RelationPrompt framework in enabling zero-shot generalization to unseen relations?
• RQ3 Can the Triplet Search Decoding method reliably extract multiple relation triplets from a single sentence in the zero-shot setting?
• RQ4 What is the optimal scale of synthetic data for balancing performance and avoiding overfitting?
• RQ5 How does the model's generalization capability compare to existing zero-shot relation classification baselines?

Key findings

• RelationPrompt significantly outperforms prior zero-shot relation classification methods on the ZeroRTE benchmark, demonstrating strong zero-shot generalization.
• The method achieves state-of-the-art performance on FewRel and Wiki-ZSL datasets, validating its effectiveness in low-resource relation extraction.
• Generating approximately 1000 synthetic samples per relation yields optimal performance, as larger amounts lead to overfitting due to noise.
• Qualitative analysis shows that the generator produces contextually plausible sentences for most relations, though entity-relation alignment can be inconsistent for certain semantic relations.
• The Triplet Search Decoding method effectively extracts multiple triplets from a single sentence, improving robustness and interpretability.
• The approach generalizes well to real-world relations such as 'Investor', 'Defeated By', and 'Currency Of', indicating strong semantic grounding in the generated data.