[Paper Review] An Overview of Distant Supervision for Relation Extraction with a Focus on Denoising and Pre-training Methods

William J. Hogan|arXiv (Cornell University)|Jul 17, 2022

Topic Modeling4 citations

TL;DR

This survey reviews distant supervision for relation extraction, focusing on denoising and pre-training methods. It introduces Fine-grained Contrastive Learning (FineCL), which leverages training instance learning order to improve relation representations by weighting early-learned, likely-correct instances more heavily during contrastive pre-training, achieving state-of-the-art performance on DocRED, TACRED, and SemEval datasets.

ABSTRACT

Relation Extraction (RE) is a foundational task of natural language processing. RE seeks to transform raw, unstructured text into structured knowledge by identifying relational information between entity pairs found in text. RE has numerous uses, such as knowledge graph completion, text summarization, question-answering, and search querying. The history of RE methods can be roughly organized into four phases: pattern-based RE, statistical-based RE, neural-based RE, and large language model-based RE. This survey begins with an overview of a few exemplary works in the earlier phases of RE, highlighting limitations and shortcomings to contextualize progress. Next, we review popular benchmarks and critically examine metrics used to assess RE performance. We then discuss distant supervision, a paradigm that has shaped the development of modern RE methods. Lastly, we review recent RE works focusing on denoising and pre-training methods.

Motivation & Objective

• To provide a comprehensive overview of the evolution of relation extraction (RE) methods from pattern-based to large language model-based approaches.
• To analyze the limitations of traditional RE methods, particularly low recall in pattern-based systems and data scarcity in statistical models.
• To examine the role of distant supervision in enabling large-scale, weakly supervised relation extraction using knowledge graphs and unstructured text.
• To investigate recent advances in denoising and pre-training techniques that improve robustness and performance in distantly supervised RE.
• To evaluate the effectiveness of Fine-grained Contrastive Learning (FineCL) in leveraging training dynamics to enhance relation representation learning.

Proposed method

• FineCL uses off-the-shelf language models (e.g., RoBERTa) pre-trained on distantly supervised data via cross-entropy loss.
• The method records the training order in which relation instances are first correctly predicted, assuming earlier-learned instances are more likely to be correctly labeled.
• It computes instance-specific weights based on learning order, assigning higher weights to instances learned earlier (presumed cleaner data).
• These instance weights are used to modulate a contrastive learning loss function that encourages clustering of semantically similar relations.
• The resulting contrastive objective improves the quality of relation representations during pre-training.
• Final models are fine-tuned on gold-labeled datasets using the enhanced representations from the pre-training phase.

Experimental results

Research questions

• RQ1 How does the learning order of training instances in distantly supervised relation extraction correlate with label accuracy?
• RQ2 Can fine-grained supervision based on training dynamics improve relation representation learning in low-resource settings?
• RQ3 To what extent does incorporating instance learning order into contrastive pre-training enhance performance on downstream relation extraction benchmarks?
• RQ4 How do denoising and pre-training methods mitigate noise in distantly supervised relation extraction data?
• RQ5 What is the comparative performance of FineCL against existing pre-training and denoising baselines on standard RE benchmarks?

Key findings

• FineCL achieves state-of-the-art performance on the DocRED test set, with an F1-micro score of 59.5% on 100% of the training data, outperforming all baselines including RoBERTa and ERICA.
• On the TACRED dataset, FineCL achieves an F1-micro score of 70.3% with 100% of the training data, surpassing the next best baseline (ERICA RoBERTa) by 0.8 points.
• On SemEval-2010 Task 8, FineCL achieves 88.7% F1-micro with 100% data, exceeding the best baseline (ERICA RoBERTa) by 0.9 points.
• The method shows consistent gains across all data settings (1%, 10%, 100%) on all benchmarks, indicating robustness to data scarcity.
• The use of learning-order-based weighting in contrastive learning leads to improved generalization, especially in low-data regimes.
• The correlation between early learning and label accuracy is empirically validated, supporting the core assumption of FineCL.