[Paper Review] Improving Distantly-Supervised Relation Extraction through BERT-based Label & Instance Embeddings

Despina Christou, Grigorios Tsoumakas|arXiv (Cornell University)|Feb 1, 2021

Topic Modeling27 references4 citations

TL;DR

REDSandT is a BERT-based relation extraction model that improves distantly supervised RE by learning informative instance and label embeddings through a structured input focusing on the sub-tree connecting entity pairs and their types. By fine-tuning BERT on this RE-specific input and using relation embeddings as attention over instances, it achieves state-of-the-art AUC of 0.424 on NYT-10, outperforming prior methods in detecting long-tail relations with higher confidence.

ABSTRACT

Distantly-supervised relation extraction (RE) is an effective method to scale RE to large corpora but suffers from noisy labels. Existing approaches try to alleviate noise through multi-instance learning and by providing additional information, but manage to recognize mainly the top frequent relations, neglecting those in the long-tail. We propose REDSandT (Relation Extraction with Distant Supervision and Transformers), a novel distantly-supervised transformer-based RE method, that manages to capture a wider set of relations through highly informative instance and label embeddings for RE, by exploiting BERT's pre-trained model, and the relationship between labels and entities, respectively. We guide REDSandT to focus solely on relational tokens by fine-tuning BERT on a structured input, including the sub-tree connecting an entity pair and the entities' types. Using the extracted informative vectors, we shape label embeddings, which we also use as attention mechanism over instances to further reduce noise. Finally, we represent sentences by concatenating relation and instance embeddings. Experiments in the NYT-10 dataset show that REDSandT captures a broader set of relations with higher confidence, achieving state-of-the-art AUC (0.424).

Motivation & Objective

• To address the challenge of noisy labels in distantly supervised relation extraction (DS-RE), especially for low-frequency, long-tail relations.
• To improve model performance by leveraging BERT’s pre-trained knowledge through a structured, RE-specific input format.
• To reduce noise and enhance representation learning by focusing on relational tokens via sub-tree parsing (STP) and entity type information.
• To develop a lightweight yet effective method that maintains low computational complexity while outperforming existing SOTA models.

Proposed method

• Fine-tune BERT on a structured input that includes the shortest dependency sub-tree (STP) connecting an entity pair and their entity types, to generate informative instance embeddings.
• Use the resulting BERT-encoded vectors to shape label embeddings via the TransE model, representing relations as the vector difference between entity embeddings.
• Apply the learned label embeddings as a relation-wise attention mechanism over sentence representations to suppress less-informative tokens.
• Concatenate relation embeddings with weighted instance embeddings to form final sentence representations for bag-level classification.
• Perform relation classification at the bag level by aggregating predictions from all sentences in a bag using a weighted sum.
• Minimize architectural changes by adapting BERT with minimal modifications, preserving its bidirectional context understanding.

Experimental results

Research questions

• RQ1 Can a BERT-based model fine-tuned on a structured, RE-specific input improve relation extraction performance in a distantly supervised setting?
• RQ2 How does using label embeddings derived from BERT-encoded vectors affect the detection of long-tail relations?
• RQ3 To what extent does focusing on relational tokens via sub-tree parsing (STP) reduce noise and improve model efficiency?
• RQ4 Can label embeddings serve as an effective attention mechanism over instances to enhance representation learning?
• RQ5 Does the proposed method achieve state-of-the-art performance on standard benchmarks like NYT-10, particularly in AUC and precision at top-k recall?

Key findings

• REDSandT achieves a state-of-the-art AUC of 0.424 on the NYT-10 dataset, surpassing prior methods by 1.0 and 0.2 units over Vashishth et al. (2018) and Alt et al. (2019), respectively.
• The model demonstrates a 7–10% improvement in precision at top-100, 200, and 300 predictions compared to Alt et al. (2019), indicating better performance at high recall.
• REDSandT successfully detects relations in the long-tail that are often missed by existing models, due to its ability to learn informative label and instance embeddings.
• The use of STP encoding preserves syntactic grammar information and enables the model to focus on relational content, improving representation quality with low computational overhead.
• Label embeddings derived from BERT’s vector space and used as attention mechanisms significantly reduce the impact of noisy, non-relational tokens in sentences.
• Fine-tuning BERT on a structured input that includes entity types and STP leads to more robust and informative sentence representations, enhancing overall classification performance.