[Paper Review] Prototypical Representation Learning for Relation Extraction

Ning Ding, Xiaobin Wang+8 more|arXiv (Cornell University)|Mar 22, 2021

Topic Modeling68 references38 citations

TL;DR

The paper introduces a prototypical representation learning framework for relation extraction that learns relation prototypes and statement encodings from distantly labeled data, optimizing statement-statement and prototype-statement objectives to improve robustness, interpretability, and performance across supervised, few-shot, and zero-shot settings.

ABSTRACT

Recognizing relations between entities is a pivotal task of relational learning. Learning relation representations from distantly-labeled datasets is difficult because of the abundant label noise and complicated expressions in human language. This paper aims to learn predictive, interpretable, and robust relation representations from distantly-labeled data that are effective in different settings, including supervised, distantly supervised, and few-shot learning. Instead of solely relying on the supervision from noisy labels, we propose to learn prototypes for each relation from contextual information to best explore the intrinsic semantics of relations. Prototypes are representations in the feature space abstracting the essential semantics of relations between entities in sentences. We learn prototypes based on objectives with clear geometric interpretation, where the prototypes are unit vectors uniformly dispersed in a unit ball, and statement embeddings are centered at the end of their corresponding prototype vectors on the surface of the ball. This approach allows us to learn meaningful, interpretable prototypes for the final classification. Results on several relation learning tasks show that our model significantly outperforms the previous state-of-the-art models. We further demonstrate the robustness of the encoder and the interpretability of prototypes with extensive experiments.

Motivation & Objective

• Motivate robust relation representation learning from noisy distantly labeled data.
• Propose prototypes for each relation to capture essential semantics.
• Develop geometry-inspired objectives that enforce intra-class compactness and inter-class separability.
• Pretrain on large distantly labeled data and fine-tune on target tasks.
• Demonstrate improved performance and interpretability across supervised, few-shot, and fuzzy evaluation settings.

Proposed method

• Define a relation prototype as a unit vector on a unit ball and statements as unit vectors centered near the prototype surface.
• Use a geometry-based similarity measure d(s_i,s_j) and d(s,z) based on normalized embeddings (angles on the unit ball).
• Introduce statement-statement loss L_S2S to encourage intra-class compactness and inter-class separation.
• Introduce prototype-statement losses L_S2Z and L_S2Z' to enforce that statements cluster around their prototype and are distant from other prototypes.
• Include a prototype-level classifier loss L_CLS to regularize prototypes as label-regularized centers.
• Combine losses with hyperparameters to form the full objective L = λ1 L_S2S + λ2 (L_S2Z + L_S2Z') + λ3 L_CLS.
• Adopt a pretraining-finetuning paradigm, pretraining on distantly labeled data and fine-tuning for supervised and few-shot settings.
• Evaluate on supervised RE (SemEval 2010 Task 8), few-shot (FewRel), and a proposed fuzzy relation dataset (FuzzyRED).
• Implement encoder as BERT-base with a transformer-based relation encoder and prototype-based objectives; pretrain on ~0.86M distantly labeled statements aligned from Wikidata-Wikipedia, then fine-tune.

Experimental results

Research questions

• RQ1 Can prototypical representations learned from distantly labeled data yield robust and interpretable relation representations across supervision regimes?
• RQ2 Do prototype-centered losses improve intra-class compactness and inter-class separability compared to conventional cross-entropy in relation extraction?
• RQ3 How well does the pretrained prototypical encoder transfer to supervised, few-shot, and zero-shot relation learning tasks?
• RQ4 Does the approach support interpretability of relation clusters via geometric prototype visualization and probing datasets (FuzzyRED)?

Key findings

• Significant performance gains over state-of-the-art baselines in supervised, few-shot, and zero-shot settings.
• Prototypes learnable as unit vectors on a unit ball, with statements clustering around prototype directions; geometry-driven losses yield interpretable clusters.
• The prototype-level regularization reduces decision boundary distortion under noisy labels and improves robustness in distant supervision scenarios.
• The model achieves strong results on FewRel in 5-way/1-shot and 10-way/1-shot tasks, approaching or surpassing human performance in some configurations.
• FuzzyRED evaluation shows pretrained encoders outperform distant-label baselines, with COL (combined losses) providing best robustness to label noise.