[Paper Review] Learning from Context or Names? An Empirical Study on Neural Relation Extraction

Hao Peng, Tianyu Gao+6 more|arXiv (Cornell University)|Oct 5, 2020

Topic Modeling39 references18 citations

TL;DR

This paper investigates the reliance of neural relation extraction (NRE) models on textual context versus entity mentions, revealing that models heavily depend on mention-based type information and are prone to exploiting shallow cues in mentions. To address this, the authors propose an entity-masked contrastive pre-training framework that enhances contextual understanding while reducing bias toward entity names, significantly improving robustness and performance—especially in low-resource settings—across multiple benchmarks.

ABSTRACT

Neural models have achieved remarkable success on relation extraction (RE) benchmarks. However, there is no clear understanding which type of information affects existing RE models to make decisions and how to further improve the performance of these models. To this end, we empirically study the effect of two main information sources in text: textual context and entity mentions (names). We find that (i) while context is the main source to support the predictions, RE models also heavily rely on the information from entity mentions, most of which is type information, and (ii) existing datasets may leak shallow heuristics via entity mentions and thus contribute to the high performance on RE benchmarks. Based on the analyses, we propose an entity-masked contrastive pre-training framework for RE to gain a deeper understanding on both textual context and type information while avoiding rote memorization of entities or use of superficial cues in mentions. We carry out extensive experiments to support our views, and show that our framework can improve the effectiveness and robustness of neural models in different RE scenarios. All the code and datasets are released at https://github.com/thunlp/RE-Context-or-Names.

Motivation & Objective

• To investigate the extent to which neural relation extraction models rely on textual context versus entity mentions (especially type information).
• To identify whether existing RE benchmarks leak shallow statistical cues through entity mentions that bias model predictions.
• To improve model robustness by reducing over-reliance on superficial cues in entity names while enhancing contextual understanding.
• To develop a pre-training framework that explicitly learns relational semantics from context while masking entity mentions to avoid memorization.

Proposed method

• The authors conduct ablation studies by training models on only context (OnlyC), only mentions (OnlyM), and both (C+M) to isolate the contribution of each input modality.
• They propose an entity-masked contrastive pre-training (CP) framework that masks entity mentions during pre-training to prevent models from memorizing entity identities.
• The framework uses a contrastive objective to learn semantically similar sentence pairs expressing the same relation, using Wikidata to identify such pairs.
• It leverages pre-trained language models (e.g., BERT) and applies masking to entity mentions during pre-training to encourage the model to focus on relational patterns in context.
• The method is trained on Wikipedia sentences and evaluated on diverse benchmarks including TACRED, Wiki80, FewRel, and ChemProt.
• Performance is evaluated across full-resource and low-resource/few-shot settings to assess robustness and generalization.

Experimental results

Research questions

• RQ1 To what extent do neural relation extraction models rely on textual context versus entity mentions for relation classification?
• RQ2 Do existing RE benchmarks contain shallow statistical cues in entity mentions that models exploit instead of learning contextual patterns?
• RQ3 Can a contrastive pre-training framework that masks entity mentions improve model generalization and reduce reliance on superficial cues in mentions?
• RQ4 Does the proposed method enhance performance across diverse domains and low-resource settings?

Key findings

• Models achieve high performance even when given only entity mentions as input, indicating that existing benchmarks leak shallow cues through mention-based type information.
• The proposed entity-masked contrastive pre-training (CP) framework improves performance across all benchmarks, with gains of 7% on 10-way 1-shot FewRel 1.0 and 18% on the 1% setting of TACRED.
• The CP framework shows particularly strong improvements in low-resource and few-shot settings, demonstrating enhanced robustness and reduced overfitting to superficial cues.
• The method improves both context understanding (OnlyC) and mention-based type utilization (OnlyM), leading to better overall performance on C+M settings.
• The model generalizes well across domains, performing strongly on biomedical datasets like ChemProt despite being pre-trained on Wikipedia.
• Case studies confirm that the model learns to focus on relational patterns in context rather than memorizing entity identities, validating the effectiveness of the masking and contrastive objective.