[Paper Review] Document-Level Relation Extraction with Adaptive Thresholding and Localized Context Pooling

Wenxuan Zhou, Kevin Huang+2 more|arXiv (Cornell University)|Oct 21, 2020

Topic Modeling17 citations

TL;DR

This paper proposes ATLOP, a novel document-level relation extraction model that addresses multi-entity and multi-label challenges through adaptive thresholding and localized context pooling. By learning entity-specific thresholds and leveraging pre-trained attention heads to focus on relevant context, ATLOP achieves an F1 score of 63.4 on the DocRED benchmark, outperforming prior state-of-the-art models.

ABSTRACT

Document-level relation extraction (RE) poses new challenges compared to its sentence-level counterpart. One document commonly contains multiple entity pairs, and one entity pair occurs multiple times in the document associated with multiple possible relations. In this paper, we propose two novel techniques, adaptive thresholding and localized context pooling, to solve the multi-label and multi-entity problems. The adaptive thresholding replaces the global threshold for multi-label classification in the prior work with a learnable entities-dependent threshold. The localized context pooling directly transfers attention from pre-trained language models to locate relevant context that is useful to decide the relation. We experiment on three document-level RE benchmark datasets: DocRED, a recently released large-scale RE dataset, and two datasets CDRand GDA in the biomedical domain. Our ATLOP (Adaptive Thresholding and Localized cOntext Pooling) model achieves an F1 score of 63.4, and also significantly outperforms existing models on both CDR and GDA.

Motivation & Objective

• To address the multi-entity and multi-label challenges in document-level relation extraction, where multiple entity pairs and multiple relations per pair complicate classification.
• To eliminate heuristic threshold tuning in multi-label classification by learning instance-specific thresholds for better generalization.
• To enhance entity representations by focusing on context relevant to each entity pair, reducing noise from irrelevant document content.
• To improve performance on document-level RE without relying on explicit graph structures, leveraging pre-trained language model attention directly.
• To achieve state-of-the-art results on benchmark datasets including DocRED, CDR, and GDA in the biomedical domain.

Proposed method

• Adaptive thresholding replaces global thresholds with learnable threshold classes per entity pair, trained via a rank-based loss that pushes positive logits above and negative logits below the threshold.
• Localized context pooling transfers attention heads from pre-trained language models to identify context relevant to a specific entity pair by multiplying attention weights from both entities.
• The method dynamically focuses on tokens that are jointly informative for both entities, such as 'born' and 'died' in the example, improving relation prediction accuracy.
• Entity embeddings are refined using context-aware attention, avoiding the use of fixed, global representations across all entity pairs.
• The model integrates these components into a unified framework that jointly optimizes for relation classification and threshold adaptation.
• The approach avoids constructing document graphs, relying instead on attention transfer and learnable thresholds to model long-range dependencies and multi-relational facts.

Experimental results

Research questions

• RQ1 Can a learnable, entity-pair-specific threshold improve multi-label classification in document-level relation extraction compared to fixed global thresholds?
• RQ2 Can localized context pooling, derived from pre-trained model attention, effectively identify relevant context for each entity pair and reduce noise from irrelevant document content?
• RQ3 Does combining adaptive thresholding and localized context pooling lead to significant performance gains on document-level relation extraction benchmarks?
• RQ4 Is it possible to achieve state-of-the-art results without explicitly modeling document graphs, relying instead on attention transfer and end-to-end learning?
• RQ5 How does the model perform on diverse document-level RE datasets, including general-domain (DocRED) and biomedical (CDR, GDA) benchmarks?

Key findings

• ATLOP achieves an F1 score of 63.4 on the DocRED benchmark, setting a new state-of-the-art performance.
• The model significantly outperforms existing approaches on both the CDR and GDA biomedical datasets, demonstrating strong generalization across domains.
• Adaptive thresholding eliminates the need for manual threshold tuning and improves prediction reliability by learning instance-specific thresholds.
• Localized context pooling successfully identifies relevant context tokens such as 'born' and 'died' for the entity pair (John Stanistreet, Bendigo), as shown in attention visualizations.
• The ablation study confirms that both adaptive thresholding and localized context pooling contribute meaningfully to the model's performance.
• The model achieves SOTA results without relying on document graphs, showing that attention transfer from pre-trained models is sufficient for effective long-range dependency modeling.