[Paper Review] DocRED: A Large-Scale Document-Level Relation Extraction Dataset

Yuan Yao, Deming Ye+8 more|arXiv (Cornell University)|Jun 14, 2019

Topic Modeling41 references42 citations

TL;DR

DocRED introduces a large-scale, human-annotated document-level relation extraction dataset from Wikipedia and Wikidata, plus a distantly supervised set, to push document-level RE beyond sentence-level methods. It shows existing models struggle on document-scale reasoning and provides benchmarks for supervised and weakly supervised settings.

ABSTRACT

Multiple entities in a document generally exhibit complex inter-sentence relations, and cannot be well handled by existing relation extraction (RE) methods that typically focus on extracting intra-sentence relations for single entity pairs. In order to accelerate the research on document-level RE, we introduce DocRED, a new dataset constructed from Wikipedia and Wikidata with three features: (1) DocRED annotates both named entities and relations, and is the largest human-annotated dataset for document-level RE from plain text; (2) DocRED requires reading multiple sentences in a document to extract entities and infer their relations by synthesizing all information of the document; (3) along with the human-annotated data, we also offer large-scale distantly supervised data, which enables DocRED to be adopted for both supervised and weakly supervised scenarios. In order to verify the challenges of document-level RE, we implement recent state-of-the-art methods for RE and conduct a thorough evaluation of these methods on DocRED. Empirical results show that DocRED is challenging for existing RE methods, which indicates that document-level RE remains an open problem and requires further efforts. Based on the detailed analysis on the experiments, we discuss multiple promising directions for future research.

Motivation & Objective

• Motivate and enable document-level relation extraction by providing a large, manually annotated dataset from Wikipedia/Wikidata.
• Show that document-level RE is significantly harder than sentence-level RE for state-of-the-art models.
• Provide both supervised and distantly supervised data to support diverse learning paradigms.
• Analyze reasoning types, supporting evidence, and model limitations to guide future research.
• Offer benchmarks and baselines to spur development of more capable document-level RE methods.

Proposed method

• Construct DocRED from English Wikipedia and Wikidata with four-stage human annotation: distantly supervised candidate generation, named entity and coreference annotation, entity linking, and relation with supporting evidence labeling.
• Provide 96 relation types spanning diverse domains and 132,375 relational facts across 5,053 documents.
• Create a large distantly supervised dataset by aligning Wikipedia with Wikidata and re-identifying entities with BERT fine-tuned on human-annotated data.
• Adapt existing sentence-level RE models to document-level RE and evaluate under supervised and weakly supervised settings.
• Assess model performance using F1 and AUC, includingIgn F1/Ign AUC to mitigate training/dev/test overlap bias.
• Offer a neural and feature-ablation analysis to understand the impact of entity types, coreference, and distance features.

Experimental results

Research questions

• RQ1 How does document-level RE differ in difficulty from sentence-level RE when evaluated on a large, diverse, human-annotated dataset?
• RQ2 Can existing RE models be effectively adapted to document-level tasks, and what are their limitations?
• RQ3 What is the impact of distantly supervised data on document-level RE performance and reliability?
• RQ4 What kinds of reasoning are required to extract document-level relations, and how can models be designed to handle them?
• RQ5 How well can models predict supporting evidence for extracted relations?

Key findings

• DocRED is larger than prior RE datasets and requires multi-sentence reasoning for a substantial portion of relations.
• Most relations (61.1%) require reasoning beyond simple pattern matching, with logical, coreference, and common-sense reasoning commonly needed.
• Human performance significantly exceeds current models, indicating room for substantial improvement in document-level RE.
• Contextual and long-range encoding (BiLSTM-based architectures) generally outperform CNNs, but no model clearly dominates, highlighting the need for better inter-sentence reasoning.
• Distant supervision helps scale data but introduces label noise; models trained on human-annotated data generally outperform those trained on distantly supervised data.
• Predicting supporting evidence for relations is feasible with neural predictors and improves explainability, though still challenging.