[Paper Review] A Frustratingly Easy Approach for Entity and Relation Extraction

Zexuan Zhong, Danqi Chen|arXiv (Cornell University)|Oct 24, 2020

Topic ModelingComputer Science37 references35 citations

TL;DR

PURE presents a simple pipelined two-encoder approach with typed markers for end-to-end entity and relation extraction, achieving new state-of-the-art results on ACE04, ACE05, and SciERC, plus an efficient inference approximation.

ABSTRACT

End-to-end relation extraction aims to identify named entities and extract relations between them. Most recent work models these two subtasks jointly, either by casting them in one structured prediction framework, or performing multi-task learning through shared representations. In this work, we present a simple pipelined approach for entity and relation extraction, and establish the new state-of-the-art on standard benchmarks (ACE04, ACE05 and SciERC), obtaining a 1.7%-2.8% absolute improvement in relation F1 over previous joint models with the same pre-trained encoders. Our approach essentially builds on two independent encoders and merely uses the entity model to construct the input for the relation model. Through a series of careful examinations, we validate the importance of learning distinct contextual representations for entities and relations, fusing entity information early in the relation model, and incorporating global context. Finally, we also present an efficient approximation to our approach which requires only one pass of both entity and relation encoders at inference time, achieving an 8-16$ imes$ speedup with a slight reduction in accuracy.

Motivation & Objective

• Revisit end-to-end relation extraction to assess whether a pipeline with separate encoders can outperform joint models.
• Demonstrate that distinct contextual representations for entities and relations are beneficial.
• Show that early fusion of entity information and typed markers improves relation understanding.
• Propose an efficient inference approximation to accelerate runtime without large accuracy loss.

Proposed method

• Split into two independent encoders: an entity model for span-level entity prediction and a relation model for pairwise span relation prediction.
• In the relation model, insert typed markers around subject and object spans to create input that yields pair-specific contextual representations.
• Fuse entity information (boundary and type) at the input layer of the relation model to improve discrimination of relations.
• optionally extend input with cross-sentence context within a fixed window to utilize broader context.
• Train with standard cross-entropy losses for both tasks; use gold entities to train the relation model and predicted entities at inference time.
• Provide an efficient batch-based approximation that reuses text-token computations by tying marker positions to span token starts/ends and constraining attention, enabling batching of multiple span-pair inputs at inference time.

Experimental results

Research questions

• RQ1 Does a two-encoder, pipelined design outperform joint models in end-to-end relation extraction on standard benchmarks?
• RQ2 Do distinct contextual representations for entities and relations and early fusion of entity information yield performance gains over shared representations?
• RQ3 Can an efficient inference approximation maintain competitive accuracy while significantly speeding up end-to-end relation extraction?
• RQ4 What is the impact of cross-sentence context on entity recognition and relation extraction in this framework?

Key findings

• The proposed PURE pipeline with separate encoders outperforms previous joint models on ACE04, ACE05, and SciERC in relation F1 when using the same pre-trained encoders.
• Learning distinct contextual representations for entities and relations is advantageous over sharing representations.
• Early fusion of entity boundary and type information at the relation model input significantly improves relation understanding.
• Cross-sentence context provides additional gains for both entity and relation predictions.
• An approximation for batch inference yields 8–16x speedups with only a small drop in relation F1 (e.g., ~1% on ACE05).