[Paper Review] Improving Relation Extraction by Pre-trained Language Representations

Christoph Alt, Marc P. Hübner+1 more|arXiv (Cornell University)|Jun 7, 2019

Topic ModelingComputer Science29 references53 citations

TL;DR

TRE uses pre-trained language representations within a Transformer framework to perform relation extraction, achieving state-of-the-art results on TACRED and SemEval 2010 Task 8 and showing improved sample efficiency.

ABSTRACT

Current state-of-the-art relation extraction methods typically rely on a set of lexical, syntactic, and semantic features, explicitly computed in a pre-processing step. Training feature extraction models requires additional annotated language resources, which severely restricts the applicability and portability of relation extraction to novel languages. Similarly, pre-processing introduces an additional source of error. To address these limitations, we introduce TRE, a Transformer for Relation Extraction, extending the OpenAI Generative Pre-trained Transformer [Radford et al., 2018]. Unlike previous relation extraction models, TRE uses pre-trained deep language representations instead of explicit linguistic features to inform the relation classification and combines it with the self-attentive Transformer architecture to effectively model long-range dependencies between entity mentions. TRE allows us to learn implicit linguistic features solely from plain text corpora by unsupervised pre-training, before fine-tuning the learned language representations on the relation extraction task. TRE obtains a new state-of-the-art result on the TACRED and SemEval 2010 Task 8 datasets, achieving a test F1 of 67.4 and 87.1, respectively. Furthermore, we observe a significant increase in sample efficiency. With only 20% of the training examples, TRE matches the performance of our baselines and our model trained from scratch on 100% of the TACRED dataset. We open-source our trained models, experiments, and source code.

Motivation & Objective

• Motivate reducing reliance on explicit linguistic feature engineering for relation extraction.
• Introduce TRE, a Transformer-based model that uses pre-trained language representations for relation classification.
• Demonstrate that unsupervised pre-training improves performance and sample efficiency on standard benchmarks.

Proposed method

• Use a decoder-only Transformer architecture to process structured input for relation extraction.
• Adopt an input representation with BPE subword tokens and task-specific delimiters to encode relation arguments and the sentence.
• Pre-train the model on a language modeling objective with plain text, then fine-tune on relation extraction with an auxiliary LM objective during fine-tuning.
• Fine-tune by predicting relation labels from the final Transformer state using a linear softmax classifier, optionally weighting the LM objective (lambda).
• Experiment with entity masking strategies (UNK, NE, GR, NE+GR) to study generalization and regularization effects.

Experimental results

Research questions

• RQ1 Does pre-training with language representations improve relation extraction performance without explicit linguistic features?
• RQ2 How does TRE compare to state-of-the-art models on TACRED and SemEval 2010 Task 8?
• RQ3 What is the impact of entity masking on generalization and sample efficiency?
• RQ4 How sample-efficient is TRE relative to baselines when training data is limited?

Key findings

• TRE achieves state-of-the-art F1 on TACRED (67.4) and SemEval 2010 Task 8 (87.1).
• Pre-trained language representations substantially improve performance, especially when entities are not masked, indicating regularization benefits.
• Entity masking (NE+GR) yields strong performance, suggesting language representations capture informative features similar to entity type and role information.
• TRE shows marked sample efficiency, reaching high F1 with as little as 20% of TACRED training data.
• Unmasked entities can lead to overfitting; masking strategies help generalization to unseen entities.