[Paper Review] Simple BERT Models for Relation Extraction and Semantic Role Labeling

Peng Shi, Jimmy Lin|arXiv (Cornell University)|Apr 10, 2019

Natural Language Processing TechniquesComputer Science23 references194 citations

TL;DR

The paper shows simple BERT-based architectures for relation extraction and semantic role labeling that achieve state-of-the-art or strong results without external lexical or syntactic features.

ABSTRACT

We present simple BERT-based models for relation extraction and semantic role labeling. In recent years, state-of-the-art performance has been achieved using neural models by incorporating lexical and syntactic features such as part-of-speech tags and dependency trees. In this paper, extensive experiments on datasets for these two tasks show that without using any external features, a simple BERT-based model can achieve state-of-the-art performance. To our knowledge, we are the first to successfully apply BERT in this manner. Our models provide strong baselines for future research.

Motivation & Objective

• Demonstrate that BERT-based models without linguistic features can achieve strong performance on relation extraction and SRL.
• Propose simple, entity-aware and predicate-aware architectures built on BERT.
• Evaluate on standard benchmarks to establish competitive baselines for future work.

Proposed method

• Encode sentences with BERT, replacing entity mentions with masks to create entity-aware inputs for relation extraction.
• Concatenate BERT contextual embeddings with position embeddings for subject/object and pass through BiLSTM and MLP for relation prediction.
• For SRL, append predicate indicator embeddings to BERT representations and use a BiLSTM with an MLP for argument labeling.
• Unify span-based and dependency-based SRL within a single framework by using predicate-aware input sequences and BIO tagging.
• Evaluate using standard datasets (TACRED for RE; CoNLL 2005/2012, 2009 for SRL) and compare with state-of-the-art models.

Experimental results

Research questions

• RQ1 Can BERT-based models achieve state-of-the-art performance on relation extraction without external linguistic features?
• RQ2 Can simple architectures built on top of BERT effectively handle semantic role labeling for both span-based and dependency-based formulations?
• RQ3 How do entity/predicate masking and indicator embeddings affect task performance?

Key findings

• On TACRED, BERT-based models with simple architectures outperform several feature-rich baselines.
• BERT-LSTM-base achieves 73.3 P, 63.10 R, 67.8 F1 on TACRED test when compared to prior single-model methods.
• BERT-LSTM-large and variants achieve strong results on SRL tasks, often surpassing existing baselines without linguistic features.
• For SRL dependency-based results, BERT-LSTM-large yields notable F1 gains over previous state-of-the-art methods in end-to-end evaluation.