[Paper Review] Benchmarking BioRelEx for Entity Tagging and Relation Extraction

Abhinav Bhatt, Kaustubh Dhole|arXiv (Cornell University)|May 31, 2020

Topic Modeling21 references4 citations

TL;DR

This paper benchmarks multiple entity tagging and relation extraction models on the BioRelEx dataset, demonstrating that span-based multi-task architectures like DYGIEPP achieve 4.9% and 6% absolute F-score improvements over prior state-of-the-art in entity tagging and relation extraction, respectively. The study further shows that domain-specific embeddings such as BioELMo significantly boost performance, especially in entity tagging, while multi-task learning with relation and coreference propagation enhances results.

ABSTRACT

Extracting relationships and interactions between different biological entities is still an extremely challenging problem but has not received much attention as much as extraction in other generic domains. In addition to the lack of annotated data, low benchmarking is still a major reason for slow progress. In order to fill this gap, we compare multiple existing entity and relation extraction models over a recently introduced public dataset, BioRelEx of sentences annotated with biological entities and relations. Our straightforward benchmarking shows that span-based multi-task architectures like DYGIE show 4.9% and 6% absolute improvements in entity tagging and relation extraction respectively over the previous state-of-art and that incorporating domain-specific information like embeddings pre-trained over related domains boosts performance.

Motivation & Objective

• To establish a stronger benchmark for entity tagging and relation extraction in biological text using the recently released BioRelEx dataset.
• To evaluate the effectiveness of modern multi-task span-based architectures, such as DYGIE, on domain-specific biomedical text.
• To investigate the impact of domain-specific embeddings and architectural components like relation and coreference propagation on performance.
• To identify the optimal configuration of models and embeddings for BioRelEx, improving upon the previous state-of-the-art.

Proposed method

• The study evaluates a range of models, including BiLSTM-CRF, BiGRU-CRF, CNN-CRF, and the state-of-the-art DYGIEPP, on the BioRelEx validation set.
• It incorporates various embeddings—GloVe, FastText, Byte Pair, Flair, and BioELMo—across different model architectures to assess their impact.
• The authors use ELMo, GloVe, and character-level embeddings as input features and apply multi-task learning with joint optimization of entity tagging, relation extraction, and coreference resolution.
• Coreference and relation propagation mechanisms are systematically evaluated to determine their contribution to span representation learning.
• Hyperparameter tuning is performed, including adjusting the number of hidden units and layers in BiLSTM components, with 400 units and 2 layers found optimal.
• Performance is evaluated using micro-averaged precision, recall, and F1 scores via the official evaluation script from Khachatrian et al. (2019).

Experimental results

Research questions

• RQ1 Can span-based multi-task architectures like DYGIEPP outperform traditional models such as BiLSTM-CRF on the BioRelEx dataset?
• RQ2 To what extent do domain-specific embeddings such as BioELMo improve entity tagging and relation extraction performance on BioRelEx?
• RQ3 How do architectural components like relation and coreference propagation affect model performance on BioRelEx?
• RQ4 Does the use of contextual embeddings like ELMo or BioELMo surpass BERT and SciBERT in this biomedical NLP benchmark?
• RQ5 Are there discrepancies in evaluation due to data conversion issues, and how do they affect performance comparisons?

Key findings

• DYGIEPP achieves a 4.9% absolute improvement in F1 score for entity tagging and a 6% absolute improvement for relation extraction over the previous state-of-the-art on the BioRelEx dataset.
• The use of BioELMo embeddings leads to a 1.7% absolute F1 gain in entity tagging compared to standard ELMo embeddings.
• Coreference propagation increases performance in both entity tagging and relation extraction, while relation propagation shows a stronger positive effect in relation extraction.
• Multi-task architectures with ELMo, GloVe, and character embeddings outperform BERT and SciBERT on this benchmark, contrary to findings on GLUE.
• The evaluation script's data conversion process may cause minor discrepancies, limiting the maximum F1 score to approximately 91.2% on the training set, though relative model rankings remain stable.
• Increasing the number of BiLSTM layers and hidden units improves performance, with 400 units and 2 layers identified as optimal for the best results.