[Paper Review] A Multi-Task Learning Framework for Extracting Drugs and Their Interactions from Drug Labels

Tung Tran, Ramakanth Kavuluru+1 more|arXiv (Cornell University)|May 17, 2019

Biomedical Text Mining and OntologiesBiochemistry, Genetics and Molecular Biology20 references3 citations

TL;DR

This paper presents a multi-task learning framework that jointly performs named entity recognition (NER) and drug-drug interaction (DDI) relation extraction from drug labels using a deep neural network with voting-based ensemble learning. The system achieved second place in both NER (33.00% and 38.25% F1 on Test Sets 1 and 2) and relation extraction (21.59% and 23.55% F1), demonstrating improved recall through ensemble modeling despite limited gains from large external datasets.

ABSTRACT

Preventable adverse drug reactions as a result of medical errors present a growing concern in modern medicine. As drug-drug interactions (DDIs) may cause adverse reactions, being able to extracting DDIs from drug labels into machine-readable form is an important effort in effectively deploying drug safety information. The DDI track of TAC 2018 introduces two large hand-annotated test sets for the task of extracting DDIs from structured product labels with linkage to standard terminologies. Herein, we describe our approach to tackling tasks one and two of the DDI track, which corresponds to named entity recognition (NER) and sentence-level relation extraction respectively. Namely, our approach resembles a multi-task learning framework designed to jointly model various sub-tasks including NER and interaction type and outcome prediction. On NER, our system ranked second (among eight teams) at 33.00% and 38.25% F1 on Test Sets 1 and 2 respectively. On relation extraction, our system ranked second (among four teams) at 21.59% and 23.55% on Test Sets 1 and 2 respectively.

Motivation & Objective

• To address the challenge of extracting drug-drug interactions (DDIs) from structured product labels (SPLs) to improve drug safety and prevent adverse drug reactions.
• To jointly model named entity recognition (NER) for precipitants, triggers, and effects, along with interaction type and outcome prediction.
• To develop a robust deep learning system that generalizes across diverse drug label structures and annotation styles.
• To evaluate the impact of training data scale and annotation consistency on model performance in real-world clinical NLP tasks.

Proposed method

• A single deep neural network model with shared and task-specific layers for joint NER and relation extraction.
• A voting-based ensemble of ten independently trained models to improve prediction stability and recall by aggregating predictions based on vote frequency.
• Pre/post-processing rules to handle coordinated modifiers (e.g., 'X and Y inducers') by splitting single entities into multiple constituents during inference.
• Use of external NLM-180 dataset (180 drug labels) alongside the official Training-22 set to enhance model generalization.
• Model training with cross-entropy loss for NER and multi-label classification for interaction type and outcome prediction.
• Post-processing to resolve overlapping predictions by selecting higher-vote annotations and discarding conflicting ones.

Experimental results

Research questions

• RQ1 Can a multi-task learning framework jointly improve performance on named entity recognition and drug-drug interaction relation extraction from drug labels?
• RQ2 How does ensemble learning affect recall and stability in NLP models for clinical text extraction?
• RQ3 To what extent does using a large external dataset (NLM-180) improve performance on smaller, official test sets despite potential annotation drift?
• RQ4 Why does performance gain from large-scale training data diminish on test sets, especially when annotation guidelines evolve over time?
• RQ5 How do differences in annotation quality and consistency across datasets impact model generalization and evaluation metrics?

Key findings

• The voting-based ensemble model achieved second place in both NER and relation extraction tasks, with F1 scores of 33.00% and 38.25% on Test Sets 1 and 2, respectively.
• The ensemble system improved recall significantly compared to single models, while maintaining similar precision, indicating better stability and robustness.
• Despite an 800% increase in training data from NLM-180, performance gains on test sets were limited, suggesting potential annotation drift or quality differences.
• The top-performing system (Tang et al.) used significantly more training data (1,148 manually annotated sentences), making direct comparison difficult.
• The authors observed that their system performed better on denser text (Test Set 1) than on sparser text (Test Set 2), indicating sensitivity to data sparsity.
• Pre/post-processing rules effectively handled coordinated modifier patterns like 'X and Y inducers' by splitting them into distinct entities during inference.