[Paper Review] MT-BioNER: Multi-task Learning for Biomedical Named Entity Recognition using Deep Bidirectional Transformers

Muhammad Raza Khan, Morteza Ziyadi+1 more|arXiv (Cornell University)|Jan 24, 2020

Topic Modeling19 references34 citations

TL;DR

MT-BioNER presents a multi-task transformer-based slot tagging model that shares BioBERT/BERT layers across multiple biomedical NER datasets, improving recall and overall F1 while reducing training and inference time compared to single-task approaches.

ABSTRACT

Conversational agents such as Cortana, Alexa and Siri are continuously working on increasing their capabilities by adding new domains. The support of a new domain includes the design and development of a number of NLU components for domain classification, intents classification and slots tagging (including named entity recognition). Each component only performs well when trained on a large amount of labeled data. Second, these components are deployed on limited-memory devices which requires some model compression. Third, for some domains such as the health domain, it is hard to find a single training data set that covers all the required slot types. To overcome these mentioned problems, we present a multi-task transformer-based neural architecture for slot tagging. We consider the training of a slot tagger using multiple data sets covering different slot types as a multi-task learning problem. The experimental results on the biomedical domain have shown that the proposed approach outperforms the previous state-of-the-art systems for slot tagging on the different benchmark biomedical datasets in terms of (time and memory) efficiency and effectiveness. The output slot tagger can be used by the conversational agent to better identify entities in the input utterances.

Motivation & Objective

• Motivate improving Biomedical Named Entity Recognition (BioNER) for domain-specific conversational agents with limited labeled data.
• Propose a multi-task learning architecture that shares pre-trained language model layers across datasets while using dataset-specific output heads.
• Demonstrate performance gains in precision, recall, and F1 on benchmark biomedical datasets and analyze training/inference efficiency.

Proposed method

• Use BioBERT as shared encoder across tasks (datasets).
• Attach task-specific linear output layers for each dataset to learn dataset-specific entity types.
• Train with a multi-task objective that sums per-dataset cross-entropy losses, with equal dataset contribution (lambda_i = 1).
• Investigate two transfer-learning variants: freezing shared layers vs fine-tuning the entire network.
• Compare against state-of-the-art BioNER models and report training/inference time and model size.
• Explore training schemas (Algorithm I vs Algorithm II) and domain adaptation using BioBERT vs general BERT base.

Experimental results

Research questions

• RQ1 Does multi-task training with a shared transformer backbone improve BioNER performance across multiple biomedical datasets compared to single-task or other multi-task baselines?
• RQ2 What are the effects of fine-tuning strategy (shared layer fine-tuning vs freezing) on performance and training efficiency?
• RQ3 How does domain-specific pretraining (BioBERT) affect results versus general BERT base in multi-task BioNER?
• RQ4 What are the trade-offs in training/inference time and model size for a multi-task BioNER model?

Key findings

• MT-BioNER trained on three datasets achieves higher recall and F1 than competing baselines.
• MT-BioNER trained on four datasets shows improved recall and F1 over the MTM-CW multi-task approach.
• Using BioBERT as the shared encoder yields strong domain-specific representations for biomedical NER.
• Fine-tuning the shared language model alongside task-specific layers is crucial for strong performance; freezing the shared layers leads to poor results (F1 < 60%).
• The model demonstrates faster training and inference times compared to a recent multi-task baseline, with a larger but single multi-task model footprint (430 MB).
• Domain adaptation results indicate BioBERT-based MT-BioNER outperforms or matches general BERT-based setups with some trade-offs in training iterations.