[Paper Review] BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model

Hongyi Yuan, Zheng Yuan+4 more|arXiv (Cornell University)|Apr 8, 2022

Topic Modeling4 citations

TL;DR

This paper introduces BioBART, a biomedical generative language model pre-trained on PubMed abstracts using a denoising autoencoding objective. BioBART outperforms BART and state-of-the-art methods on multiple biomedical natural language generation tasks, including dialogue, summarization, entity linking, and named entity recognition, while ablation studies reveal that sentence permutation during pretraining harms performance.

ABSTRACT

Pretrained language models have served as important backbones for natural language processing. Recently, in-domain pretraining has been shown to benefit various domain-specific downstream tasks. In the biomedical domain, natural language generation (NLG) tasks are of critical importance, while understudied. Approaching natural language understanding (NLU) tasks as NLG achieves satisfying performance in the general domain through constrained language generation or language prompting. We emphasize the lack of in-domain generative language models and the unsystematic generative downstream benchmarks in the biomedical domain, hindering the development of the research community. In this work, we introduce the generative language model BioBART that adapts BART to the biomedical domain. We collate various biomedical language generation tasks including dialogue, summarization, entity linking, and named entity recognition. BioBART pretrained on PubMed abstracts has enhanced performance compared to BART and set strong baselines on several tasks. Furthermore, we conduct ablation studies on the pretraining tasks for BioBART and find that sentence permutation has negative effects on downstream tasks.

Motivation & Objective

• To address the lack of in-domain generative language models for biomedical NLP by adapting BART to the biomedical domain.
• To collate and release a comprehensive benchmark of existing biomedical natural language generation tasks for standardized evaluation.
• To investigate the impact of different pretraining objectives, particularly sentence permutation, on downstream biomedical NLG tasks.
• To establish strong baselines for biomedical NLG through systematic pretraining and evaluation on diverse biomedical datasets.
• To improve biomedical language understanding via generative modeling, especially for tasks like entity linking and named entity recognition.

Proposed method

• Pretraining BioBART using the BART architecture with a denoising autoencoding objective on PubMed abstracts.
• Using only the text-infilling pretraining task, without sentence permutation, to maintain performance on downstream NLG tasks.
• Fine-tuning BioBART on multiple biomedical NLG benchmarks, including dialogue (CovidDialog), summarization (MeQSum, iCliniq), entity linking (AAP, BC5CDR), and NER (ShARe13, GENIA).
• Conducting ablation studies by comparing models pre-trained with and without the sentence permutation objective to evaluate its impact.
• Evaluating performance using standard metrics such as Rouge, BLEU, F1, and Recall@1 across diverse biomedical NLP tasks.
• Releasing code, checkpoints, and a curated benchmark of biomedical NLG datasets to support reproducibility and future research.

Experimental results

Research questions

• RQ1 Does pretraining a generative language model like BART on biomedical text improve performance on downstream biomedical NLG tasks?
• RQ2 How does the inclusion of sentence permutation during pretraining affect the performance of generative models on biomedical NLG benchmarks?
• RQ3 Can a domain-adapted generative model like BioBART serve as a strong baseline for biomedical dialogue, summarization, and entity linking tasks?
• RQ4 To what extent does in-domain pretraining on PubMed abstracts mitigate domain shift compared to general-domain models like BART?
• RQ5 Can generative modeling effectively handle biomedical NER and entity linking tasks, and how does it compare to discriminative state-of-the-art methods?

Key findings

• BioBART-large outperforms BART-large by 1.71 Rouge-2 and 0.03 Rouge-L on the CovidDialog dialogue generation benchmark.
• On the MeQSum summarization dataset, BioBART-large achieves 1.93/1.31/2.1 Rouge-1/2/L over BART-large, surpassing the current state-of-the-art.
• For entity linking, BioBART improves Recall@1 by 0.4, 1.67, and 1.36 points on AAP, BC5CDR, and COMETA respectively, exceeding discriminative SOTA methods.
• On NER, BioBART-large improves entity-level F1 by 1.06 on ShARe13 and 1.00 on GENIA, demonstrating strong generative performance despite a gap to the best discriminative models.
• Ablation studies show that excluding the sentence permutation pretraining task leads to better performance, as its inclusion degrades results across multiple NLG tasks.
• Qualitative examples confirm BioBART’s improved biomedical common sense and document understanding, especially in correcting factual errors present in BART’s generations.