[Paper Review] RuBioRoBERTa: a pre-trained biomedical language model for Russian language biomedical text mining

Alexander Yalunin, Alexander I. Nesterov+1 more|arXiv (Cornell University)|Apr 8, 2022

Biomedical Text Mining and Ontologies4 citations

TL;DR

This paper introduces RuBioRoBERTa, a biomedical language model pre-trained on 1.2 billion words of Russian medical text from CyberLeninka, achieving state-of-the-art performance on the RuMedBench benchmark across text classification, question answering, NLI, NER, and symptom prediction tasks. Fine-tuning on downstream tasks shows it outperforms both general-domain models and prior SOTA, with a 71.54% overall score and surpassing human performance on drug review classification.

ABSTRACT

This paper presents several BERT-based models for Russian language biomedical text mining (RuBioBERT, RuBioRoBERTa). The models are pre-trained on a corpus of freely available texts in the Russian biomedical domain. With this pre-training, our models demonstrate state-of-the-art results on RuMedBench - Russian medical language understanding benchmark that covers a diverse set of tasks, including text classification, question answering, natural language inference, and named entity recognition.

Motivation & Objective

• To develop a specialized pre-trained language model for Russian biomedical text mining, addressing the lack of domain-adapted NLP resources in non-English languages.
• To improve performance on diverse medical NLP tasks—such as text classification, NER, and natural language inference—by fine-tuning a model pre-trained on a large corpus of Russian medical literature.
• To establish a new benchmark for Russian medical NLP by releasing RuBioRoBERTa and RuBioBERT as strong baselines for future research.
• To validate the effectiveness of domain-specific pre-training by comparing biomedical fine-tuned models against general-domain counterparts on the RuMedBench benchmark.

Proposed method

• Pre-trained three BERT-based models—RuBERT, ruBERT, and ruRoBERTa—on a general-domain Russian corpus before further pre-training on a curated biomedical corpus.
• Constructed a biomedical corpus of 338,000 Russian medical articles (1.2B words) from CyberLeninka, covering fundamental medicine, clinical medicine, biotechnology, and health sciences.
• Performed additional pre-training on the biomedical corpus for one epoch using 3×V100 GPUs, with a learning rate warm-up to 5e-5 and cosine decay, over 12 days.
• Fine-tuned the resulting RuBioRoBERTa model on five downstream tasks from the RuMedBench benchmark using standard hyperparameters: 10 epochs, batch size 32, peak LR 3e-5 with cosine annealing.
• Utilized HuggingFace for model training and inference, with vocabulary sizes of 120K (BERT) and 50K (RoBERTa), and model parameters ranging from 178M to 355M.
• Evaluated performance on text classification, symptom recommendation, yes/no question answering, natural language inference, and named entity recognition using standard metrics.

Experimental results

Research questions

• RQ1 Can additional pre-training on a large-scale Russian biomedical corpus significantly improve performance on downstream medical NLP tasks?
• RQ2 How does a domain-adapted model like RuBioRoBERTa compare to general-domain models (e.g., ruRoBERTa) and previous SOTA on the RuMedBench benchmark?
• RQ3 To what extent can a Russian biomedical language model outperform human experts in specific medical text understanding tasks?
• RQ4 What qualitative improvements does biomedical pre-training bring in recognizing medical abbreviations, diseases, and symptoms?

Key findings

• RuBioRoBERTa achieved the highest overall score of 71.54% on the RuMedBench benchmark, outperforming the previous SOTA by 4.34 percentage points.
• On the Top3 task (ICD-10 code prediction), RuBioRoBERTa achieved 46.72% accuracy and 72.87% hit@3, surpassing the previous best of 47.45%/70.44%.
• In the SymRec task (symptom recommendation), RuBioRoBERTa achieved 44.01% accuracy and 58.95% hit@3, significantly outperforming ruRoBERTa’s 40.92%/54.37%.
• RuBioRoBERTa achieved 82.77% accuracy on the NLI task and 97.19% F1 on NER, both exceeding the previous SOTA, with the NER F1 surpassing human performance (76.18%).
• The model correctly identified medical abbreviations such as ACA (Acute Cerebrovascular Accident) in NLI tasks, demonstrating improved understanding of clinical terminology.
• Qualitative analysis confirmed that RuBioRoBERTa better recognized disease names (e.g., pneumonia) and provided more relevant symptom recommendations not explicitly mentioned in input texts.