[Paper Review] CamemBERT-bio: Leveraging Continual Pre-training for Cost-Effective Models on French Biomedical Data

Rian Touchent, Laurent Romary+1 more|arXiv (Cornell University)|Jun 27, 2023

Topic Modeling4 citations

TL;DR

This paper introduces CamemBERT-bio, a publicly available French biomedical language model fine-tuned via continual pre-training on a 413-million-word French biomedical corpus. The model achieves a 2.54-point average F1-score improvement on named entity recognition tasks, demonstrating the effectiveness of domain-specific pre-training for French clinical NLP despite recent claims to the contrary.

ABSTRACT

Clinical data in hospitals are increasingly accessible for research through clinical data warehouses. However these documents are unstructured and it is therefore necessary to extract information from medical reports to conduct clinical studies. Transfer learning with BERT-like models such as CamemBERT has allowed major advances for French, especially for named entity recognition. However, these models are trained for plain language and are less efficient on biomedical data. Addressing this gap, we introduce CamemBERT-bio, a dedicated French biomedical model derived from a new public French biomedical dataset. Through continual pre-training of the original CamemBERT, CamemBERT-bio achieves an improvement of 2.54 points of F1-score on average across various biomedical named entity recognition tasks, reinforcing the potential of continual pre-training as an equally proficient yet less computationally intensive alternative to training from scratch. Additionally, we highlight the importance of using a standard evaluation protocol that provides a clear view of the current state-of-the-art for French biomedical models.

Motivation & Objective

• To address the poor performance of general-purpose French language models like CamemBERT on biomedical text due to domain mismatch.
• To create a publicly available, privacy-compliant French biomedical language model for use across healthcare institutions.
• To demonstrate that continual pre-training on domain-specific French biomedical text significantly improves performance on named entity recognition tasks.
• To advocate for standardized evaluation protocols to ensure fair and reproducible benchmarking of French biomedical models.

Proposed method

• Continual pre-training of the CamemBERT base model on a newly curated French biomedical corpus (biomed-fr) containing 413 million words from drug leaflets and scientific literature.
• The pre-training corpus was constructed from French clinical reports, drug information sheets, and biomedical publications, ensuring domain relevance.
• The model was fine-tuned on multiple French biomedical named entity recognition benchmarks using standard NLP evaluation metrics.
• Evaluation followed a standardized protocol using exact match character offsets to avoid pre-processing bias, aligning with CLEF eHealth's BRATeval framework.
• The training was conducted on two Tesla V100 GPUs over ~39 hours, with estimated CO2 emissions of 0.84 kg CO2 eq.
• A comparative analysis was performed against recent models like DrBERT and BioBERT to assess performance and methodological consistency.

Experimental results

Research questions

• RQ1 Can continual pre-training of a French language model on domain-specific biomedical text significantly improve performance on French biomedical NLP tasks?
• RQ2 Why do recent claims suggesting that continual pre-training on French biomedical data is ineffective require re-evaluation?
• RQ3 To what extent does using a standardized evaluation protocol impact the interpretation of model performance in French biomedical NLP?
• RQ4 How does CamemBERT-bio compare to other state-of-the-art models like DrBERT and BioBERT in terms of F1-score and class-wise performance?
• RQ5 What role does the 'O' class (non-entity tokens) play in determining model ranking, and how does it affect performance interpretation?

Key findings

• CamemBERT-bio achieves an average F1-score improvement of 2.54 points across multiple French biomedical named entity recognition benchmarks compared to the base CamemBERT model.
• The model establishes a new state-of-the-art performance on the evaluated French biomedical NLP tasks, outperforming both general-purpose and specialized models in the tested settings.
• Despite claims of performance degradation from continual pre-training, this study demonstrates that such results may stem from flawed evaluation methodologies rather than inherent model limitations.
• The DrBERT model outperforms CamemBERT-bio in macro F1 when evaluated with the 'entity-without-O' protocol, highlighting the critical influence of evaluation design on model ranking.
• The study underscores the importance of standardized evaluation protocols—such as BRATeval's exact match with character offsets—to ensure fair and reproducible benchmarking.
• The environmental impact of pre-training CamemBERT-bio was minimal, with only 0.84 kg CO2 eq emitted, indicating cost-effective and sustainable model adaptation.