[论文解读] Learning structures of the French clinical language:development and validation of word embedding models using 21 million clinical reports from electronic health records

Basile Dura, Charline Jean+5 more|arXiv (Cornell University)|Jul 26, 2022

Machine Learning in Healthcare被引用 6

一句话总结

本研究基于巴黎公立医院集团（APHP）电子健康记录中的2100万份法语临床报告，开发并验证了两个基于CamemBERT的词嵌入模型。在临床NLP任务（APMed）上，模型F1得分提升了3个百分点（达到91%），表明领域特定的预训练能显著提升性能，且从零开始训练与微调相比无显著优势。

ABSTRACT

Background Clinical studies using real-world data may benefit from exploiting clinical reports, a particularly rich albeit unstructured medium. To that end, natural language processing can extract relevant information. Methods based on transfer learning using pre-trained language models have achieved state-of-the-art results in most NLP applications; however, publicly available models lack exposure to speciality-languages, especially in the medical field. Objective We aimed to evaluate the impact of adapting a language model to French clinical reports on downstream medical NLP tasks. Methods We leveraged a corpus of 21M clinical reports collected from August 2017 to July 2021 at the Greater Paris University Hospitals (APHP) to produce two CamemBERT architectures on speciality language: one retrained from scratch and the other using CamemBERT as its initialisation. We used two French annotated medical datasets to compare our language models to the original CamemBERT network, evaluating the statistical significance of improvement with the Wilcoxon test. Results Our models pretrained on clinical reports increased the average F1-score on APMed (an APHP-specific task) by 3 percentage points to 91%, a statistically significant improvement. They also achieved performance comparable to the original CamemBERT on QUAERO. These results hold true for the fine-tuned and from-scratch versions alike, starting from very few pre-training samples. Conclusions We confirm previous literature showing that adapting generalist pre-train language models such as CamenBERT on speciality corpora improves their performance for downstream clinical NLP tasks. Our results suggest that retraining from scratch does not induce a statistically significant performance gain compared to fine-tuning.

研究动机与目标

• 评估在法语临床报告上进行预训练对下游医疗NLP任务的影响。
• 比较在临床数据上对现有CamemBERT模型进行微调与从零开始重新训练的差异。
• 评估学习有意义的临床语言表征所需的最小数据量和训练步数。
• 在两个法语医学NLP基准（APMed和QUAERO）上验证模型性能。
• 确认针对专科语言的预训练是否在法语临床NLP中优于通用模型。

提出的方法

• 预训练了两个基于CamemBERT的模型：一个使用2100万份法语临床报告从零开始训练，另一个基于原始CamemBERT进行微调。
• 使用来自APHP临床数据仓库（EDS）的精选语料库，涵盖39家医院和1100万名患者。
• 采用分层抽样方法平衡文档元类型（如门诊记录、影像报告、手术记录）并减少行政噪声。
• 采用EDS-NLP（一个基于spaCy的开源管道）检测并移除非临床文本污染（如患者权利声明）。
• 在两个标注数据集上微调模型：APMed（药物提及识别及剂量信息）和QUAERO（跨UMLS语义组的医学实体识别）。
• 使用F1得分评估性能，并应用Wilcoxon符号秩检验评估性能提升的统计显著性。

实验结果

研究问题

• RQ1 与通用模型相比，在法语临床报告上进行预训练是否能提升下游医疗NLP任务的性能？
• RQ2 在临床数据上从零开始重新训练与微调现有CamemBERT模型相比，是否存在统计上显著的性能提升？
• RQ3 从零开始学习有效的临床语言表征，需要多少训练步数和多少数据量？
• RQ4 在临床数据上预训练的模型是否能良好泛化到其他医学NLP任务（如在多样化语料上的实体识别）？
• RQ5 当预训练数据有限时，初始化方式（从零开始 vs. 微调）是否会影响下游任务性能？

主要发现

• 在APMed任务上，基于临床报告预训练的模型F1得分提升了3个百分点，达到91%，且通过Wilcoxon检验确认具有统计显著性。
• APMed任务上的性能增益在从零开始训练和微调两种变体中均保持一致，表明关键因素是领域特定的预训练，而非初始化方法。
• 在QUAERO基准上，模型性能与原始CamemBERT相当，表明其在其他医学NLP任务中具有良好的泛化能力。
• 从零开始训练与微调模型之间未发现统计上显著的性能差异，表明微调已足够且更具效率。
• 结果证实，即使数据有限，针对专科临床语料的预训练也能显著提升医疗NLP模型性能，且微调是全量重训练的可行且高效替代方案。