[论文解读] Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction

Laila Rasmy, Yang Xiang+3 more|arXiv (Cornell University)|May 22, 2020

Machine Learning in Healthcare参考文献 43被引用 61

一句话总结

Med-BERT 将 BERT 框架适配到大规模结构化的 EHR 数据，以生成上下文嵌入，提升疾病预测性能，尤其在小型微调集上。

ABSTRACT

Deep learning (DL) based predictive models from electronic health records (EHR) deliver impressive performance in many clinical tasks. Large training cohorts, however, are often required to achieve high accuracy, hindering the adoption of DL-based models in scenarios with limited training data size. Recently, bidirectional encoder representations from transformers (BERT) and related models have achieved tremendous successes in the natural language processing domain. The pre-training of BERT on a very large training corpus generates contextualized embeddings that can boost the performance of models trained on smaller datasets. We propose Med-BERT, which adapts the BERT framework for pre-training contextualized embedding models on structured diagnosis data from 28,490,650 patients EHR dataset. Fine-tuning experiments are conducted on two disease-prediction tasks: (1) prediction of heart failure in patients with diabetes and (2) prediction of pancreatic cancer from two clinical databases. Med-BERT substantially improves prediction accuracy, boosting the area under receiver operating characteristics curve (AUC) by 2.02-7.12%. In particular, pre-trained Med-BERT substantially improves the performance of tasks with very small fine-tuning training sets (300-500 samples) boosting the AUC by more than 20% or equivalent to the AUC of 10 times larger training set. We believe that Med-BERT will benefit disease-prediction studies with small local training datasets, reduce data collection expenses, and accelerate the pace of artificial intelligence aided healthcare.

研究动机与目标

• 在结构化 EHR 数据中使用预训练的上下文嵌入以在标注数据有限时改进疾病预测。
• 利用大规模 EHR 语料库来预训练可转移到下游临床预测任务的表征。
• 在数据稀疏的环境中展示对具体疾病的预测准确性提升。

提出的方法

• 将 BERT 框架适配到来自非常大规模 EHR 队列的结构化诊断数据（28,490,650 名患者）。
• 在结构化 EHR 数据上预训练上下文嵌入并在下游任务上进行微调。
• 在两个疾病预测任务上进行评估：来自两个临床数据库的糖尿病患者中的心衰和胰腺癌。
• 将性能与非预训练基线进行比较，以量化预训练带来的提升。
• 报告在不同微调数据量下 AUC 的改进，突出小数据的益处。

实验结果

研究问题

• RQ1 在结构化 EHR 数据上预训练的 Med-BERT 是否能相比非预训练模型提高疾病预测性能？
• RQ2 当微调数据稀缺时（例如 300–500 份样本），预训练如何影响性能？
• RQ3 这些提升是否在不同疾病和不同临床数据库之间具有普遍性？

主要发现

• Med-BERT 显著提升预测准确性，在各任务中将 AUC 提升了 2.02%–7.12%。
• 预训练的 Med-BERT 在非常小的微调集（300–500 份样本）上显著提升性能，将 AUC 提升超过 20%。
• 小数据的增益相当于使用更大训练集（10 倍）的效果。
• 该方法支持使用较小的本地数据集进行疾病预测研究并可降低数据收集成本。