[论文解读] Feature Robustness in Non-stationary Health Records: Caveats to Deployable Model Performance in Common Clinical Machine Learning Tasks

Bret Nestor, Matthew B. A. McDermott+6 more|arXiv (Cornell University)|Aug 2, 2019

Machine Learning in Healthcare参考文献 42被引用 54

一句话总结

该论文表明，在历史电子病历(EHR)数据上训练的ML模型在面对未来数据时难以泛化，因为非平稳性，但将特征聚合成临床概念显著降低性能衰减。

ABSTRACT

When training clinical prediction models from electronic health records (EHRs), a key concern should be a model's ability to sustain performance over time when deployed, even as care practices, database systems, and population demographics evolve. Due to de-identification requirements, however, current experimental practices for public EHR benchmarks (such as the MIMIC-III critical care dataset) are time agnostic, assigning care records to train or test sets without regard for the actual dates of care. As a result, current benchmarks cannot assess how well models trained on one year generalise to another. In this work, we obtain a Limited Data Use Agreement to access year of care for each record in MIMIC and show that all tested state-of-the-art models decay in prediction quality when trained on historical data and tested on future data, particularly in response to a system-wide record-keeping change in 2008 (0.29 drop in AUROC for mortality prediction, 0.10 drop in AUROC for length-of-stay prediction with a random forest classifier). We further develop a simple yet effective mitigation strategy: by aggregating raw features into expert-defined clinical concepts, we see only a 0.06 drop in AUROC for mortality prediction and a 0.03 drop in AUROC for length-of-stay prediction. We demonstrate that this aggregation strategy outperforms other automatic feature preprocessing techniques aimed at increasing robustness to data drift. We release our aggregated representations and code to encourage more deployable clinical prediction models.

研究动机与目标

• 动机并量化时序漂移如何影响在去识别化EHR数据上训练的临床预测模型。
• 评估多种输入表示、模型和训练策略对年度间变化的鲁棒性。
• 识别能够提高对未来数据批次泛化能力的缓解策略。

提出的方法

• 在有限数据使用协议下，使用MIMIC-III的就诊年份数据来评估时序泛化。
• 比较四种表示：Raw、PCA、CUI Code Spanning 与 Clinical Aggregations。
• 在三种训练策略（Year-Agnostic、Prior Year、Full History）下，对四种模型LR、RF、LSTM、GRU-D进行基准测试。
• 在ICU数据的前24小时内评估两项二元任务：ICU死亡率和长住院时间。
• 使用AUROC衡量性能，并在非平稳环境下报告年度间下降。

实验结果

研究问题

• RQ1 在MIMIC-III中，当模型在历史数据上训练、在未来数据上评估时，性能如何下降？
• RQ2 哪些输入表示和训练策略最能抵抗临床概念漂移？
• RQ3 与自动预处理方法相比，基于临床动机的特征聚合是否能提高鲁棒性？
• RQ4 EHR系统变更（CareVue到MetaVision）对跨年度预测性能有何影响？

主要发现

• 原始表示在2008年EHR切换后表现显著下降（使用RF时死亡率AUROC下降最多0.29）。
• 临床聚合表示在全历史训练下将AUROC下降降至0.06（死亡）和0.03（LOS）。
• PCA和自动CUIs对年度间波动的鲁棒性有限，临床聚合在各模型上优于它们。
• GRU-D随着时间有一定恢复，表明缺失数据处理有助于漂移鲁棒性，但聚合仍然更优。
• 以年为无关的评估高估了性能，尤其是原始特征，凸显了公开基准的可部署性差距。
• 这些任务的模型性能很快饱和，表明可能需要转向更困难的问题或超出181个实验/生命体征特征的新信号。