[论文解读] Uncertainty Quantification on Clinical Trial Outcome Prediction

Tianyi Chen, Yingzhou Lu+5 more|arXiv (Cornell University)|Jan 7, 2024

Machine Learning in Healthcare被引用 6

一句话总结

该论文将 selective classification 引入 Hierarchical Interaction Network (HINT)，以量化临床试验结果预测中的不确定性，在各阶段试验中实现了显著的 PR-AUC 和准确率提升。该方法在低置信度情况下选择放弃预测，以提高整体预测性能。

ABSTRACT

The importance of uncertainty quantification is increasingly recognized in the diverse field of machine learning. Accurately assessing model prediction uncertainty can help provide deeper understanding and confidence for researchers and practitioners. This is especially critical in medical diagnosis and drug discovery areas, where reliable predictions directly impact research quality and patient health. In this paper, we proposed incorporating uncertainty quantification into clinical trial outcome predictions. Our main goal is to enhance the model's ability to discern nuanced differences, thereby significantly improving its overall performance. We have adopted a selective classification approach to fulfill our objective, integrating it seamlessly with the Hierarchical Interaction Network (HINT), which is at the forefront of clinical trial prediction modeling. Selective classification, encompassing a spectrum of methods for uncertainty quantification, empowers the model to withhold decision-making in the face of samples marked by ambiguity or low confidence, thereby amplifying the accuracy of predictions for the instances it chooses to classify. A series of comprehensive experiments demonstrate that incorporating selective classification into clinical trial predictions markedly enhances the model's performance, as evidenced by significant upticks in pivotal metrics such as PR-AUC, F1, ROC-AUC, and overall accuracy. Specifically, the proposed method achieved 32.37\%, 21.43\%, and 13.27\% relative improvement on PR-AUC over the base model (HINT) in phase I, II, and III trial outcome prediction, respectively. When predicting phase III, our method reaches 0.9022 PR-AUC scores. These findings illustrate the robustness and prospective utility of this strategy within the area of clinical trial predictions, potentially setting a new benchmark in the field.

研究动机与目标

• 在药物开发决策中提高临床试验结果预测的不确定性量化以驱动决策。
• 通过引入选择性分类来扩展 HINT 模型，在低置信度预测时进行放弃。
• 利用外部知识和多模态嵌入提升预测准确性和可靠性。

提出的方法

• 将选择性分类与 HINT 结合，对不确定样本予以保留预测。
• 使用多模态输入嵌入（药物、疾病和试验方案）以及外部知识预训练（ADMET 和疾病风险）。
• 构建一个包含输入、外部知识、聚合和预测节点的分层相互作用图，然后应用动态注意力 GC N 进行预测。
• 在数据不完整时引入一个插补模块以恢复缺失的药物信息。
• 定义一个带校准的选择性分类框架，以设定阈值，在高选择性准确性下放弃不确定样本。

实验结果

研究问题

• RQ1 选择性分类相较于基础 HINT 模型在临床试验结果预测中的预测性能有何影响？
• RQ2 在量化不确定性并对不确定情况放弃预测时，PR-AUC、F1、ROC-AUC 以及总体准确率的提升有多大？
• RQ3 外部知识（ADMET 和疾病风险）在 HINT 框架中能否改进试验结果预测？
• RQ4 模型如何处理缺失的分子数据，其对性能的影响是什么？

主要发现

• 相对于基础 HINT 模型，在阶段 I、II、III 的试验结果预测中，PR-AUC 的相对提升分别为 32.37%、21.43% 和 13.27%。
• 在预测阶段 III 时，该方法实现了 PR-AUC 为 0.9022。
• 通过选择性分类，框架在关键指标（PR-AUC、F1、ROC-AUC 和总体准确率）方面显示显著提升。
• 该方法将不确定性量化与基于图的预测模型相结合，以在试验启动前提高可靠性。