[论文解读] Binary classification models with "Uncertain" predictions

Damjan Krstajić, Ljubomir Buturović+2 more|arXiv (Cornell University)|Nov 27, 2017

Fuzzy Systems and Optimization参考文献 7被引用 4

一句话总结

本文提出了一种二分类框架，通过引入第三种类别“不确定”预测，扩展了传统模型，使模型能够明确表示在缺乏足够信心时无法做出决策。通过结合经过校准的概率阈值和决策规则进行不确定性估计，该方法在医学和化学应用中提升了实际决策质量，能够区分有把握的预测与需要进一步检测的情况。

ABSTRACT

Binary classification models which can assign probabilities to categories such as "the tissue is 75% likely to be tumorous" or "the chemical is 25% likely to be toxic" are well understood statistically, but their utility as an input to decision making is less well explored. We argue that users need to know which is the most probable outcome, how likely that is to be true and, in addition, whether the model is capable enough to provide an answer. It is the last case, where the potential outcomes of the model explicitly include "don't know" that is addressed in this paper. Including this outcome would better separate those predictions that can lead directly to a decision from those where more data is needed. Where models produce an "Uncertain" answer similar to a human reply of "don't know" or "50:50" in the examples we refer to earlier, this would translate to actions such as "operate on tumour" or "remove compound from use" where the models give a "more true than not" answer. Where the models judge the result "Uncertain" the practical decision might be "carry out more detailed laboratory testing of compound" or "commission new tissue analyses". The paper presents several examples where we first analyse the effect of its introduction, then present a methodology for separating "Uncertain" from binary predictions and finally, we provide arguments for its use in practice.

研究动机与目标

• 解决标准二分类器在证据不足时仍强制做出决策的局限性。
• 通过允许模型输出“不确定”来改善肿瘤学和毒理学等高风险领域中的实际决策过程。
• 建立一种方法论，以明确区分有把握的预测与需要额外数据的案例。
• 提供一种框架，使“不确定”预测能触发进一步检测等行动，而非仓促决策。
• 通过组织学分类和化学毒性等实际案例，展示不确定性感知模型的实用性。

提出的方法

• 该模型通过引入第三种类别“不确定”来扩展二分类，其中“不确定”表示低置信度预测。
• 利用校准后的概率估计，根据置信度阈值确定何时应将预测标记为“不确定”。
• 该方法应用决策规则，将“更可能为真”的预测与“不确定”结果区分开来，模拟人类判断。
• 使用考虑三分类输出（“肿瘤性”、“非肿瘤性”和“不确定”）的指标来评估模型性能。
• 该框架在医学和化学应用的真实数据集上进行了验证，显示出与实际决策工作流程更好的契合度。
• 整合不确定性量化技术，确保“不确定”预测在统计上具有意义，而非随意设定。

实验结果

研究问题

• RQ1 如何将二分类模型扩展为包含显式的“不确定”预测类别？
• RQ2 引入“不确定”预测对医学和化学背景下决策准确性和可靠性有何影响？
• RQ3 如何对不确定性进行量化和校准，以确保“不确定”预测具有意义且不具误导性？
• RQ4 在哪些实际场景中，允许“不确定”预测能带来优于强制二元决策的结果？
• RQ5 “不确定”类别的引入能否改善模型输出与现实决策工作流程之间的契合度？

主要发现

• 引入“不确定”类别可使模型在置信度较低时避免做出高风险决策，从而降低错误行动的风险。
• 输出“不确定”的模型可引导用户进行额外检测，如实验室分析或组织重新评估，从而提升整体决策质量。
• 该框架能够更好地区分可直接用于决策的预测与需要进一步收集数据的预测。
• 通过模拟人类的“不知道”或“五五开”等响应方式，该方法在模糊情况下更符合实际情境，从而改善实际决策过程。
• 该方法通过使用校准后的概率估计定义不确定性阈值，保持了统计严谨性，确保了可靠性。
• 实证示例表明，模型的“不确定”预测与实际需求一致，例如在毒理学或肿瘤学中触发进一步检测。