[论文解读] To what extent should we trust AI models when they extrapolate?

Roozbeh Yousefzadeh, Xuenan Cao|arXiv (Cornell University)|Jan 27, 2022

Explainable Artificial Intelligence (XAI)被引用 4

一句话总结

本文提出一种几何审计方法，用于检测并报告人工智能模型的外推行为——即模型在训练数据范围之外做出预测——并证明此类外推行为频繁发生，具有显著的社会影响，且常被忽视。该方法利用凸包分析识别预测是否依赖于外推，主张通过自动化报告外推的范围和维度，提升高风险人工智能应用中的透明度与问责性。

ABSTRACT

Many applications affecting human lives rely on models that have come to be known under the umbrella of machine learning and artificial intelligence. These AI models are usually complicated mathematical functions that map from an input space to an output space. Stakeholders are interested to know the rationales behind models' decisions and functional behavior. We study this functional behavior in relation to the data used to create the models. On this topic, scholars have often assumed that models do not extrapolate, i.e., they learn from their training samples and process new input by interpolation. This assumption is questionable: we show that models extrapolate frequently; the extent of extrapolation varies and can be socially consequential. We demonstrate that extrapolation happens for a substantial portion of datasets more than one would consider reasonable. How can we trust models if we do not know whether they are extrapolating? Given a model trained to recommend clinical procedures for patients, can we trust the recommendation when the model considers a patient older or younger than all the samples in the training set? If the training set is mostly Whites, to what extent can we trust its recommendations about Black and Hispanic patients? Which dimension (race, gender, or age) does extrapolation happen? Even if a model is trained on people of all races, it still may extrapolate in significant ways related to race. The leading question is, to what extent can we trust AI models when they process inputs that fall outside their training set? This paper investigates several social applications of AI, showing how models extrapolate without notice. We also look at different sub-spaces of extrapolation for specific individuals subject to AI models and report how these extrapolations can be interpreted, not mathematically, but from a humanistic point of view.

研究动机与目标

• 挑战人工智能模型在预测时主要进行内插而非外推的普遍假设。
• 证明人工智能模型中的外推行为频繁发生、不可忽视，并可能在医疗、金融和就业等高风险领域产生重大社会后果。
• 开发一种系统性的几何审计方法，用于识别并量化在连续特征和分类特征中的外推行为。
• 倡导将外推报告集成到人工智能流水线中，以增强透明度，特别是在如GDPR等法规框架下。
• 通过将外推程度与模型复杂性及性能权衡关联，指导模型选择。

提出的方法

• 将训练数据的凸包（$\mathcal{H}^{tr}$）定义为所有训练样本的几何边界，其中边界内部的点为内插，边界外部的点为外推。
• 对于任意测试样本 $x$，判断其是否位于 $\mathcal{H}^{tr}$ 内部或外部，以分类预测为内插或外推。
• 通过几何分解识别对内推贡献最大的特征（如年龄、BMI、种族），实现维度特定的报告。
• 在因隐私顾虑而无法完全访问训练数据时，采用近似技术（如Blum等人的方法）估计凸包。
• 将审计流程集成到人工智能流水线中，自动记录并报告每次预测中外推的范围和方向。
• 建议将存在统计上过度外推的模型标记为需人工审查，尤其在医疗和刑事司法等敏感领域。

实验结果

研究问题

• RQ1 在医疗、薪资预测和算法交易等现实应用场景中，人工智能模型在多大程度上存在外推行为？
• RQ2 如何系统性地检测并量化具有混合特征类型（连续、分类、整数）的模型中的外推行为？
• RQ3 在用于高风险决策的人工智能模型中，未被检测到的外推行为会产生哪些社会后果？
• RQ4 外推程度与模型复杂性和性能之间有何关系？它能否用于指导模型选择？
• RQ5 如何在人工智能流水线中构建审计机制，以实现隐私保护且可解释的外推报告？

主要发现

• 人工智能模型在现实应用中频繁发生外推，外推现象在相当大比例的测试样本中存在，往往超出预期。
• 外推不仅限于连续变量，也发生在种族和性别等分类特征中，即使训练数据包含多样化群体也是如此。
• 外推程度在个体之间差异显著，可能产生社会影响，例如当模型为年龄超过训练集中任何患者的患者推荐临床治疗时。
• 即使在基于多样化人群训练的模型中，外推在与种族相关的维度上仍可能显著，表明仅靠代表性本身无法防止外推。
• 深度学习模型在性能优于简单模型的任务中，往往表现出更高的外推程度，表明模型复杂性与外推风险之间存在关联。
• 自动化外推审计（报告外推的范围和特征维度）可提升透明度，并支持监管合规，如GDPR要求。