[论文解读] Failure Modes in Machine Learning Systems

Ram Shankar Siva Kumar, David O'Brien+3 more|arXiv (Cornell University)|Nov 25, 2019

Information and Cyber Security被引用 40

一句话总结

本文提出一种分类法，将 ML 系统故障分为有意的（对抗性）与无意的（本质上不安全的结果），并讨论其对从业者和政策制定者的适用性。

ABSTRACT

In the last two years, more than 200 papers have been written on how machine learning (ML) systems can fail because of adversarial attacks on the algorithms and data; this number balloons if we were to incorporate papers covering non-adversarial failure modes. The spate of papers has made it difficult for ML practitioners, let alone engineers, lawyers, and policymakers, to keep up with the attacks against and defenses of ML systems. However, as these systems become more pervasive, the need to understand how they fail, whether by the hand of an adversary or due to the inherent design of a system, will only become more pressing. In order to equip software developers, security incident responders, lawyers, and policy makers with a common vernacular to talk about this problem, we developed a framework to classify failures into "Intentional failures" where the failure is caused by an active adversary attempting to subvert the system to attain her goals; and "Unintentional failures" where the failure is because an ML system produces an inherently unsafe outcome. After developing the initial version of the taxonomy last year, we worked with security and ML teams across Microsoft, 23 external partners, standards organization, and governments to understand how stakeholders would use our framework. Throughout the paper, we attempt to highlight how machine learning failure modes are meaningfully different from traditional software failures from a technology and policy perspective.

研究动机与目标

• 提供一个共同的说法，方便开发者、安全响应人员、律师和政策制定者讨论 ML 故障。
• 引入一个将故障分为有意和无意两类的分类法。
• 展示 ML 故障模式在技术和政策视角下与传统软件故障的差异。

提出的方法

• 开发一个框架，将 ML 故障分为 Intentional 和 Unintentional 两类。
• 与微软、外部合作伙伴、标准组织和政府合作，以验证对利益相关方的实用性。
• 在技术与政策情境下，将 ML 故障模式与传统软件故障进行比较。

实验结果

研究问题

• RQ1 如何将 ML 系统故障分类为可供不同利益相关方行动的类别？
• RQ2 有意与无意的 ML 故障的定义特征是什么？
• RQ3 在技术和政策层面，ML 故障模式与传统软件故障模式有何不同？

主要发现

• 将有意故障（由主动对手引起）与无意故障（本质上不安全的结果）区分的分类法。
• 该框架设计供软件开发者、事件响应者、律师和政策制定者使用。
• ML 故障模式在技术和政策层面上，与传统软件故障在性质上存在显著差异。