[论文解读] The Necessity of AI Audit Standards Boards

David Manheim, Sammy Martin+3 more|arXiv (Cornell University)|Apr 11, 2024

Ethics and Social Impacts of AI被引用 7

一句话总结

本文认为静态的 AI 审计标准不足以应对现实，且可能带来危害，提出设立专门的 AI 审计标准委员会，以在 AI 生命周期内开发和完善审计方法，融入治理、文化和多方参与。

ABSTRACT

Auditing of AI systems is a promising way to understand and manage ethical problems and societal risks associated with contemporary AI systems, as well as some anticipated future risks. Efforts to develop standards for auditing Artificial Intelligence (AI) systems have therefore understandably gained momentum. However, we argue that creating auditing standards is not just insufficient, but actively harmful by proliferating unheeded and inconsistent standards, especially in light of the rapid evolution and ethical and safety challenges of AI. Instead, the paper proposes the establishment of an AI Audit Standards Board, responsible for developing and updating auditing methods and standards in line with the evolving nature of AI technologies. Such a body would ensure that auditing practices remain relevant, robust, and responsive to the rapid advancements in AI. The paper argues that such a governance structure would also be helpful for maintaining public trust in AI and for promoting a culture of safety and ethical responsibility within the AI industry. Throughout the paper, we draw parallels with other industries, including safety-critical industries like aviation and nuclear energy, as well as more prosaic ones such as financial accounting and pharmaceuticals. AI auditing should emulate those fields, and extend beyond technical assessments to include ethical considerations and stakeholder engagement, but we explain that this is not enough; emulating other fields' governance mechanisms for these processes, and for audit standards creation, is a necessity. We also emphasize the importance of auditing the entire development process of AI systems, not just the final products...

研究动机与目标

• 论证静态且不断扩增的 AI 审计标准不足以应对并可能带来危害。
• 提出设立 AI 审计标准委员会，以制定和更新审计方法与标准。
• 主张对整个 AI 开发生命周期进行审计，而不仅仅是最终产品，以应对不断演变的风险与伦理问题。
• 强调文化、治理和利益相关者参与在有效 AI 审计中的作用。

提出的方法

• 审查来自安全关键行业（航空、核能、药品）现有的 AI 审计实践、标准和治理模式。
• 批判性分析当前静态标准和以模型为中心的评估的局限性。
• 提出三管齐下的方法：审计过程、培养安全文化、授权独立的标准委员会。
• 倡导对开发与部署全过程进行审计、持续的利益相关者参与，以及生命周期风险分析。
• 主张快速、适应性强的标准制定，以跟上基础模型能力与滥用风险的步伐。

实验结果

研究问题

• RQ1 当前静态 AI 审计标准和以模型为中心的评估有哪些局限性？
• RQ2 独立的 AI 审计标准委员会如何提升 AI 审计的相关性、鲁棒性和公众信任？
• RQ3 应从安全关键行业借鉴哪些做法应用于 AI 审计，以解决过程、文化和治理问题？
• RQ4 为什么在训练阶段及整个开发生命周期内进行审计对于新兴的 AI 能力是必要的？

主要发现

• 静态的、部署前的标准未能跟上 AI 的快速演变和新兴能力。
• 审计必须覆盖整个开发生命周期，包括数据来源、训练和部署后的监控。
• 以安全文化为导向的多方治理模型提升审计效果和公众信任。
• 一个独立的 AI 审计标准委员会用于制定、更新并公开审计规范，超越内部做法。
• 持续、动态的风险评估对于防止安全洗白并应对未来的滥用风险至关重要。