[论文解读] Towards Concrete and Connected AI Risk Assessment (C$^2$AIRA): A Systematic Mapping Study

Boming Xia, Qinghua Lu+5 more|arXiv (Cornell University)|Jan 27, 2023

Ethics and Social Impacts of AI被引用 8

一句话总结

系统性映射了16个行业驱动的AI风险评估框架，分析其特征、过程与差距，并提出一个具体且互联的C2AIRA框架。

ABSTRACT

The rapid development of artificial intelligence (AI) has led to increasing concerns about the capability of AI systems to make decisions and behave responsibly. Responsible AI (RAI) refers to the development and use of AI systems that benefit humans, society, and the environment while minimising the risk of negative consequences. To ensure responsible AI, the risks associated with AI systems' development and use must be identified, assessed and mitigated. Various AI risk assessment frameworks have been released recently by governments, organisations, and companies. However, it can be challenging for AI stakeholders to have a clear picture of the available frameworks and determine the most suitable ones for a specific context. Additionally, there is a need to identify areas that require further research or development of new frameworks, as well as updating and maintaining existing ones. To fill the gap, we present a mapping study of 16 existing AI risk assessment frameworks from the industry, governments, and non-government organizations (NGOs). We identify key characteristics of each framework and analyse them in terms of RAI principles, stakeholders, system lifecycle stages, geographical locations, targeted domains, and assessment methods. Our study provides a comprehensive analysis of the current state of the frameworks and highlights areas of convergence and divergence among them. We also identify the deficiencies in existing frameworks and outlines the essential characteristics of a concrete and connected framework AI risk assessment (C$^2$AIRA) framework. Our findings and insights can help relevant stakeholders choose suitable AI risk assessment frameworks and guide the design of future frameworks towards concreteness and connectedness.

研究动机与目标

• 调查并综合现有工业AI风险评估框架，以理解其特征、能力和局限性。
• 将框架映射到负责任AI原则以识别覆盖差距。
• 评估框架如何处理利益相关者、生命周期阶段、领域和评估过程。
• 识别缺陷并为未来的具体且互联的C2AIRA框架勾勒基本特征。

提出的方法

• 从行业、政府和非政府组织获取的16个工业AI风险评估框架的系统性映射。
• 按人口统计、RAI原则、利益相关者、生命周期阶段、地理位置和领域对框架进行分类。
• 分析程序性框架与描述性框架的输入、过程和输出。
• 与澳大利亚的AI伦理原则进行框架的交叉映射以标准化比较。
• 定性综合，突出趋同/分歧并识别未来框架设计的差距。

实验结果

研究问题

• RQ1 RQ1: 现有AI风险评估框架有哪些特征？（人口统计、RAI原则、利益相关者、生命周期阶段、地理、领域）
• RQ2 RQ2: 各框架在输入、过程、输出方面如何评估AI风险，风险因素与缓解的共性与差距是什么？

主要发现

• 分析了16个工业AI风险评估框架，其中62.5%在2022年发布或更新。
• 大多数框架来自美国、英国、欧盟、加拿大、澳大利亚及非政府组织，且许多是政府发布。
• 在16个框架中有11个规定了指导性的RAI原则；所有规定原则的框架都强调健康安全与福祉、人本价值观，并广泛覆盖公平性、可靠性/安全性，以及透明度/可解释性。
• 在10个框架中提及利益相关者参与，但多层次的多样化利益相关者角色常被低估。
• 大多数框架覆盖危害与暴露，通常考虑脆弱性；缓解风险的关注度较低。
• 7个框架未具体说明生命周期阶段，其他框架在规划、设计、部署和运营阶段映射风险；若干框架面向特定领域情境（公共部门、医疗保健、面向儿童的AI）。
• 程序性框架往往更具体，包含模板、清单或自动评分；描述性框架则更高层次。
• 输入主要是主观问答模板；只有部分包含互动工具或自动评分；各框架的缓解指南提供情况各异。
• 识别的差距包括对风险因素区分不足、缓解风险的处理不足，以及缺乏在特定情境下调整框架的明确信息。