[論文レビュー] Towards Concrete and Connected AI Risk Assessment (C$^2$AIRA): A Systematic Mapping Study

Boming Xia, Qinghua Lu+5 more|arXiv (Cornell University)|Jan 27, 2023

Ethics and Social Impacts of AI被引用数 8

ひとこと要約

A systematic mapping of 16 industry-driven AI risk assessment frameworks analyzes their characteristics, processes, and gaps to propose a concrete and connected C2AIRA framework.

ABSTRACT

The rapid development of artificial intelligence (AI) has led to increasing concerns about the capability of AI systems to make decisions and behave responsibly. Responsible AI (RAI) refers to the development and use of AI systems that benefit humans, society, and the environment while minimising the risk of negative consequences. To ensure responsible AI, the risks associated with AI systems' development and use must be identified, assessed and mitigated. Various AI risk assessment frameworks have been released recently by governments, organisations, and companies. However, it can be challenging for AI stakeholders to have a clear picture of the available frameworks and determine the most suitable ones for a specific context. Additionally, there is a need to identify areas that require further research or development of new frameworks, as well as updating and maintaining existing ones. To fill the gap, we present a mapping study of 16 existing AI risk assessment frameworks from the industry, governments, and non-government organizations (NGOs). We identify key characteristics of each framework and analyse them in terms of RAI principles, stakeholders, system lifecycle stages, geographical locations, targeted domains, and assessment methods. Our study provides a comprehensive analysis of the current state of the frameworks and highlights areas of convergence and divergence among them. We also identify the deficiencies in existing frameworks and outlines the essential characteristics of a concrete and connected framework AI risk assessment (C$^2$AIRA) framework. Our findings and insights can help relevant stakeholders choose suitable AI risk assessment frameworks and guide the design of future frameworks towards concreteness and connectedness.

研究の動機と目的

• Survey and synthesize existing industrial AI risk assessment frameworks to understand their characteristics, capabilities, and limitations.
• Map frameworks to Responsible AI principles to identify coverage gaps.
• Assess how frameworks handle stakeholders, lifecycle stages, domains, and assessment processes.
• Identify deficiencies and outline essential characteristics for a future concrete and connected C2AIRA framework.

提案手法

• Systematic mapping of 16 industrial AI risk assessment frameworks sourced from industry, government, and NGOs.
• Classification of frameworks by demographics, RAI principles, stakeholders, lifecycle stages, geography, and domains.
• Analysis of inputs, processes, and outputs of procedural vs descriptive frameworks.
• Cross-mapping of frameworks to Australia’s AI ethics principles to standardize comparisons.
• Qualitative synthesis highlighting convergence/divergence and identifying gaps for future framework design.

実験結果

リサーチクエスチョン

• RQ1 RQ1: What are the characteristics of the existing AI risk assessment frameworks? (demographics, RAI principles, stakeholders, lifecycle stages, geography, domains)
• RQ2 RQ2: How are AI risks assessed (inputs, processes, outputs) across frameworks, and what are the commonalities and gaps in risk factors and mitigation?

主な発見

• 16 industrial AI risk assessment frameworks were analyzed, with 62.5% published or updated in 2022.
• Most frameworks originate from the US, UK, EU, Canada, Australia, and NGOs, and many are government-issued.
• 11 of the 16 frameworks specify guiding RAI principles; all that specify principles emphasize HSE wellbeing and human-centered values, with broad coverage of fairness, reliability/safety, and transparency/explainability.
• Stakeholder involvement is noted in 10 frameworks, but diverse, multi-level stakeholder roles are often underrepresented.
• Most frameworks cover hazard and exposure, with vulnerability commonly considered; mitigation risks are less frequently addressed.
• 7 frameworks do not specify lifecycle stages, while others map risks across planning, design, deployment, and operation stages; several frameworks target domain-specific contexts (public sector, healthcare, AI for children).
• Procedural frameworks tend to be more concrete and include templates, checklists, or automated scoring, whereas descriptive frameworks are more high-level.
• Inputs are primarily subjective Q&A templates; only a subset includes interactive tools or automated scoring; mitigation guidance is variably provided across frameworks.
• Gaps identified include limited differentiation of risk factors, insufficient treatment of mitigation risks, and lack of clear guidance on adapting frameworks to specific contexts.