[論文レビュー] Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

Victor Ojewale, Ryan Steed+3 more|arXiv (Cornell University)|Feb 27, 2024

Big Data and Business Intelligence被引用数 14

ひとこと要約

本論文はAI監査ツールを棚卸し、実務家へのインタビューを行い、現行のツールは評価に焦点を当てており、説明責任には不十分であると主張し、AI説明責任のためのより広範なインフラを求めている。

ABSTRACT

Audits are critical mechanisms for identifying the risks and limitations of deployed artificial intelligence (AI) systems. However, the effective execution of AI audits remains incredibly difficult, and practitioners often need to make use of various tools to support their efforts. Drawing on interviews with 35 AI audit practitioners and a landscape analysis of 435 tools, we compare the current ecosystem of AI audit tooling to practitioner needs. While many tools are designed to help set standards and evaluate AI systems, they often fall short in supporting accountability. We outline challenges practitioners faced in their efforts to use AI audit tools and highlight areas for future tool development beyond evaluation -- from harms discovery to advocacy. We conclude that the available resources do not currently support the full scope of AI audit practitioners' needs and recommend that the field move beyond tools for just evaluation and towards more comprehensive infrastructure for AI accountability.

研究の動機と目的

• Map the current ecosystem of AI audit tools across the audit process stages.
• Understand which AI audit tools are actually used in practice and how they are applied.
• Identify gaps between available tooling and practitioners' accountability needs.
• Provide recommendations for researchers, policymakers, and practitioners to develop infrastructure beyond evaluation.

提案手法

• Collated a dataset of 390 AI audit tools.
• Developed a taxonomy organizing tools into seven stages of the audit process.
• Conducted 27 semi-structured interviews with 35 audit practitioners across 24 organizations.
• Applied qualitative coding (descriptive and values coding) to interview transcripts.
• Performed landscape analysis augmented by sources like Crunchbase, GitHub, and Google Scholar API.
• Provided an interactive dataset at tools.auditing-ai.com and accompanying analysis at github.com/ryansteed/oat-analysis.

Figure 1 . Stages of the tool-supported audit process surfaced in our survey of AI audit tooling. We taxonomize tools by the stage of the AI audit process in which they are meant to be used.

実験結果

リサーチクエスチョン

• RQ1 RQ1: What AI audit tooling is available to support AI audit work?
• RQ2 RQ2: Which AI audit tools are actually used and how?
• RQ3 RQ3: What do AI audit practitioners actually need?

主な発見

• There are many tools to support evaluation and standards management, but fewer tools for stages essential to accountability such as harms discovery, data collection, transparency infrastructure, advocacy, and audit communication.
• Practitioners frequently adapt or build their own tools to fit workflows, revealing gaps between available tooling and real-world needs.
• Access to high-quality, uncompromised data, consistent holistic standards, audit integrity, and cross-disciplinary collaboration are persistent challenges.
• Most tools identified are publicly available or open-source, yet practical impact of tooling is limited without infrastructure supporting accountability.
• The study advocates moving beyond evaluation toward comprehensive AI accountability infrastructure that supports harms discovery, stakeholder inclusion, and advocacy.

Figure 2 . Number of tools in each category of our taxonomy, grouped by type of organization.