[논문 리뷰] A Collaborative, Human-Centred Taxonomy of AI, Algorithmic, and Automation Harms

Gavin Abercrombie, Djalel Benbouzid외 14명|arXiv (Cornell University)|2024. 07. 01.

Ethics and Social Impacts of AI인용 수 5

한 줄 요약

The paper presents a collaborative, human-centered taxonomy of AI harms, organized into nine harm types and 69 specific harms, developed from incident data and crowdsourced annotation. It emphasizes openness, extensibility, and broad accessibility for civil society, educators, policymakers, and the public.

ABSTRACT

This paper introduces a collaborative, human-centred taxonomy of AI, algorithmic and automation harms. We argue that existing taxonomies, while valuable, can be narrow, unclear, typically cater to practitioners and government, and often overlook the needs of the wider public. Drawing on existing taxonomies and a large repository of documented incidents, we propose a taxonomy that is clear and understandable to a broad set of audiences, as well as being flexible, extensible, and interoperable. Through iterative refinement with topic experts and crowdsourced annotation testing, we propose a taxonomy that can serve as a powerful tool for civil society organisations, educators, policymakers, product teams and the general public. By fostering a greater understanding of the real-world harms of AI and related technologies, we aim to increase understanding, empower NGOs and individuals to identify and report violations, inform policy discussions, and encourage responsible technology development and deployment.

연구 동기 및 목표

• Develop a clear, human-centered harms taxonomy applicable to diverse audiences.
• Leverage real-world incidents from the AIAAIC Repository to ground taxonomy definitions.
• Create an extensible, interoperable, and machine-readable resource for education, policy, and risk management.
• Engage civil society, educators, policymakers, and the public in iterative refinement.
• Position the taxonomy as a living tool to inform reporting, policy discussions, and responsible AI deployment.

제안 방법

• Open, collaborative development with a diverse, international working group.
• Annotation testing using a custom open-source tool to classify incidents and compute agreement (Krippendorff’s alpha).
• Literature review and case analysis to map existing taxonomies and identify gaps.
• Definition of a two-level structure: harm type and specific harms, with formal definitions.
• Iterative refinement through expert feedback, public review, and crowdsourced validation.

Figure 1 . A screenshot of the annotation testing tool, tailored for collectively assessing and improving the taxonomy.

실험 결과

연구 질문

• RQ1 What harm types and specific harms should a broad AI harms taxonomy include to be understandable and actionable?
• RQ2 How can the taxonomy be designed to be flexible, extensible, and interoperable across contexts and technologies?
• RQ3 In what ways can the taxonomy support reporting, policy, journalism, and risk management activities?
• RQ4 How does this taxonomy compare to existing frameworks (e.g., OECD, CSET, SHAS) in coverage and usability?

주요 결과

• Nine harm types are defined, with sixty-nine sub-categories (specific harms).
• The taxonomy is grounded in the AIAAIC Repository, incorporating over 1,000 annotations across 39 incidents in early rounds.
• An open annotation tool produces Sankey diagrams and Krippendorff’s alpha to monitor inter-annotator agreement and guide refinement.
• The taxonomy is designed to be dynamic and open for ongoing updates through broad community input.
• It is intended to improve literacy, reporting, journalism, advocacy, policy development, and risk management across diverse audiences.
• Compared with other taxonomies, the work emphasizes a broad, outside-in perspective and sociotechnical harms, aiming for interoperability and real-world applicability.

Figure 2 . An overview of the AI, algorithmic and automation harms taxonomy