[Paper Review] Indexing AI Risks with Incidents, Issues, and Variants

Sean McGregor, Kevin Paeth+1 more|arXiv (Cornell University)|Nov 18, 2022

Adversarial Robustness in Machine Learning7 citations

TL;DR

This paper proposes a two-tiered indexing system for AI risks—'incidents' (realized harms or near harms) and 'issues' (potential harms not yet realized)—along with 'variants' to manage repetitive incidents. The approach enhances scalability, supports policy compliance, and enables interoperable, citable incident tracking across global databases.

ABSTRACT

Two years after publicly launching the AI Incident Database (AIID) as a collection of harms or near harms produced by AI in the world, a backlog of "issues" that do not meet its incident ingestion criteria have accumulated in its review queue. Despite not passing the database's current criteria for incidents, these issues advance human understanding of where AI presents the potential for harm. Similar to databases in aviation and computer security, the AIID proposes to adopt a two-tiered system for indexing AI incidents (i.e., a harm or near harm event) and issues (i.e., a risk of a harm event). Further, as some machine learning-based systems will sometimes produce a large number of incidents, the notion of an incident "variant" is introduced. These proposed changes mark the transition of the AIID to a new version in response to lessons learned from editing 2,000+ incident reports and additional reports that fall under the new category of "issue."

Motivation & Objective

• Address the growing backlog of AI harm reports that do not meet strict incident criteria but still signal significant risks.
• Improve the scalability and sustainability of AI incident databases amid rising reporting volumes and incomplete data.
• Support policy development by enabling flexible reporting tiers that distinguish between realized incidents and potential risks.
• Reduce editorial burden by introducing 'variants' to group similar incidents with shared causes and impacts.
• Enable interoperability across national and organizational AI incident databases through standardized, namespaced identifiers.

Proposed method

• Introduce a two-tier classification: 'AI Incident' (an alleged harm or near harm involving an AI system) and 'AI Issue' (a potential harm not yet realized).
• Define 'AI Incident Variant' as events sharing the same causative factors, harms, and systems as a known incident, reducing redundant indexing.
• Propose a new incident identifier format: NAMESPACE:INCIDENT#.VARIANT#, ensuring backward compatibility and future interoperability.
• Adopt inspiration from aviation safety and cybersecurity models (e.g., CVE) to differentiate between events and risks.
• Use real-world examples—such as facial recognition at Palestinian checkpoints—to illustrate the need for issue categorization and variant handling.
• Design the system to support voluntary and mandatory reporting frameworks, with incentives for early disclosure via the issue tier.

Experimental results

Research questions

• RQ1 How can AI incident databases effectively capture and classify potential harms that have not yet materialized but pose significant risk?
• RQ2 What structural changes are needed to scale incident reporting without overwhelming editorial capacity?
• RQ3 How can incident variants be meaningfully grouped and identified to reduce redundancy while preserving traceability?
• RQ4 What role can a two-tier system (incidents vs. issues) play in supporting regulatory compliance and policy development?
• RQ5 How can incident identifiers be extended to support interoperability across federated, national, or organizational databases?

Key findings

• The AIID has indexed over 300 incidents and 2,000 incident reports, revealing a growing backlog of reports that do not meet current incident criteria but still signal real risks.
• The introduction of 'issues' allows the database to retain and track salient potential harms—such as surveillance by facial recognition systems—without requiring proof of actual misidentification.
• Incident variants, such as repeated misidentifications by the same facial recognition system, can be grouped under a single incident with a variant identifier, reducing editorial load.
• The proposed NAMESPACE:INCIDENT#.VARIANT# identifier format enables backward compatibility and supports future federation of incident databases across jurisdictions.
• The two-tier system provides a practical pathway for regulatory compliance, offering incentives such as penalty waivers for early reporting of issues.
• The framework is designed to be interoperable and scalable, supporting global adoption and alignment with international standards like those being developed by the OECD.