[Paper Review] Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing

Inioluwa Deborah Raji, Andrew Smart+7 more|arXiv (Cornell University)|Jan 3, 2020

Ethics and Social Impacts of AI56 references154 citations

TL;DR

This paper introduces SMACTR, an internal audit framework to assess AI systems against declared principles throughout development, aiming to close the accountability gap before deployment.

ABSTRACT

Rising concern for the societal implications of artificial intelligence systems has inspired a wave of academic and journalistic literature in which deployed systems are audited for harm by investigators from outside the organizations deploying the algorithms. However, it remains challenging for practitioners to identify the harmful repercussions of their own systems prior to deployment, and, once deployed, emergent issues can become difficult or impossible to trace back to their source. In this paper, we introduce a framework for algorithmic auditing that supports artificial intelligence system development end-to-end, to be applied throughout the internal organization development lifecycle. Each stage of the audit yields a set of documents that together form an overall audit report, drawing on an organization's values or principles to assess the fit of decisions made throughout the process. The proposed auditing framework is intended to contribute to closing the accountability gap in the development and deployment of large-scale artificial intelligence systems by embedding a robust process to ensure audit integrity.

Motivation & Objective

• Propose internal algorithmic audits as a pre-deployment mechanism to evaluate alignment with AI principles.
• Embed audit artifacts within the product development lifecycle to strengthen governance and accountability.
• Provide a structured framework to surface ethical risks and guide design mitigations.
• Demonstrate how internal audits can supplement external accountability and improve transparency.

Proposed method

• Introduce SMACTR as a five-stage process: Scoping, Mapping, Artifact Collection, Testing, and Reflection.
• Define governance practices drawing on responsible innovation and system-theoretic concepts to anticipate sociotechnical risks.
• Specify documentation artifacts produced at each stage and the roles of auditors, engineers, and management.
• Argue for audit integrity via procedural justice and a transparent, auditable methodology.
• Use a hypothetical Company X and two client scenarios to illustrate practical application and outcomes.

Experimental results

Research questions

• RQ1 What is the end-to-end process for an internal algorithmic audit that aligns AI development with organizational principles?
• RQ2 How can internal audits influence design decisions and governance to mitigate ethical and societal risks before deployment?
• RQ3 What artifacts and governance structures are required to support credible internal AI auditing?
• RQ4 How can internal audits complement external accountability mechanisms in AI governance?

Key findings

• An internal audit framework (SMACTR) can structure pre-deployment evaluation across scoping, mapping, artifact collection, testing, and reflection.
• Audits should integrate principled governance, anticipatory thinking, and stakeholder inclusion to surface ethical risks.
• Documented artifacts and transparent procedures enhance audit integrity and legitimacy (procedural justice).
• Internal audits can reveal gaps in design processes and inform organizational changes to align with AI principles.
• Examples show how audits apply to high-stakes (child abuse screening) and lower-stakes (smile detection) contexts to illustrate risk considerations.