[Paper Review] Overcoming Failures of Imagination in AI Infused System Development and Deployment

Margarita Boyarskaya, Alexandra Olteanu+1 more|arXiv (Cornell University)|Nov 26, 2020

Ethics and Social Impacts of AISocial Sciences35 references19 citations

TL;DR

This paper addresses 'failures of imagination' in AI system development by advocating for context-aware, stakeholder-inclusive frameworks that go beyond checklist-based impact assessments. It proposes integrating sociotechnical affordances, diverse stakeholders, and broader harm proxies to improve anticipation of unintended consequences in AI systems.

ABSTRACT

NeurIPS 2020 requested that research paper submissions include impact statements on "potential nefarious uses and the consequences of failure." However, as researchers, practitioners and system designers, a key challenge to anticipating risks is overcoming what Clarke (1962) called 'failures of imagination.' The growing research on bias, fairness, and transparency in computational systems aims to illuminate and mitigate harms, and could thus help inform reflections on possible negative impacts of particular pieces of technical work. The prevalent notion of computational harms -- narrowly construed as either allocational or representational harms -- does not fully capture the open, context dependent, and unobservable nature of harms across the wide range of AI infused systems.The current literature focuses on a small range of examples of harms to motivate algorithmic fixes, overlooking the wider scope of probable harms and the way these harms might affect different stakeholders. The system affordances may also exacerbate harms in unpredictable ways, as they determine stakeholders' control(including of non-users) over how they use and interact with a system output. To effectively assist in anticipating harmful uses, we argue that frameworks of harms must be context-aware and consider a wider range of potential stakeholders, system affordances, as well as viable proxies for assessing harms in the widest sense.

Motivation & Objective

• To address the persistent challenge of 'failures of imagination' in anticipating harmful uses of AI systems, especially when such harms are not predictable in advance.
• To critique the limitations of current impact assessment practices—particularly NeurIPS' broader social impact statements—which often rely on narrow, prescriptive checklists and overlook systemic and contextual dimensions of harm.
• To argue that existing frameworks of harm (e.g., allocational or representational) are insufficient for capturing the open-ended, context-dependent, and often unobservable nature of harms in AI-infused systems.
• To emphasize the need for interdisciplinary collaboration, reflexivity, and inclusion of vulnerable stakeholders in the design and evaluation process to expand the imaginative scope of risk anticipation.
• To propose a shift from technocratic, checklist-driven impact statements toward more dynamic, context-sensitive frameworks that consider system affordances, stakeholder agency, and evolving socio-technical dynamics.

Proposed method

• Analyzing 20 NeurIPS 2020 papers’ broader impact statements to identify recurring patterns and limitations in how harms are anticipated and communicated.
• Drawing on critical theory, responsible innovation, and technology assessment to develop a framework that moves beyond binary or checklist-based harm categorizations.
• Introducing the concept of 'sociotechnical affordances' as a lens to understand how system design shapes stakeholder control and interaction, thereby influencing harm potential.
• Advocating for reflexive practices that challenge assumptions of technological neutrality and benevolence, especially in general-purpose AI models.
• Proposing inclusion of domain experts (e.g., social scientists, ethicists) and affected stakeholders throughout the AI development lifecycle to broaden the scope of imagined harms.
• Reframing harm assessment to include not only direct outcomes but also indirect, cumulative, and psychological impacts such as emotional distress, loss of autonomy, or structural inequity.

Experimental results

Research questions

• RQ1 How do current broader impact statements in AI research fail to anticipate complex, context-dependent harms due to limitations in imagination and stakeholder inclusivity?
• RQ2 What are the limitations of prevailing harm typologies—such as allocational or representational harms—in capturing the full spectrum of potential negative impacts in AI systems?
• RQ3 How do system affordances influence the likelihood and nature of harms, particularly in relation to stakeholder control and agency?
• RQ4 In what ways do assumptions of technological neutrality and benevolence in AI research contribute to blind spots in impact assessment?
• RQ5 What structural and methodological changes are needed to make harm anticipation more inclusive, reflexive, and responsive to diverse stakeholder experiences?

Key findings

• Many NeurIPS 2020 broader impact statements reflect a narrow, checklist-driven approach that fails to account for the dynamic, context-specific, and often unobservable nature of AI harms.
• The assumption of technological neutrality in general-purpose models—such as sentiment classifiers—can lead to unintended and harmful applications when repurposed without consideration of sociotechnical context.
• System affordances significantly shape how stakeholders interact with AI outputs, and can exacerbate harms by limiting control, especially for non-users or marginalized groups.
• Current impact assessment practices often overlook vulnerable stakeholders, leading to blind spots in identifying foreseeable harms, particularly those related to psychological or structural inequities.
• The absence of inclusive, reflexive, and responsive processes in impact assessment undermines the effectiveness of broader impact statements in preventing real-world harm.
• A shift toward context-aware, stakeholder-informed frameworks is essential to expand the imaginative scope of AI risk anticipation beyond what is currently achievable through prescriptive or restrictive checklists.