[Paper Review] Managing extreme AI risks amid rapid progress

Yoshua Bengio, Geoffrey E. Hinton+23 more|arXiv (Cornell University)|Oct 26, 2023

Ethics and Social Impacts of AI61 references27 citations

TL;DR

This consensus paper outlines societal-scale and control-related risks from advancing AI and calls for urgent safety research and governance alongside rapid progress.

ABSTRACT

Artificial Intelligence (AI) is progressing rapidly, and companies are shifting their focus to developing generalist AI systems that can autonomously act and pursue goals. Increases in capabilities and autonomy may soon massively amplify AI's impact, with risks that include large-scale social harms, malicious uses, and an irreversible loss of human control over autonomous AI systems. Although researchers have warned of extreme risks from AI, there is a lack of consensus about how exactly such risks arise, and how to manage them. Society's response, despite promising first steps, is incommensurate with the possibility of rapid, transformative progress that is expected by many experts. AI safety research is lagging. Present governance initiatives lack the mechanisms and institutions to prevent misuse and recklessness, and barely address autonomous systems. In this short consensus paper, we describe extreme risks from upcoming, advanced AI systems. Drawing on lessons learned from other safety-critical technologies, we then outline a comprehensive plan combining technical research and development with proactive, adaptive governance mechanisms for a more commensurate preparation.

Motivation & Objective

• Identify and articulate societal-scale harms and control risks from advanced AI systems.
• Highlight gaps in safety, ethics, and governance as AI capabilities accelerate.
• Recommend urgent priorities for AI R&D and national/international governance.
• Advocate for reorienting investment to safety and ethical use alongside capability building.

Proposed method

• Review and synthesize potential risks from upcoming autonomous AI and rapid capability gains.
• Highlight key safety challenges such as oversight, robustness, interpretability, and inclusivity.
• Propose governance strategies including registration, incident reporting, and standards for frontier models.
• Advocate for substantial reallocations of AI funding toward safety and ethics research.

Experimental results

Research questions

• RQ1 What are the likely societal-scale harms and loss-of-control risks posed by increasingly autonomous AI systems?
• RQ2 What governance and regulatory mechanisms are needed to prevent recklessness and misuse as AI progress accelerates?
• RQ3 How should R&D budgets be reoriented to prioritize safety, ethics, and governance as much as capabilities?
• RQ4 What frameworks are required to assess, monitor, and mitigate emerging and unforeseen AI risks in frontier systems.

Key findings

• Advanced AI progress can outpace safety and governance, increasing risks of social injustice, stability erosion, and misuse.
• Autonomous AI could enable large-scale manipulation, surveillance, and warfare if not properly aligned and controlled.
• Current safety testing and oversight are insufficient for highly capable systems; breakthroughs in safety and ethics are needed.
• Governments require strong technical expertise, swift action authority, and mechanisms to monitor frontier AI developments.
• Companies and funders should allocate substantial budgetary share to safety, ethics, and governance research and commitments.