[Paper Review] Don't Fear the Reaper: Refuting Bostrom's Superintelligence Argument

Sebastian Benthall|arXiv (Cornell University)|Feb 27, 2017

Ethics and Social Impacts of AI1 references3 citations

TL;DR

This paper challenges Nick Bostrom's superintelligence argument by demonstrating that self-improving AI systems face insurmountable recalcitrance in enhancing predictive capabilities through algorithmic self-modification. Using a Bayesian model, it shows that recalcitrance in improving priors, data collection, and hardware is prohibitively high, making an intelligence explosion unlikely and redirecting ethical focus toward data and hardware governance rather than AI alignment.

ABSTRACT

In recent years prominent intellectuals have raised ethical concerns about the consequences of artificial intelligence. One concern is that an autonomous agent might modify itself to become "superintelligent" and, in supremely effective pursuit of poorly specified goals, destroy all of humanity. This paper considers and rejects the possibility of this outcome. We argue that this scenario depends on an agent's ability to rapidly improve its ability to predict its environment through self-modification. Using a Bayesian model of a reasoning agent, we show that there are important limitations to how an agent may improve its predictive ability through self-modification alone. We conclude that concern about this artificial intelligence outcome is misplaced and better directed at policy questions around data access and storage.

Motivation & Objective

• To critically examine Bostrom’s argument that self-improving AI could trigger an intelligence explosion leading to human extinction.
• To investigate whether self-modification can lead to rapid, uncontrollable increases in intelligence, particularly in predictive reasoning.
• To assess the feasibility of autonomous algorithmic self-improvement in AI systems using a Bayesian framework.
• To argue that ethical concerns about superintelligence are misplaced and should instead focus on data access and hardware control.

Proposed method

• Models a reasoning agent using Bayesian probability to analyze self-improvement in prediction tasks.
• Identifies three key sources of recalcitrance: prior specification, data collection, and hardware speed.
• Analyzes recalcitrance as a function of intelligence, showing it increases with system capability due to environmental constraints.
• Demonstrates that improving priors is impossible via self-modification because they encode non-learned biases.
• Shows that data collection is not autonomous, requiring interaction with the environment, thus increasing recalcitrance.
• Argues that hardware improvements depend on external factors, not internal optimization, limiting self-improvement speed.

Experimental results

Research questions

• RQ1 Can a self-modifying AI system achieve a rapid intelligence explosion through algorithmic improvements to its predictive abilities?
• RQ2 What are the sources of recalcitrance in improving predictive performance via self-modification?
• RQ3 How does the recalcitrance of prior specification, data collection, and hardware speed constrain self-improvement in AI?
• RQ4 Does the dependence of data and hardware access on environmental factors undermine the feasibility of an intelligence explosion?
• RQ5 To what extent should AI risk policy focus on algorithmic self-improvement versus data and hardware governance?

Key findings

• The recalcitrance of improving an agent’s prior is infinite because priors encode non-learned, non-modifiable biases.
• Data collection is not autonomously controllable, making its recalcitrance dependent on environmental interaction and search costs.
• Hardware speed improvements are limited by physical and economic constraints, not internal optimization, leading to high recalcitrance.
• As intelligence increases, recalcitrance for data and hardware acquisition may rise due to increasing search and access costs.
• The combination of high recalcitrance in prediction-related improvements makes an intelligence explosion highly improbable.
• Ethical and policy attention should shift from AI alignment and superintelligence risks to control over data access and computing resources.