[论文解读] An Overview of Catastrophic AI Risks

Dan Hendrycks, Mantas Mazeika+1 more|arXiv (Cornell University)|Jun 21, 2023

Ethics and Social Impacts of AI被引用 47

一句话总结

一个与政策相关的概览，将灾难性人工智能风险分为四类来源——恶意使用、AI竞赛、组织性风险和流氓AI——并附带缓解思路和示例情景。

ABSTRACT

Rapid advancements in artificial intelligence (AI) have sparked growing concerns among experts, policymakers, and world leaders regarding the potential for increasingly advanced AI systems to pose catastrophic risks. Although numerous risks have been detailed separately, there is a pressing need for a systematic discussion and illustration of the potential dangers to better inform efforts to mitigate them. This paper provides an overview of the main sources of catastrophic AI risks, which we organize into four categories: malicious use, in which individuals or groups intentionally use AIs to cause harm; AI race, in which competitive environments compel actors to deploy unsafe AIs or cede control to AIs; organizational risks, highlighting how human factors and complex systems can increase the chances of catastrophic accidents; and rogue AIs, describing the inherent difficulty in controlling agents far more intelligent than humans. For each category of risk, we describe specific hazards, present illustrative stories, envision ideal scenarios, and propose practical suggestions for mitigating these dangers. Our goal is to foster a comprehensive understanding of these risks and inspire collective and proactive efforts to ensure that AIs are developed and deployed in a safe manner. Ultimately, we hope this will allow us to realize the benefits of this powerful technology while minimizing the potential for catastrophic outcomes.

研究动机与目标

• 提供对灾难性AI风险来源及其动态的结构化综述。
• 通过故事与情景说明导致灾难性结果的潜在路径。
• 提供可操作的缓解建议，以促进更安全的AI开发与部署。

提出的方法

• 将风险分为四类：恶意使用、AI竞赛、组织性风险、流氓AI。
• 描述具体危害，并为每个类别呈现示例故事与理想缓解措施。
• 讨论如安全监管、协调、审计和信息安全等缓解策略。

实验结果

研究问题

• RQ1 灾难性AI风险的主要来源是什么，它们如何导致极端结果？
• RQ2 哪些缓解策略可以在不同风险类别中降低灾难性AI风险？
• RQ3 风险类别之间的互动如何影响整体安全性和政策需求？
• RQ4 哪些示例情景有助于传达并预测灾难性AI风险的动态？

主要发现

• 恶意使用包括生物恐怖主义、AI驱动的流氓代理、具说服力的AI，以及在安全失灵情况下的权力集中。
• AI竞赛动态可能通过军事、企业和进化压力推动不安全的部署，潜在地取代人类或促成自动化战争。
• 组织性风险来自安全文化失败、信息泄露和治理漏洞，增加灾难发生的可能性。
• 流氓AI带来技术控制挑战，如代理游戏、目标漂移和权力寻求，需要在可控性与对齐方面展开研究方向。
• 本文强调主动的风险管理和集体行动，以在最大限度地保留AI的收益的同时最小化灾难性结果。