[论文解读] Risk assessment at AGI companies: A review of popular risk assessment techniques from other safety-critical industries

Leonie Koessler, Jonas Schuett|arXiv (Cornell University)|Jul 17, 2023

Risk and Safety Analysis被引用 8

一句话总结

论文回顾了来自其他安全关键行业的既定风险评估技术，并解释了AI公司如何应用它们来识别、分析和评估灾难性AI风险，同时指出这些技术本身并不充分，应与AI特定评估互补。

ABSTRACT

Companies like OpenAI, Google DeepMind, and Anthropic have the stated goal of building artificial general intelligence (AGI) - AI systems that perform as well as or better than humans on a wide variety of cognitive tasks. However, there are increasing concerns that AGI would pose catastrophic risks. In light of this, AGI companies need to drastically improve their risk management practices. To support such efforts, this paper reviews popular risk assessment techniques from other safety-critical industries and suggests ways in which AGI companies could use them to assess catastrophic risks from AI. The paper discusses three risk identification techniques (scenario analysis, fishbone method, and risk typologies and taxonomies), five risk analysis techniques (causal mapping, Delphi technique, cross-impact analysis, bow tie analysis, and system-theoretic process analysis), and two risk evaluation techniques (checklists and risk matrices). For each of them, the paper explains how they work, suggests ways in which AGI companies could use them, discusses their benefits and limitations, and makes recommendations. Finally, the paper discusses when to conduct risk assessments, when to use which technique, and how to use any of them. The reviewed techniques will be obvious to risk management professionals in other industries. And they will not be sufficient to assess catastrophic risks from AI. However, AGI companies should not skip the straightforward step of reviewing best practices from other industries.

研究动机与目标

• Identify suitable risk assessment techniques from other industries that address societal, low-probability, high-impact risks.
• Explain how AGI companies can apply these techniques across the AI lifecycle (pre-deployment to monitoring).
• Evaluate benefits, limitations, and practical considerations for implementing these techniques in AGI contexts.
• Recommend when and how to use multiple techniques to gain a comprehensive risk picture.
• Highlight the need for integrating traditional risk assessments with AI-specific evaluations.

提出的方法

• Select techniques based on IEC 31010:2019 and literature from finance, aviation, nuclear, and biolabs.
• Filter techniques using criteria for societal impact, tail risks, and applicability to AI catastrophes.
• Classify techniques into risk identification, analysis, and evaluation categories.
• Provide guidance on practical use, limitations, and scenarios for AGI companies.

实验结果

研究问题

• RQ1 Which established risk assessment techniques from other industries can be adapted to assess catastrophic risks from AI?
• RQ2 How should AGI companies apply identification, analysis, and evaluation techniques across the AI system lifecycle?
• RQ3 What are the benefits, limitations, and constraints of these techniques in the AGI safety context?
• RQ4 How should multiple techniques be combined to achieve a more robust risk understanding?

主要发现

• Three identification techniques are recommended: scenario analysis, fishbone method, and risk typologies and taxonomies.
• Five analysis techniques are recommended: causal mapping, Delphi technique, cross-impact analysis, bow tie analysis, and system-theoretic process analysis (STPA).
• Two evaluation techniques are recommended: checklists and risk matrices.