[Paper Review] On the Societal Impact of Open Foundation Models

Sayash Kapoor, Rishi Bommasani+23 more|arXiv (Cornell University)|Feb 27, 2024

3D Modeling in Geospatial Applications11 citations

TL;DR

This paper defines five distinctive properties of open foundation models, develops a marginal-risk framework for misuse, and argues for empirically grounding the claimed societal benefits and risks to inform policy and practice.

ABSTRACT

Foundation models are powerful technologies: how they are released publicly directly shapes their societal impact. In this position paper, we focus on open foundation models, defined here as those with broadly available model weights (e.g. Llama 2, Stable Diffusion XL). We identify five distinctive properties (e.g. greater customizability, poor monitoring) of open foundation models that lead to both their benefits and risks. Open foundation models present significant benefits, with some caveats, that span innovation, competition, the distribution of decision-making power, and transparency. To understand their risks of misuse, we design a risk assessment framework for analyzing their marginal risk. Across several misuse vectors (e.g. cyberattacks, bioweapons), we find that current research is insufficient to effectively characterize the marginal risk of open foundation models relative to pre-existing technologies. The framework helps explain why the marginal risk is low in some cases, clarifies disagreements about misuse risks by revealing that past work has focused on different subsets of the framework with different assumptions, and articulates a way forward for more constructive debate. Overall, our work helps support a more grounded assessment of the societal impact of open foundation models by outlining what research is needed to empirically validate their theoretical benefits and risks.

Motivation & Objective

• Identify how open foundation models differ from closed ones and why these differences matter for society.
• Develop a framework to assess the marginal misuse risk of open foundation models across key misuse vectors.
• Articulate the societal benefits (innovation, competition, transparency, etc.) and the conditions under which they materialize.
• Provide policy and research recommendations to better validate benefits and mitigate risks.

Proposed method

• Define open foundation models as those with widely available model weights and contrast them with closed models.
• Enumerate five distinctive properties of open models: broader access, greater customizability, local inference, irreversibility of access, and weaker monitoring.
• Propose a six-step risk assessment framework for marginal misuse risk (threat identification, existing risk, existing defenses, etc.).
• Survey seven misuse vectors (e.g., disinformation, biosecurity, cybersecurity, NCII, scams) to evaluate marginal risk.
• Discuss how framework clarifies disagreements in prior work and guides empirical validation.

Experimental results

Research questions

• RQ1 What distinctive properties differentiate open foundation models from closed ones, and how do these properties translate into societal benefits and risks?
• RQ2 How should we assess the marginal risk of open foundation models across various misuse vectors, and what evidence is needed to empirically validate these risks?
• RQ3 In what ways can policymakers, researchers, and developers leverage the framework to improve safety, transparency, and innovation while mitigating harms?

Key findings

• Open foundation models can broaden access, enable customization, support local inference, and potentially improve transparency, which together influence innovation and competition.
• A marginal-risk framework can explain why some misuse risks appear low and why prior studies disagree by focusing on different framework components.
• Empirical evidence for marginal risk is currently weak for several misuse vectors, signaling the need for more grounded research.
• The paper offers concrete guidance for developers, researchers, regulators, and policymakers to better assess societal impact and design appropriate safeguards.