[Paper Review] Reducing malicious use of synthetic media research: Considerations and potential release practices for machine learning

Aviv Ovadya, Jess Whittlestone|arXiv (Cornell University)|Jul 25, 2019

Explainable Artificial Intelligence (XAI)7 references12 citations

TL;DR

This paper proposes a nuanced framework for managing the release of machine learning research on synthetic media to reduce malicious use, advocating for context-sensitive, expert-informed release practices rather than blanket openness or secrecy. It outlines risk mitigation strategies, promotes community norms for impact assessment, and recommends institutional support for responsible research dissemination to balance innovation with societal safety.

ABSTRACT

The aim of this paper is to facilitate nuanced discussion around research norms and practices to mitigate the harmful impacts of advances in machine learning (ML). We focus particularly on the use of ML to create "synthetic media" (e.g. to generate or manipulate audio, video, images, and text), and the question of what publication and release processes around such research might look like, though many of the considerations discussed will apply to ML research more broadly. We are not arguing for any specific approach on when or how research should be distributed, but instead try to lay out some useful tools, analogies, and options for thinking about these issues. We begin with some background on the idea that ML research might be misused in harmful ways, and why advances in synthetic media, in particular, are raising concerns. We then outline in more detail some of the different paths to harm from ML research, before reviewing research risk mitigation strategies in other fields and identifying components that seem most worth emulating in the ML and synthetic media research communities. Next, we outline some important dimensions of disagreement on these issues which risk polarizing conversations. Finally, we conclude with recommendations, suggesting that the machine learning community might benefit from: working with subject matter experts to increase understanding of the risk landscape and possible mitigation strategies; building a community and norms around understanding the impacts of ML research, e.g. through regular workshops at major conferences; and establishing institutions and systems to support release practices that would otherwise be onerous and error-prone.

Motivation & Objective

• To address growing concerns about the malicious use of synthetic media generated via machine learning, particularly deepfakes and AI-generated disinformation.
• To reduce polarization in the ML community around open vs. restricted research release by proposing a spectrum of release practices grounded in risk analysis.
• To support the development of community norms and institutional mechanisms that enable responsible research dissemination without compromising safety or innovation.
• To encourage collaboration between ML researchers and subject matter experts in misinformation, security, and policy to better understand and mitigate real-world harms.
• To establish structured, repeatable processes for evaluating and managing the risks of releasing potentially harmful ML research, especially in synthetic media.

Proposed method

• Develop a taxonomy of research hazards—product, data, and attention hazards—based on potential misuse pathways.
• Adapt risk mitigation strategies from other high-stakes fields (e.g., dual-use research in biotechnology and nuclear science) to machine learning contexts.
• Propose a range of release options beyond binary open/closed models, including delayed release, restricted access, and redacted publication.
• Advocate for expert impact evaluation of research proposals, involving domain specialists to assess risks and mitigation potential.
• Design prototype vetting systems to securely share sensitive models, reducing reliance on ad-hoc verification by individual researchers.
• Establish recurring workshops and community forums to institutionalize impact assessment and foster shared norms around responsible research practices.

Experimental results

Research questions

• RQ1 How can machine learning research on synthetic media be released in ways that minimize malicious use while preserving scientific openness?
• RQ2 What are the key types of hazards (product, data, attention) associated with synthetic media research, and how do they differ in risk profile?
• RQ3 How can the machine learning community develop shared norms and institutional structures to evaluate and manage the societal risks of ML research?
• RQ4 What lessons can be drawn from dual-use research in other scientific fields for managing risks in synthetic media ML research?
• RQ5 What release practices can balance the values of transparency, innovation, and societal safety in ML research?

Key findings

• The paper identifies three distinct hazard types—product, data, and attention hazards—that help categorize how ML research on synthetic media can enable malicious use.
• It demonstrates that the debate over research release is not a simple binary between open and closed access, but involves a spectrum of context-sensitive options.
• The authors find that expert impact evaluation and institutional support can significantly reduce the burden on individual researchers while improving risk assessment accuracy.
• They show that current ad-hoc methods for verifying requesters of sensitive models are error-prone and unsustainable, necessitating scalable vetting systems.
• The paper concludes that responsible release practices are feasible and necessary, and that institutionalizing impact assessment can help the ML community mature in its societal responsibilities.
• It emphasizes that proactive engagement with affected communities and subject matter experts is essential to avoid risk hyperbole and ensure realistic, actionable risk mapping.