[Paper Review] Open Problems in Cooperative AI

Allan Dafoe, Edward Hughes+6 more|arXiv (Cornell University)|Dec 15, 2020

Evolutionary Game Theory and CooperationSocial Sciences322 references17 citations

TL;DR

This paper proposes Cooperative AI as a new research direction focused on equipping artificial intelligence systems with the capabilities needed to cooperate effectively with humans and other agents. It outlines key cooperative capabilities—understanding, communication, commitment, and institutions—while emphasizing interdisciplinary integration and the mitigation of risks like coercion and exclusion.

ABSTRACT

Problems of cooperation--in which agents seek ways to jointly improve their welfare--are ubiquitous and important. They can be found at scales ranging from our daily routines--such as driving on highways, scheduling meetings, and working collaboratively--to our global challenges--such as peace, commerce, and pandemic preparedness. Arguably, the success of the human species is rooted in our ability to cooperate. Since machines powered by artificial intelligence are playing an ever greater role in our lives, it will be important to equip them with the capabilities necessary to cooperate and to foster cooperation. We see an opportunity for the field of artificial intelligence to explicitly focus effort on this class of problems, which we term Cooperative AI. The objective of this research would be to study the many aspects of the problems of cooperation and to innovate in AI to contribute to solving these problems. Central goals include building machine agents with the capabilities needed for cooperation, building tools to foster cooperation in populations of (machine and/or human) agents, and otherwise conducting AI research for insight relevant to problems of cooperation. This research integrates ongoing work on multi-agent systems, game theory and social choice, human-machine interaction and alignment, natural-language processing, and the construction of social tools and platforms. However, Cooperative AI is not the union of these existing areas, but rather an independent bet about the productivity of specific kinds of conversations that involve these and other areas. We see opportunity to more explicitly focus on the problem of cooperation, to construct unified theory and vocabulary, and to build bridges with adjacent communities working on cooperation, including in the natural, social, and behavioural sciences.

Motivation & Objective

• To address the growing need for AI systems to cooperate effectively with humans and other agents in real-world settings.
• To identify and systematize core capabilities required for cooperation, such as understanding, communication, commitment, and institutional design.
• To bridge AI research with social sciences by creating a unified theoretical and lexical framework for cooperation problems.
• To study the potential risks of cooperative AI, including exclusion, collusion, and coercion, and to guide development toward human welfare.
• To foster interdisciplinary collaboration between AI, game theory, social choice, and human-computer interaction for scalable cooperation solutions.

Proposed method

• Categorize cooperative opportunities along dimensions such as strategic context, common vs. conflicting interests, and individual vs. planner perspectives.
• Structure cooperative capabilities into four core components: understanding (of world, behavior, preferences, and recursive beliefs), communication (common ground, bandwidth, teaching, mixed motives), commitment (unilateral/multilateral, conditional/unconditional), and institutions (decentralized norms, centralized systems, trust mechanisms).
• Propose the development of training environments and tasks in machine learning where cooperative skills are essential, learnable, and non-trivial.
• Integrate insights from multi-agent systems, game theory, mechanism design, natural language processing, and interpretability to build cooperative AI systems.
• Incorporate human values and ethical norms into cooperative AI through preference learning and alignment techniques.
• Design tools and platforms that support cooperation in populations of human and machine agents, including reputation systems and mediation algorithms.

Experimental results

Research questions

• RQ1 How can AI agents be designed to understand the beliefs, preferences, and behaviors of other agents in cooperative settings?
• RQ2 What communication mechanisms enable efficient, robust, and fair cooperation under mixed-motive conditions?
• RQ3 What forms of cooperative commitment—unilateral, multilateral, conditional, or hardware-embedded—best support long-term coordination?
• RQ4 How can institutions and norms be designed to support scalable, decentralized cooperation among AI and human agents?
• RQ5 What are the risks of cooperative AI capabilities, such as exclusion or coercion, and how can they be mitigated through technical and institutional design?

Key findings

• Cooperative AI is not a mere synthesis of existing AI subfields but a distinct research agenda requiring focused, interdisciplinary conversations.
• The paper identifies four core capabilities—understanding, communication, commitment, and institutions—as essential for building cooperative AI systems.
• Effective cooperation in AI requires integrating insights from game theory, social choice, multi-agent systems, and human-computer interaction.
• The development of cooperative AI must include attention to risks such as collusion, coercion, and exclusion, particularly in systems with high autonomy.
• Training environments and tasks in machine learning should be designed to make cooperative skills both crucial and learnable, enabling scalable cooperation.
• The research agenda calls for a unified theory and shared vocabulary to connect AI research with broader scientific and societal efforts on cooperation.