[Paper Review] Social Influence as Intrinsic Motivation for Multi-Agent Deep Reinforcement Learning

Natasha Jaques, Angeliki Lazaridou+6 more|arXiv (Cornell University)|Oct 19, 2018

Experimental Behavioral Economics Studies262 citations

TL;DR

The paper introduces a social influence intrinsic reward for MARL that measures how much an agent can causally influence others via counterfactual reasoning, coordinating agents and enabling meaningful emergent communication without centralized training.

ABSTRACT

We propose a unified mechanism for achieving coordination and communication in Multi-Agent Reinforcement Learning (MARL), through rewarding agents for having causal influence over other agents' actions. Causal influence is assessed using counterfactual reasoning. At each timestep, an agent simulates alternate actions that it could have taken, and computes their effect on the behavior of other agents. Actions that lead to bigger changes in other agents' behavior are considered influential and are rewarded. We show that this is equivalent to rewarding agents for having high mutual information between their actions. Empirical results demonstrate that influence leads to enhanced coordination and communication in challenging social dilemma environments, dramatically increasing the learning curves of the deep RL agents, and leading to more meaningful learned communication protocols. The influence rewards for all agents can be computed in a decentralized way by enabling agents to learn a model of other agents using deep neural networks. In contrast, key previous works on emergent communication in the MARL setting were unable to learn diverse policies in a decentralized manner and had to resort to centralized training. Consequently, the influence reward opens up a window of new opportunities for research in this area.

Motivation & Objective

• Motivate coordination and communication in multi-agent RL through intrinsic rewards based on social influence.
• Define and compute causal influence using counterfactual reasoning to quantify an agent's impact on others.
• Demonstrate that influence reward aligns with maximizing mutual information between agents' actions to promote coordination.
• Show that influence can enable independent training via internal models of other agents (MOA) and still achieve coordination.

Proposed method

• Define an intrinsic influence reward that quantifies how much an agent changes another agent's action distribution using counterfactual actions.
• Relate the influence reward to mutual information between agents' actions and empirically validate improved coordination.
• Extend the framework to include an explicit communication channel guided by the influence reward and evaluate emergent communication quality.
• Introduce a Model of Other Agents (MOA) to enable independent training and compute influence without centralized access.
• Train policies end-to-end from pixels using recurrent architectures and A3C-style updates, with curriculum learning for the influence weight.

Experimental results

Research questions

• RQ1 Can an intrinsic reward based on causal influence improve coordination in multi-agent environments without centralized training?
• RQ2 Does maximizing causal influence between agents lead to more meaningful emergent communication?
• RQ3 Can agents equipped with a MOA train independently yet still achieve coordinated behavior?
• RQ4 Is the influence reward related to maximizing mutual information between agents' actions in practice?

Key findings

• Agents trained with the social influence reward achieve higher collective rewards than baseline and ablated agents in Sequential Social Dilemmas (SSDs).
• Influence-enabled communication leads to faster learning and higher collective rewards, with more meaningful and coordinated messaging.
• Agents using a MOA can compute influence internally and achieve coordination without centralized controllers, outperforming baselines.
• There is a significant correlation between being influenced by communication and receiving higher individual rewards, supporting informative communication.
• Influence can reduce gradient variance in large MARL setups by creating explicit dependencies among agents’ actions.
• The influence mechanism can elicit emergent communication that aligns with the listener’s environmental rewards and improves cooperation.