[Paper Review] Learning Altruistic Behaviours in Reinforcement Learning without External Rewards

Tim Franzmeyer, Mateusz Malinowski+1 more|arXiv (Cornell University)|Jul 20, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a reinforcement learning framework where agents learn altruistic behavior by maximizing the future choice (state reachability) of other agents, without external rewards or knowledge of their goals. The method uses intrinsic curiosity to estimate and maximize the leader agent's potential future states, achieving performance comparable to or better than supervised baselines in multi-agent environments.

ABSTRACT

Can artificial agents learn to assist others in achieving their goals without knowing what those goals are? Generic reinforcement learning agents could be trained to behave altruistically towards others by rewarding them for altruistic behaviour, i.e., rewarding them for benefiting other agents in a given situation. Such an approach assumes that other agents' goals are known so that the altruistic agent can cooperate in achieving those goals. However, explicit knowledge of other agents' goals is often difficult to acquire. In the case of human agents, their goals and preferences may be difficult to express fully; they might be ambiguous or even contradictory. Thus, it is beneficial to develop agents that do not depend on external supervision and learn altruistic behaviour in a task-agnostic manner. We propose to act altruistically towards other agents by giving them more choice and allowing them to achieve their goals better. Some concrete examples include opening a door for others or safeguarding them to pursue their objectives without interference. We formalize this concept and propose an altruistic agent that learns to increase the choices another agent has by preferring to maximize the number of states that the other agent can reach in its future. We evaluate our approach in three different multi-agent environments where another agent's success depends on altruistic behaviour. Finally, we show that our unsupervised agents can perform comparably to agents explicitly trained to work cooperatively, in some cases even outperforming them.

Motivation & Objective

• To develop agents that act altruistically without knowing other agents' goals or receiving external rewards.
• To investigate whether maximizing an agent's future choice (state reachability) serves as a general proxy for altruism.
• To evaluate if unsupervised, intrinsic reward-based learning can match or exceed supervised cooperative training in multi-agent environments.
• To identify failure modes where maximizing choice may lead to suboptimal behavior despite theoretical justification.

Proposed method

• The altruistic agent uses an intrinsic reward signal based on the leader agent’s future state reachability, computed via an intrinsic curiosity (IC) module.
• The IC is estimated using a predictive model that forecasts the leader’s state distribution over time, encouraging the altruistic agent to preserve or expand the leader’s options.
• The method is implemented in a multi-agent reinforcement learning framework where the altruistic agent observes all agents but is not observed, and acts to maximize the leader’s choice.
• The approach is trained using standard deep RL algorithms (e.g., SAC) with the intrinsic curiosity reward, without any extrinsic reward shaping.
• Three environments—Gridworld, Locomotion, and Tag—are used to evaluate the method under varying degrees of task-agnostic cooperation.
• The method is compared against random, boundary-penalized, and supervised baselines, including a variant with shared rewards.

Experimental results

Research questions

• RQ1 Can agents learn to act altruistically toward others without knowing their goals or receiving external rewards?
• RQ2 Does maximizing the number of reachable future states for another agent serve as a valid proxy for altruism in diverse environments?
• RQ3 How does the performance of an unsupervised altruistic agent compare to supervised cooperative agents in multi-agent settings?
• RQ4 What are the failure modes of the choice-maximization objective, and how do they depend on hyperparameters?

Key findings

• In the Tag environment, the proposed method reduced the number of times the leader was caught to 2.87 ± 0.96 per episode, significantly outperforming the supervised baseline (6.94 ± 1.13).
• The unsupervised altruistic agents achieved better performance than random, cage-penalized, and even supervised agents in the Tag environment, indicating the effectiveness of the intrinsic curiosity-based reward.
• In the Locomotion environment, the method outperformed all unsupervised baselines and matched or exceeded supervised performance, demonstrating robustness across tasks.
• The Gridworld experiments showed that the altruistic agents successfully opened doors and enabled the leader to reach more states, confirming the mechanism’s functional success.
• The method identified failure cases where maximizing choice led to suboptimal behavior, particularly when the leader’s goals were not aligned with state diversity, highlighting the need for careful hyperparameter tuning.
• Results suggest that intrinsic curiosity-based choice maximization is a viable, scalable, and effective proxy for altruism in multi-agent RL, especially when goal information is unavailable.