[Paper Review] Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning

Baicen Xiao, Bhaskar Ramasubramanian+1 more|arXiv (Cornell University)|Jan 12, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes AREL, an attention-based method for dense reward redistribution in episodic multi-agent reinforcement learning. By combining temporal attention over trajectories and agent attention over joint states, AREL identifies critical states and redistributes episodic rewards to improve credit assignment, resulting in higher cumulative rewards and improved win rates compared to state-of-the-art methods on Particle World and StarCraft.

ABSTRACT

This paper considers multi-agent reinforcement learning (MARL) tasks where agents receive a shared global reward at the end of an episode. The delayed nature of this reward affects the ability of the agents to assess the quality of their actions at intermediate time-steps. This paper focuses on developing methods to learn a temporal redistribution of the episodic reward to obtain a dense reward signal. Solving such MARL problems requires addressing two challenges: identifying (1) relative importance of states along the length of an episode (along time), and (2) relative importance of individual agents' states at any single time-step (among agents). In this paper, we introduce Agent-Temporal Attention for Reward Redistribution in Episodic Multi-Agent Reinforcement Learning (AREL) to address these two challenges. AREL uses attention mechanisms to characterize the influence of actions on state transitions along trajectories (temporal attention), and how each agent is affected by other agents at each time-step (agent attention). The redistributed rewards predicted by AREL are dense, and can be integrated with any given MARL algorithm. We evaluate AREL on challenging tasks from the Particle World environment and the StarCraft Multi-Agent Challenge. AREL results in higher rewards in Particle World, and improved win rates in StarCraft compared to three state-of-the-art reward redistribution methods. Our code is available at https://github.com/baicenxiao/AREL.

Motivation & Objective

• To address the challenge of delayed or episodic rewards in cooperative multi-agent reinforcement learning, where agents receive a global reward only at episode end.
• To improve credit assignment by identifying the relative importance of states along the episode timeline (temporal dimension) and across agents at each time-step (agent dimension).
• To develop a scalable, attention-based method that avoids joint observation space explosion in multi-agent settings.
• To enable integration with existing MARL algorithms by generating dense, learnable reward signals.
• To demonstrate that effective reward redistribution can outperform strategic exploration in episodic MARL environments.

Proposed method

• AREL employs a shared temporal attention module that processes each agent’s individual observation sequence to model long-term dependencies and identify influential state transitions.
• It uses an agent attention module to compute the relative contribution of each agent’s state to the global reward at each time-step, based on attention weights over agents.
• The output of the agent attention module is used to generate a dense, temporally redistributed reward signal for each agent at each time-step.
• The method integrates a variance-based regularization term in the loss function to encourage sparse but meaningful reward distribution across critical states.
• The redistributed rewards are compatible with any value-based MARL algorithm, such as QMIX or MADDPG, enabling plug-and-play integration.
• AREL does not modify the underlying MARL policy; it acts as a reward shaping module to improve learning efficiency.

Experimental results

Research questions

• RQ1 Can attention mechanisms effectively identify critical states in long-horizon MARL tasks with episodic rewards?
• RQ2 How can temporal and agent-wise credit assignment be jointly modeled in a scalable way without requiring joint observation processing?
• RQ3 Does dense reward redistribution via attention lead to improved learning performance compared to state-of-the-art credit assignment methods?
• RQ4 To what extent does effective reward redistribution outperform strategic exploration in episodic MARL?
• RQ5 Can the attention mechanism learn interpretable and semantically meaningful reward signals aligned with task objectives?

Key findings

• AREL achieved higher cumulative rewards than three state-of-the-art reward redistribution methods in the Particle World environment, demonstrating improved learning efficiency.
• In the StarCraft Multi-Agent Challenge, AREL improved win rates compared to baseline methods, indicating better policy learning under episodic rewards.
• The redistributed rewards generated by AREL were not uniform across the episode, with higher values assigned to states where agents coordinated effectively, such as navigating toward distinct landmarks.
• Visualizations showed that AREL’s attention mechanism learned to assign higher rewards when agents covered different landmarks, aligning with task objectives.
• Experiments revealed that effective reward redistribution was more beneficial than strategic exploration (e.g., MAVEN) in episodic settings, especially when rewards were delayed.
• The method successfully scaled to multi-agent settings without requiring joint observation processing, maintaining computational efficiency.