[Paper Review] Agent-Time Attention for Sparse Rewards Multi-Agent Reinforcement Learning

Jennifer She, Jayesh K. Gupta+1 more|arXiv (Cornell University)|Oct 31, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes Agent-Time Attention (ATA), a neural network model that redistributes sparse and delayed global rewards in multi-agent reinforcement learning by jointly modeling credit assignment across agents and time using attention mechanisms. ATA outperforms strong baselines on extended MiniGrid environments with sparse rewards, demonstrating superior credit assignment and policy learning in collaborative MARL settings.

ABSTRACT

Sparse and delayed rewards pose a challenge to single agent reinforcement learning. This challenge is amplified in multi-agent reinforcement learning (MARL) where credit assignment of these rewards needs to happen not only across time, but also across agents. We propose Agent-Time Attention (ATA), a neural network model with auxiliary losses for redistributing sparse and delayed rewards in collaborative MARL. We provide a simple example that demonstrates how providing agents with their own local redistributed rewards and shared global redistributed rewards motivate different policies. We extend several MiniGrid environments, specifically MultiRoom and DoorKey, to the multi-agent sparse delayed rewards setting. We demonstrate that ATA outperforms various baselines on many instances of these environments. Source code of the experiments is available at https://github.com/jshe/agent-time-attention.

Motivation & Objective

• To address the challenge of sparse and delayed rewards in collaborative multi-agent reinforcement learning (MARL), where credit assignment is difficult across both time and agents.
• To develop a method that redistributes global sparse rewards into dense, local rewards for individual agents while preserving coordination.
• To enable effective training of decentralized policies under the centralized training decentralized execution (CTDE) paradigm with minimal modifications to existing RL algorithms.
• To evaluate the model on environments with long horizons and sparse rewards, such as extended MiniGrid environments (MultiRoom, DoorKey).

Proposed method

• ATA uses a transformer-based architecture to compute attention over agent-specific and temporal representations for reward redistribution.
• The model learns to predict both global team rewards and individual agent rewards using auxiliary prediction losses.
• It employs a dual supervision strategy: predicting rewards at intermediate timesteps and at the final timestep to improve temporal credit assignment.
• The reward redistribution is trained via a combination of global and local prediction losses, with ablation studies to evaluate loss component importance.
• The method is compatible with standard policy gradient methods like IPG and IAC, enabling plug-and-play integration into existing MARL frameworks.
• Architectural ablations compare LSTM vs. transformer encoders and mean vs. attention-based aggregation for internal representations.

Experimental results

Research questions

• RQ1 How does joint credit assignment across agents and time improve learning in sparse-reward multi-agent environments?
• RQ2 What is the relative contribution of temporal vs. agent-wise credit assignment in MARL with delayed rewards?
• RQ3 How do architectural choices (e.g., transformer vs. LSTM, attention vs. mean aggregation) affect the performance of reward redistribution models?
• RQ4 What loss configurations (global/local, intermediate/final timesteps) lead to optimal policy learning in sparse-reward MARL?
• RQ5 Can ATA generalize across diverse multi-agent environments with long horizons and sparse rewards?

Key findings

• ATA outperforms strong baselines, including methods that perform credit assignment only along the agent axis or time axis, on extended MiniGrid environments with sparse rewards.
• Training with intermediate timestep prediction losses significantly improves performance, suggesting that early supervision enhances temporal credit assignment.
• The transformer architecture consistently outperforms LSTM in early training, indicating better sequence modeling for credit assignment.
• Simple mean aggregation of agent representations often outperforms more complex attention mechanisms, possibly due to gradient stability in low-agent settings.
• The combination of global and local prediction losses does not consistently improve performance; in some cases, only global loss yields the best results.
• IAC performs poorly with or without reward redistribution, likely due to poor critic signal in sparse-reward settings, highlighting the importance of robust value learning.