[Paper Review] Curiosity-driven Exploration in Sparse-reward Multi-agent Reinforcement Learning

Jiong Li, Pratik Gajane|arXiv (Cornell University)|Feb 21, 2023

Behavioral Health and Interventions4 citations

TL;DR

This paper proposes I-Go-Explore, a novel method that combines the Intrinsic Curiosity Module (ICM) with the Go-Explore framework to enhance sample efficiency in sparse-reward multi-agent reinforcement learning. By integrating intrinsic curiosity with Go-Explore’s trajectory-based exploration, the method reduces detachment and derailment, achieving significantly faster training and improved performance—especially in long-horizon tasks—demonstrated by a 12% reduction in training time and superior cumulative rewards in multi-agent predator-prey environments.

ABSTRACT

Sparsity of rewards while applying a deep reinforcement learning method negatively affects its sample-efficiency. A viable solution to deal with the sparsity of rewards is to learn via intrinsic motivation which advocates for adding an intrinsic reward to the reward function to encourage the agent to explore the environment and expand the sample space. Though intrinsic motivation methods are widely used to improve data-efficient learning in the reinforcement learning model, they also suffer from the so-called detachment problem. In this article, we discuss the limitations of intrinsic curiosity module in sparse-reward multi-agent reinforcement learning and propose a method called I-Go-Explore that combines the intrinsic curiosity module with the Go-Explore framework to alleviate the detachment problem.

Motivation & Objective

• Address the sample inefficiency of deep reinforcement learning in sparse-reward multi-agent environments.
• Overcome the detachment and derailment problems inherent in intrinsic curiosity modules (ICM) during exploration.
• Improve exploration efficiency by integrating ICM with the Go-Explore framework to maintain long-term trajectory diversity.
• Evaluate the method in partially observable, mixed cooperative-competitive multi-agent environments to assess robustness.
• Demonstrate that intrinsic curiosity combined with structured exploration leads to faster convergence and better cumulative rewards.

Proposed method

• Integrate the Intrinsic Curiosity Module (ICM) to generate intrinsic rewards based on prediction error in state transitions.
• Apply the Go-Explore framework to preserve and revisit high-reward trajectories, preventing premature abandonment of promising paths.
• Use a centralized training with decentralized execution (CTDE) setup, leveraging MADDPG for policy optimization.
• Structure exploration into phases where agents first explore via ICM-driven curiosity, then exploit promising trajectories via Go-Explore’s trajectory memory.
• Maintain a memory of discovered high-return trajectories to guide future exploration and reduce detachment.
• Balance intrinsic and extrinsic rewards by combining ICM’s prediction error with environment-provided sparse rewards.

Experimental results

Research questions

• RQ1 Can combining ICM with Go-Explore mitigate the detachment problem in sparse-reward multi-agent reinforcement learning?
• RQ2 Does the I-Go-Explore framework improve sample efficiency compared to standalone ICM in long-horizon tasks?
• RQ3 How does exploration phase length affect the performance of I-Go-Explore in partially observable environments?
• RQ4 To what extent does I-Go-Explore outperform ICM in terms of training time and cumulative reward in mixed cooperative-competitive settings?
• RQ5 Does the integration of intrinsic curiosity with trajectory-based exploration lead to more stable and scalable learning in complex multi-agent environments?

Key findings

• I-Go-Explore reduced training time by approximately 12% compared to ICM, with I-Go-Explore taking 15,805 seconds versus ICM’s 18,005 seconds on the same hardware.
• The prey agent achieved a significantly higher average reward in I-Go-Explore (-268.83) compared to ICM (-786.62), indicating better escape and survival strategies.
• Predators in I-Go-Explore achieved a higher average reward (8.88) than in ICM (16.76), suggesting improved coordination and pursuit efficiency.
• In short-term training (10 steps), ICM slightly outperformed I-Go-Explore for the prey, but in long-term training (20 steps), I-Go-Explore showed superior and more stable performance.
• I-Go-Explore with a shorter exploration phase (10 steps) outperformed both longer-phase I-Go-Explore and ICM, indicating that excessive arbitrary exploration can degrade policy learning.
• The method successfully reduced detachment by preserving and reusing high-return trajectories, enabling agents to maintain progress across episodes.