[Paper Review] On Hard Exploration for Reinforcement Learning: a Case Study in Pommerman

Chao Gao, Bilal Kartal+2 more|arXiv (Cornell University)|Jul 26, 2019

Reinforcement Learning in Robotics4 citations

TL;DR

This paper addresses the challenge of hard exploration in multi-agent reinforcement learning using Pommerman as a case study. It proposes a model-based action-pruning module that prevents suicidal bomb placements, enabling sample-efficient learning in environments with sparse, delayed, and deceptive rewards. The method enables PPO agents to achieve strong performance against static and baseline opponents, even when model-free RL fails entirely without pruning.

ABSTRACT

How to best explore in domains with sparse, delayed, and deceptive rewards is an important open problem for reinforcement learning (RL). This paper considers one such domain, the recently-proposed multi-agent benchmark of Pommerman. This domain is very challenging for RL --- past work has shown that model-free RL algorithms fail to achieve significant learning without artificially reducing the environment's complexity. In this paper, we illuminate reasons behind this failure by providing a thorough analysis on the hardness of random exploration in Pommerman. While model-free random exploration is typically futile, we develop a model-based automatic reasoning module that can be used for safer exploration by pruning actions that will surely lead the agent to death. We empirically demonstrate that this module can significantly improve learning.

Motivation & Objective

• To analyze why model-free reinforcement learning fails in Pommerman due to high suicide rates during random exploration.
• To identify that delayed action effects and deceptive rewards make safe exploration extremely difficult in Pommerman.
• To design a model-based reasoning module that prunes actions leading to certain death, improving exploration safety.
• To empirically validate that action pruning enables sample-efficient learning in both Free-For-All and Team modes.
• To demonstrate that agents trained against static opponents can outperform the SimpleAgent baseline in competitive settings.

Proposed method

• The paper introduces a model-based reasoning module that analyzes bomb placement actions for safety before execution.
• The module uses a forward simulation of bomb explosions over 10 time steps to predict whether any agent will be killed.
• It prunes actions that result in the agent’s own death, based on the predicted explosion radius and agent positions.
• The module is integrated with PPO, allowing the agent to explore safely while maintaining policy learning.
• The method is evaluated in both Free-For-All and Team modes against static opponents and the SimpleAgent baseline.
• A reward shaping scheme is used to encourage movement, power-up collection, and enemy elimination.

Experimental results

Research questions

• RQ1 Why does model-free reinforcement learning fail to learn in Pommerman despite using advanced algorithms like PPO?
• RQ2 How does random exploration lead to high rates of agent suicide, and what is the impact on sample efficiency?
• RQ3 Can a model-based action-pruning module significantly improve learning in environments with delayed and sparse rewards?
• RQ4 To what extent can agents trained against static opponents generalize to compete with rule-based baselines like SimpleAgent?
• RQ5 What role does safe exploration play in enabling sample-efficient learning in complex grid-world environments?

Key findings

• Without action pruning, PPO agents failed to achieve any success after 5 days of training against static opponents in both FFA and Team modes.
• With the action-pruning module, PPO agents achieved consistent success in both FFA and Team modes, learning the pure exploration task effectively.
• The final agent won approximately 40% of games against three SimpleAgent opponents in FFA mode, exceeding the expected 25% win rate for equally skilled agents.
• In Team mode, the agent won about 70% of games against a SimpleAgent team, demonstrating strong generalization from static opponent training.
• The action-pruning module reduced the number of suicidal actions during exploration, directly improving sample efficiency and learning stability.
• An early version of this module contributed to a second-place finish in the NeurIPS 2018 learning category competition, validating its practical effectiveness.