[Paper Review] Curious Exploration and Return-based Memory Restoration for Deep Reinforcement Learning

Saeed Tafazzol, Erfan Fathi+2 more|arXiv (Cornell University)|May 2, 2021

Reinforcement Learning in Robotics11 references4 citations

TL;DR

This paper proposes a reinforcement learning framework combining Curious Exploration and Return-based Memory Restoration (RMR) to train agents in sparse-reward environments using only binary success/failure rewards. By leveraging curiosity-driven exploration and prioritizing valuable memories via RMR, the method enables DDPG agents to reliably learn to score goals in the complex Half Field Offense soccer simulation, outperforming both standard DDPG and curiosity-only baselines with a 97.8% success rate on average.

ABSTRACT

Reward engineering and designing an incentive reward function are non-trivial tasks to train agents in complex environments. Furthermore, an inaccurate reward function may lead to a biased behaviour which is far from an efficient and optimised behaviour. In this paper, we focus on training a single agent to score goals with binary success/failure reward function in Half Field Offense domain. As the major advantage of this research, the agent has no presumption about the environment which means it only follows the original formulation of reinforcement learning agents. The main challenge of using such a reward function is the high sparsity of positive reward signals. To address this problem, we use a simple prediction-based exploration strategy (called Curious Exploration) along with a Return-based Memory Restoration (RMR) technique which tends to remember more valuable memories. The proposed method can be utilized to train agents in environments with fairly complex state and action spaces. Our experimental results show that many recent solutions including our baseline method fail to learn and perform in complex soccer domain. However, the proposed method can converge easily to the nearly optimal behaviour. The video presenting the performance of our trained agent is available at http://bit.ly/HFO_Binary_Reward.

Motivation & Objective

• To address the challenge of training deep reinforcement learning agents in environments with sparse, binary success/failure rewards, where traditional reward shaping is impractical.
• To eliminate reliance on hand-crafted reward functions by using only the original RL formulation with environment-provided next state and binary reward signals.
• To improve sample efficiency and learning convergence in complex action and state spaces by enhancing exploration and memory retention.
• To demonstrate that agents can achieve nearly optimal behavior in the HFO domain using only intrinsic curiosity and memory prioritization, without reward engineering.

Proposed method

• Introduces Curious Exploration, a prediction-based exploration strategy where the agent is driven by prediction error in a learned dynamics model to seek novel, uncertain states.
• Employs a Return-based Memory Restoration (RMR) mechanism that prioritizes storing and replaying experiences with high cumulative return, increasing the density of valuable memories in the replay buffer.
• Uses a standard DDPG algorithm as the backbone policy learner, operating solely on the original RL formulation with no reward shaping.
• Trains a state dynamics predictor to estimate the next state from current state and action, with prediction error used as intrinsic curiosity reward.
• Applies RMR to modify the memory storage policy, favoring transitions that lead to higher return, thus preserving critical positive experience trajectories.
• Combines the intrinsic curiosity reward with the environment’s binary reward to guide exploration and learning in a sparse-reward setting.

Experimental results

Research questions

• RQ1 Can a DRL agent learn to score goals in the Half Field Offense domain using only a binary success/failure reward and no hand-crafted reward shaping?
• RQ2 How does Curious Exploration improve learning efficiency in sparse-reward environments compared to random exploration or standard DDPG?
• RQ3 To what extent does Return-based Memory Restoration enhance sample efficiency and learning convergence by preserving high-return experiences?
• RQ4 Can the combination of curiosity and memory restoration overcome the failure of standard DDPG in sparse-reward settings?

Key findings

• The DDPG agent with only a binary reward function failed completely, achieving 0.000 average success rate across five training rounds.
• The DDPG + Curious Exploration (CE) agent achieved a 91.9% average success rate, demonstrating that curiosity alone improves learning in sparse environments.
• The DDPG + CE + RMR agent achieved a 97.8% average success rate, significantly outperforming both baselines and showing faster convergence to optimal behavior.
• The RMR mechanism effectively preserved critical positive experience trajectories, preventing early forgetting of sparse positive signals and enabling stable learning.
• The spike in prediction loss and exploration reward at the time of first ball interaction and first goal confirmed that curiosity drives the agent toward key task-relevant behaviors.
• The method successfully enabled learning in a complex, high-dimensional action and state space environment without any reward engineering, relying only on the standard RL formulation.