[Paper Review] A new Potential-Based Reward Shaping for Reinforcement Learning Agent

Babak Badnava, Mona Esmaeili+2 more|arXiv (Cornell University)|Feb 17, 2019

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes a novel potential-based reward shaping method that extracts knowledge from the agent's own learning process by using episode cumulative rewards to shape rewards dynamically. By modeling the potential function based on relative performance (e.g., vs. best/worst episodes), the method accelerates learning in both single- and multi-task reinforcement learning environments, showing significant improvement in learning speed and final performance on Atari games like Breakout and Pong.

ABSTRACT

Potential-based reward shaping (PBRS) is a particular category of machine learning methods which aims to improve the learning speed of a reinforcement learning agent by extracting and utilizing extra knowledge while performing a task. There are two steps in the process of transfer learning: extracting knowledge from previously learned tasks and transferring that knowledge to use it in a target task. The latter step is well discussed in the literature with various methods being proposed for it, while the former has been explored less. With this in mind, the type of knowledge that is transmitted is very important and can lead to considerable improvement. Among the literature of both the transfer learning and the potential-based reward shaping, a subject that has never been addressed is the knowledge gathered during the learning process itself. In this paper, we presented a novel potential-based reward shaping method that attempted to extract knowledge from the learning process. The proposed method extracts knowledge from episodes' cumulative rewards. The proposed method has been evaluated in the Arcade learning environment and the results indicate an improvement in the learning process in both the single-task and the multi-task reinforcement learner agents.

Motivation & Objective

• To address the gap in knowledge extraction from the learning process itself, which is underexplored in transfer learning and reward shaping.
• To develop a method that leverages performance feedback from past episodes to guide future learning.
• To improve sample efficiency and learning speed in reinforcement learning without altering the optimal policy.
• To extend the method to multi-task RL, where each task maintains its own optimized potential function.
• To empirically validate the method in the Arcade Learning Environment using standard baselines.

Proposed method

• The method defines a potential function φ(s,a,t,i) based on the agent’s episode reward R^ep(i) relative to upper and lower bounds R_u^ep(t,i) and R_l^ep(t,i).
• The potential function is constructed as φ(s,a,t,i) = 1 + (R^ep(i) - R_u^ep(t,i)) / (R_u^ep(t,i) - R_l^ep(t,i)) when R(s,a) ≠ 0, otherwise 0.
• The reward shaping function F is derived from the potential using the standard PBRS formula: F(s,s') = γφ(s') - φ(s), preserving the optimal policy.
• The method dynamically updates the potential function using running estimates of episode reward bounds during training.
• The approach is compatible with any RL algorithm and is applied to Q-learning and policy gradient methods.
• For multi-task RL, the method uses task-specific parameters in the potential function (equation 23), ensuring each task maintains its own optimal policy.

Experimental results

Research questions

• RQ1 Can knowledge extracted from the agent’s own learning process—specifically episode cumulative rewards—improve sample efficiency in reinforcement learning?
• RQ2 How does shaping rewards based on relative performance (e.g., best vs. worst episodes) affect learning speed and final policy quality?
• RQ3 Can the proposed method be effectively extended to multi-task reinforcement learning while preserving task-specific optimal policies?
• RQ4 Does the method outperform standard baselines in sparse-reward environments like Atari games?

Key findings

• In the Breakout environment, the proposed method achieved a higher area under the learning curve compared to both baselines, indicating faster learning and better final performance.
• In the Pong environment, the method showed modest improvement, with a greater area under the curve than one baseline, though gains were less pronounced than in Breakout.
• For the multi-task agent learning Pong and Breakout simultaneously, the method led to better average performance over 100 test episodes compared to the baseline without reward shaping.
• The method successfully maintained the optimal policy in all tested environments by using a valid potential function, as per PBRS theory.
• The approach is robust under two assumptions: when maximum and minimum episode rewards are known in advance, and when they are estimated during training.
• The results demonstrate that self-extracted knowledge from episode-level performance can significantly enhance learning speed and policy quality in RL.