[Paper Review] Prioritizing Samples in Reinforcement Learning with Reducible Loss

Shivakanth Sujit, Somjit Nath+2 more|arXiv (Cornell University)|Aug 22, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes Reducible Loss (ReLo), a novel prioritization method for experience replay in deep reinforcement learning that selects samples based on their potential for loss reduction, not just high loss. By measuring the difference between online and target network predictions, ReLo identifies learnable transitions, leading to lower TD errors and improved sample efficiency across environments like DM Control Suite and Atari, outperforming both uniform sampling and prioritized experience replay (PER).

ABSTRACT

Most reinforcement learning algorithms take advantage of an experience replay buffer to repeatedly train on samples the agent has observed in the past. Not all samples carry the same amount of significance and simply assigning equal importance to each of the samples is a naïve strategy. In this paper, we propose a method to prioritize samples based on how much we can learn from a sample. We define the learn-ability of a sample as the steady decrease of the training loss associated with this sample over time. We develop an algorithm to prioritize samples with high learn-ability, while assigning lower priority to those that are hard-to-learn, typically caused by noise or stochasticity. We empirically show that our method is more robust than random sampling and also better than just prioritizing with respect to the training loss, i.e. the temporal difference loss, which is used in prioritized experience replay.

Motivation & Objective

• Address the inefficiency of uniform and loss-based sampling in experience replay by identifying which samples offer the most learning potential.
• Overcome the limitation of prioritized experience replay (PER), which prioritizes high-loss samples that may be unlearnable due to noise or stochasticity.
• Develop a simple, low-overhead method to prioritize samples based on their learnability, defined as the potential to reduce training loss over time.
• Demonstrate that prioritizing based on reducible loss leads to more robust and efficient training across diverse RL benchmarks.
• Provide a theoretically grounded and empirically validated alternative to PER that improves generalization and reduces validation TD error.

Proposed method

• Define reducible loss (ReLo) as the difference between the online Q-network loss and the target network loss for a given transition, capturing the potential for loss reduction.
• Use the target network in off-policy Q-learning algorithms as a proxy for a hold-out model to compute ReLo without additional training.
• Prioritize transitions with higher ReLo values—those where the loss can be meaningfully reduced—during replay buffer sampling.
• Integrate ReLo into existing off-policy RL algorithms like DQN and SAC with minimal code changes, requiring only one additional forward pass per sample.
• Maintain computational efficiency by reusing existing components (e.g., target networks) and avoiding additional model parameters or complex optimization.
• Apply ReLo in place of or alongside PER, enabling flexible integration into standard training pipelines without altering learning dynamics significantly.

Experimental results

Research questions

• RQ1 Can prioritizing samples based on their learnability—measured by reducible loss—lead to faster and more stable learning in deep RL?
• RQ2 How does ReLo compare to uniform sampling and PER in terms of reducing temporal difference (TD) error during training across diverse environments?
• RQ3 Does a lower validation TD error, achieved via ReLo, correlate with improved sample efficiency and final agent performance?
• RQ4 Can ReLo be integrated into existing off-policy RL algorithms with minimal computational overhead?
• RQ5 Does ReLo avoid the pitfalls of PER, such as repeatedly sampling unlearnable high-loss samples due to noise or stochasticity?

Key findings

• ReLo consistently reduces training TD error across environments, with the lowest losses observed in both DM Control Suite and Atari benchmarks compared to uniform sampling and PER.
• On the DM Control Suite, ReLo achieved a validation TD error of 0.12 ± 0.033 on CheetahRun, significantly lower than PER’s 0.03 ± 0.003, indicating better generalization and learning efficiency.
• In QuadrupedRun, ReLo reduced validation TD error to 0.35 ± 0.067, compared to PER’s 2.24 ± 0.127, demonstrating superior performance on complex locomotion tasks.
• On Atari, ReLo maintained or slightly improved performance over Rainbow baseline, with a 1.142 ± 0.174 TD error on Jamesbond versus 1.653 ± 0.819 for Rainbow, showing robustness in discrete control.
• The correlation between lower validation TD error and better sample efficiency was confirmed, supporting ReLo’s effectiveness as a proxy for learning potential.
• ReLo requires only one additional forward pass through the target network, adding negligible computational cost, making it practical for real-world RL deployment.