[Paper Review] Boosting Soft Actor-Critic: Emphasizing Recent Experience without Forgetting the Past

Che Wang, Keith W. Ross|arXiv (Cornell University)|Jun 10, 2019

Reinforcement Learning in Robotics34 references40 citations

TL;DR

The paper introduces Emphasizing Recent Experience (ERE) to improve SAC by biasing replay toward recent data while preserving older experiences; it also explores combining ERE with Prioritized Experience Replay (PER) and evaluates on Mujoco environments.

ABSTRACT

Soft Actor-Critic (SAC) is an off-policy actor-critic deep reinforcement learning (DRL) algorithm based on maximum entropy reinforcement learning. By combining off-policy updates with an actor-critic formulation, SAC achieves state-of-the-art performance on a range of continuous-action benchmark tasks, outperforming prior on-policy and off-policy methods. The off-policy method employed by SAC samples data uniformly from past experience when performing parameter updates. We propose Emphasizing Recent Experience (ERE), a simple but powerful off-policy sampling technique, which emphasizes recently observed data while not forgetting the past. The ERE algorithm samples more aggressively from recent experience, and also orders the updates to ensure that updates from old data do not overwrite updates from new data. We compare vanilla SAC and SAC+ERE, and show that ERE is more sample efficient than vanilla SAC for continuous-action Mujoco tasks. We also consider combining SAC with Priority Experience Replay (PER), a scheme originally proposed for deep Q-learning which prioritizes the data based on temporal-difference (TD) error. We show that SAC+PER can marginally improve the sample efficiency performance of SAC, but much less so than SAC+ERE. Finally, we propose an algorithm which integrates ERE and PER and show that this hybrid algorithm can give the best results for some of the Mujoco tasks.

Motivation & Objective

• Motivate the need to utilize replay data more effectively in off-policy DRL beyond uniform sampling.
• Propose Emphasizing Recent Experience (ERE) to prioritize recent transitions while retaining past data.
• Compare SAC+ERE with SAC, SAC+PER, and SAC+ERE+PER on continuous-control tasks.
• Assess simplicity, computational cost, and robustness of ERE relative to PER.
• Provide guidance on hyperparameters and update-order importance for ERE.

Proposed method

• Describe Soft Actor-Critic (SAC) and its uniform replay baseline.
• Introduce SAC+ERE: sample from the most recent data with an ordered update scheme to avoid overwriting new updates by old data.
• Introduce SAC+PER: apply proportional Prioritized Experience Replay to SAC using TD-error-based priorities.
• Propose SAC+ERE+PER: combine non-uniform, recent-data sampling with TD-error prioritization.
• Provide pseudocode and discuss implementation simplicity and hyperparameter sensitivity.
• Evaluate on MuJoCo continuous-control tasks with multiple seeds and fixed architectures.

Experimental results

Research questions

• RQ1 Does emphasizing recent experience (ERE) improve SAC sample efficiency without sacrificing robustness?
• RQ2 How does ERE compare to Prioritized Experience Replay (PER) for SAC in terms of performance gains and complexity?
• RQ3 Does combining ERE with PER (SAC+ERE+PER) yield additional benefits over each method alone?
• RQ4 What are the key hyperparameters (e.g., eta, c_min) and update-order effects that govern ERE's performance?
• RQ5 Are the observed gains of ERE generalizable across MuJoCo environments and seeds?

Key findings

• SAC+ERE consistently outperforms vanilla SAC across the six MuJoCo environments in early and later training stages.
• SAC+ERE can reach higher performance faster (e.g., Ant-v2 example) with relatively robust behavior under eta values in (0.994,0.999) and annealing.
• SAC+PER can improve performance in some environments (notably Ant-v2) but is less consistently beneficial than SAC+ERE across environments.
• SAC+ERE+PER can yield the best results in some environments, though with higher computational cost and less simplicity than SAC+ERE alone.
• SAC+ERE shows robustness improvements reflected by lower or comparable performance variability across seeds in several environments at 1.5M timesteps.