[Paper Review] AMBER: Adaptive Multi-Batch Experience Replay for Continuous Action Control

Seungyul Han, Youngchul Sung|arXiv (Cornell University)|Oct 12, 2017

Reinforcement Learning in Robotics14 references5 citations

TL;DR

This paper proposes AMBER, an adaptive multi-batch experience replay method for proximal policy optimization (PPO) in continuous control tasks. By dynamically selecting past policy batches based on average importance sampling (IS) weights and using pre-computed advantages and values, AMBER improves sample efficiency and reduces gradient variance, resulting in faster and more stable training than standard PPO across multiple MuJoCo and OpenAI Gym environments.

ABSTRACT

In this paper, a new adaptive multi-batch experience replay scheme is proposed for proximal policy optimization (PPO) for continuous action control. On the contrary to original PPO, the proposed scheme uses the batch samples of past policies as well as the current policy for the update for the next policy, where the number of the used past batches is adaptively determined based on the oldness of the past batches measured by the average importance sampling (IS) weight. The new algorithm constructed by combining PPO with the proposed multi-batch experience replay scheme maintains the advantages of original PPO such as random mini-batch sampling and small bias due to low IS weights by storing the pre-computed advantages and values and adaptively determining the mini-batch size. Numerical results show that the proposed method significantly increases the speed and stability of convergence on various continuous control tasks compared to original PPO.

Motivation & Objective

• To address the limited sample efficiency and instability in standard PPO for continuous control by reusing past policy experience.
• To overcome the challenge of high variance in importance sampling (IS) weights when replaying old experiences in IS-based policy gradient methods.
• To maintain PPO's low bias and random mini-batch sampling advantages while enabling effective experience replay.
• To develop an adaptive mechanism that selects only relevant past batches based on IS weight decay, avoiding outdated or high-variance samples.
• To improve convergence speed and stability without requiring manual tuning of replay memory size or clipping hyperparameters.

Proposed method

• AMBER uses a replay buffer that stores pre-computed advantages and values from past policy updates, enabling efficient replay without recomputation.
• It adaptively selects a subset of past policy batches for each update based on the average IS weight, which measures the 'oldness' of the batch.
• The number of active past batches is dynamically adjusted using a batch drop factor that linearly decays over time to remove outdated samples.
• Mini-batches are drawn randomly from the combined set of current and active past policy experiences, reducing sample correlation and improving gradient stability.
• The method integrates with PPO by applying clipping to the IS weight ratio, preserving low bias and stable optimization.
• The algorithm maintains a fixed number of updates per iteration by scaling the mini-batch size as the number of active batches increases.

Experimental results

Research questions

• RQ1 Can experience replay be effectively applied to IS-based policy gradient methods like PPO without introducing high variance due to large IS weights?
• RQ2 How can past policy batches be selected adaptively to maximize sample efficiency while minimizing bias and variance?
• RQ3 Does random mini-batch sampling from a multi-batch replay buffer reduce sample correlation and improve training stability compared to episodic sampling?
• RQ4 Can an adaptive mechanism for dropping old batches improve performance without requiring manual tuning of replay memory size?
• RQ5 How does the proposed method compare to standard PPO and other PG methods (e.g., TRPO, ACER) in terms of convergence speed and final performance on continuous control tasks?

Key findings

• PPO-AMBER with $\epsilon_b = 0.25$ automatically selects an optimal number of active batches (from 1 to 8), eliminating the need for manual replay size tuning.
• PPO-AMBER achieves significantly faster and more stable convergence than standard PPO across all tested environments, including HalfCheetah, Pendulum, Swimmer, and Humanoid.
• The performance gap between PPO and PPO-AMBER is most pronounced in tasks like Swimmer, where PPO struggles with instability due to high variance, while PPO-AMBER averages over more samples and improves robustness.
• Random mini-batch sampling from the multi-batch replay buffer outperforms episodic mini-batch sampling, confirming that reduced sample correlation enhances learning.
• PPO-AMBER outperforms both TRPO and ACER in final performance and training stability, demonstrating its effectiveness in continuous control.
• The method is most effective in low-action-dimensional tasks; performance gains diminish in high-dimensional tasks like Humanoid due to increased IS weights, suggesting a need for smaller learning rates or task-specific adaptation.