[Paper Review] On-Policy Trust Region Policy Optimisation with Replay Buffers

Dmitry Kangin, Nicolas Pugeault|arXiv (Cornell University)|Jan 18, 2019

Reinforcement Learning in Robotics28 references4 citations

TL;DR

This paper proposes a novel on-policy trust region reinforcement learning algorithm that integrates replay buffers to reuse data from multiple past policies, improving sample efficiency. By replacing heuristic trust region parameters with a fixed hyperparameter and using a trainable diagonal covariance matrix, the method achieves superior performance over SOTA on-policy (PPO, TRPO, ACKTR) and off-policy (DDPG) methods on MuJoCo environments.

ABSTRACT

Building upon the recent success of deep reinforcement learning methods, we investigate the possibility of on-policy reinforcement learning improvement by reusing the data from several consecutive policies. On-policy methods bring many benefits, such as ability to evaluate each resulting policy. However, they usually discard all the information about the policies which existed before. In this work, we propose adaptation of the replay buffer concept, borrowed from the off-policy learning setting, to create the method, combining advantages of on- and off-policy learning. To achieve this, the proposed algorithm generalises the $Q$-, value and advantage functions for data from multiple policies. The method uses trust region optimisation, while avoiding some of the common problems of the algorithms such as TRPO or ACKTR: it uses hyperparameters to replace the trust region selection heuristics, as well as the trainable covariance matrix instead of the fixed one. In many cases, the method not only improves the results comparing to the state-of-the-art trust region on-policy learning algorithms such as PPO, ACKTR and TRPO, but also with respect to their off-policy counterpart DDPG.

Motivation & Objective

• To improve sample efficiency in on-policy reinforcement learning by reusing data from multiple past policies.
• To address the limitation of on-policy methods discarding prior policy data after each update.
• To eliminate heuristic trust region parameter tuning by replacing it with a fixed hyperparameter.
• To enhance stability and performance through a trainable diagonal covariance matrix in policy optimization.
• To combine the benefits of on-policy evaluation with off-policy data reuse via replay buffers.

Proposed method

• Adapts the replay buffer concept from off-policy learning to on-policy settings, storing transitions from multiple recent policies.
• Generalizes Q-, value, and advantage functions to incorporate data from multiple policies using a weighted combination.
• Employs trust region optimization with a fixed trust region radius, replacing heuristic selection methods used in TRPO and ACKTR.
• Uses a trainable diagonal covariance matrix in the policy network instead of a fixed one, enabling adaptive policy exploration.
• Applies natural gradient optimization with K-FAC-like curvature approximation for stable policy updates.
• Maintains a rolling buffer of past policies, where each transition is associated with its corresponding policy's value and advantage estimates.

Experimental results

Research questions

• RQ1 Can replay buffers from off-policy learning be effectively adapted to on-policy reinforcement learning to improve sample efficiency?
• RQ2 Does replacing heuristic trust region parameters with a fixed hyperparameter improve training stability and performance?
• RQ3 Can a trainable diagonal covariance matrix in the policy network outperform fixed covariance matrices in on-policy trust region optimization?
• RQ4 How does the proposed method compare to state-of-the-art on-policy (PPO, TRPO, ACKTR) and off-policy (DDPG) algorithms in continuous control tasks?
• RQ5 What is the optimal replay buffer depth for balancing data reuse and policy consistency in on-policy training?

Key findings

• The proposed method outperforms PPO, TRPO, and ACKTR on 8 out of 10 MuJoCo environments, with significant improvements on HumanoidStandup and Striker.
• The method achieves better sample efficiency than DDPG on all tasks except HalfCheetah, where performance is comparable.
• Stable training was achieved on tasks where DDPG failed, including HumanoidStandup, Pusher, Striker, and Thrower.
• Using a fixed trust region radius with a trainable covariance matrix reduced hyperparameter sensitivity and improved convergence stability.
• Replay buffer depth of 3 provided the best performance on most tasks, with notable gains on HumanoidStandup.
• The method demonstrated consistent performance across multiple random seeds, indicating robustness and reduced variance in training.