[Paper Review] Mixed Policy Gradient.

Yang Guan, Jingliang Duan+4 more|arXiv (Cornell University)|Feb 23, 2021

Reinforcement Learning in Robotics24 references4 citations

TL;DR

This paper proposes Mixed Policy Gradient (MPG), a novel reinforcement learning method that combines data-driven and model-driven policy gradients to accelerate convergence without sacrificing asymptotic performance. By adaptively weighting the two PG types—increasing data-driven PG over time—MPG achieves faster learning and superior performance compared to baselines.

ABSTRACT

Reinforcement learning (RL) has great potential in sequential decision-making. At present, the mainstream RL algorithms are data-driven, relying on millions of iterations and a large number of empirical data to learn a policy. Although data-driven RL may have excellent asymptotic performance, it usually yields slow convergence speed. As a comparison, model-driven RL employs a differentiable transition model to improve convergence speed, in which the policy gradient (PG) is calculated by using the backpropagation through time (BPTT) technique. However, such methods suffer from numerical instability, model error sensitivity and low computing efficiency, which may lead to poor policies. In this paper, a mixed policy gradient (MPG) method is proposed, which uses both empirical data and the transition model to construct the PG, so as to accelerate the convergence speed without losing the optimality guarantee. MPG contains two types of PG: 1) data-driven PG, which is obtained by directly calculating the derivative of the learned Q-value function with respect to actions, and 2) model-driven PG, which is calculated using BPTT based on the model-predictive return. We unify them by revealing the correlation between the upper bound of the unified PG error and the predictive horizon, where the data-driven PG is regraded as 0-step model-predictive return. Relying on that, MPG employs a rule-based method to adaptively adjust the weights of data-driven and model-driven PGs. In particular, to get a more accurate PG, the weight of the data-driven PG is designed to grow along the learning process while the other to decrease. Besides, an asynchronous learning framework is proposed to reduce the wall-clock time needed for each update iteration. Simulation results show that the MPG method achieves the best asymptotic performance and convergence speed compared with other baseline algorithms.

Motivation & Objective

• Address the slow convergence of data-driven reinforcement learning, which requires millions of iterations to converge.
• Overcome the numerical instability, model error sensitivity, and low efficiency of model-driven RL methods based on backpropagation through time (BPTT).
• Unify data-driven and model-driven policy gradients into a single framework that maintains optimality guarantees while improving training speed.
• Develop an adaptive weighting mechanism that increases reliance on data-driven PG over time to improve gradient accuracy.
• Design an asynchronous learning framework to reduce wall-clock time per update iteration.

Proposed method

• Construct two types of policy gradients: data-driven PG via direct derivative of the learned Q-value function w.r.t. actions, and model-driven PG via BPTT on model-predictive return.
• Unify the two PGs by establishing a theoretical link between the error upper bound of the combined gradient and the predictive horizon.
• Treat data-driven PG as a 0-step model-predictive return to enable formal integration with model-driven PG.
• Implement a rule-based adaptive weighting strategy that increases the weight of data-driven PG and decreases that of model-driven PG during training.
• Introduce an asynchronous learning framework to reduce computational latency and improve training efficiency.

Experimental results

Research questions

• RQ1 How can data-driven and model-driven policy gradients be effectively unified to improve convergence speed in reinforcement learning?
• RQ2 What is the theoretical relationship between the error bound of the unified policy gradient and the predictive horizon?
• RQ3 Can an adaptive weighting scheme that evolves during training enhance gradient accuracy and learning efficiency?
• RQ4 How does the proposed MPG method compare to existing RL algorithms in terms of convergence speed and asymptotic performance?
• RQ5 To what extent can an asynchronous learning framework reduce wall-clock time without degrading policy quality?

Key findings

• MPG achieves the best asymptotic performance among all evaluated baseline algorithms in simulation.
• The proposed method significantly accelerates convergence speed compared to both pure data-driven and model-driven RL approaches.
• The adaptive weighting mechanism successfully increases the contribution of data-driven PG over time, improving gradient accuracy.
• The asynchronous learning framework reduces wall-clock time per update iteration, enhancing training efficiency.
• The theoretical analysis confirms that the error bound of the unified policy gradient decreases with longer predictive horizons, supporting the method's design.