[Paper Review] Off-Policy Shaping Ensembles in Reinforcement Learning

Anna Harutyunyan, Tim Brys+2 more|arXiv (Cornell University)|May 21, 2014

Reinforcement Learning in Robotics24 references4 citations

TL;DR

This paper proposes Off-Policy Shaping Ensembles, a real-time, convergent reinforcement learning method that trains multiple value functions using potential-based reward shaping in parallel via the Horde framework. By combining policies through rank voting, the ensemble outperforms individual shapings in the Mountain Car domain, achieving faster learning and competitive final performance without additional computational cost.

ABSTRACT

Recent advances of gradient temporal-difference methods allow to learn off-policy multiple value functions in parallel with- out sacrificing convergence guarantees or computational efficiency. This opens up new possibilities for sound ensemble techniques in reinforcement learning. In this work we propose learning an ensemble of policies related through potential-based shaping rewards. The ensemble induces a combination policy by using a voting mechanism on its components. Learning happens in real time, and we empirically show the combination policy to outperform the individual policies of the ensemble.

Motivation & Objective

• To develop a real-time, convergent policy ensemble method for off-policy reinforcement learning that improves learning speed without extra sample or computational cost.
• To explore the use of multiple potential-based shaping signals as a means to accelerate knowledge propagation in latent learning settings.
• To design a voting mechanism that combines multiple shaped policies into a single, more effective policy in real time.
• To validate the approach in a challenging off-policy, fixed-behavior setup where traditional methods often fail to converge.
• To demonstrate that ensemble combination can match or exceed the performance of the best individual shaping signal.

Proposed method

• Leverages the Horde architecture to learn multiple value functions in parallel under off-policy, fixed-behavior settings using gradient temporal-difference methods.
• Applies potential-based reward shaping to generate multiple auxiliary reward signals, each inducing a distinct value function and policy.
• Uses a rank-based voting mechanism to combine the policies of the ensemble at each state, selecting actions based on cumulative rank across all shaped policies.
• Employs Greedy-GQ and similar gradient TD methods to ensure convergence and computational efficiency in off-policy learning.
• Maintains a shared experience stream across all value functions, enabling real-time, online learning with no additional sampling.
• The ensemble is trained in a latent learning setup where behavior policy remains fixed, ensuring theoretical convergence guarantees.

Experimental results

Research questions

• RQ1 Can multiple potential-based shaping signals be effectively combined in a real-time, off-policy reinforcement learning setting to accelerate learning?
• RQ2 Does a voting-based ensemble of shaped policies outperform individual shaped policies in terms of learning speed and final performance?
• RQ3 Can such an ensemble maintain convergence guarantees under off-policy learning with function approximation?
• RQ4 How does the ensemble perform when one shaping signal is clearly superior compared to others?
• RQ5 Can the ensemble achieve performance comparable to the best individual shaping without prior knowledge of which shaping is optimal?

Key findings

• In the scenario with one clearly superior shaping, the ensemble combination matched the performance of the best individual shaping, with a final cumulative return of -180.2 ± 1.5, statistically indistinguishable from the best single shaping.
• In the scenario with no clearly superior shaping, the ensemble outperformed all individual shapings at every stage, achieving a final cumulative return of -180.2 ± 1.5, significantly better than the worst individual shaping.
• The combination policy achieved faster learning speed than any single shaping, with the cumulative return reaching -211.2 ± 94.2 after 1000 episodes, outperforming all individual shapings at that stage.
• The ensemble consistently matched or exceeded the performance of the best individual shaping across all evaluation points, validating its robustness and effectiveness.
• Even in the absence of a dominant shaping signal, the voting mechanism successfully identified and leveraged complementary strengths across multiple shapings.
• The method demonstrated that real-time, convergent ensemble learning is feasible in off-policy settings using the Horde framework, enabling faster knowledge propagation without additional computational cost.