[Paper Review] PC-PG: Policy Cover Directed Exploration for Provable Policy Gradient Learning

Alekh Agarwal, Mikael Henaff+2 more|arXiv (Cornell University)|Jul 16, 2020

Reinforcement Learning in Robotics12 citations

TL;DR

PC-PG introduces a provably efficient policy gradient method that balances exploration and exploitation using an ensemble of policies—called a policy cover—ensuring polynomial sample and runtime complexity in tabular and linear MDPs. It provides strong guarantees under model misspecification, including approximation bounds under average-case error and distribution shift.

ABSTRACT

Direct policy gradient methods for reinforcement learning are a successful approach for a variety of reasons: they are model free, they directly optimize the performance metric of interest, and they allow for richly parameterized policies. Their primary drawback is that, by being local in nature, they fail to adequately explore the environment. In contrast, while model-based approaches and Q-learning directly handle exploration through the use of optimism, their ability to handle model misspecification and function approximation is far less evident. This work introduces the the Policy Cover-Policy Gradient (PC-PG) algorithm, which provably balances the exploration vs. exploitation tradeoff using an ensemble of learned policies (the policy cover). PC-PG enjoys polynomial sample complexity and run time for both tabular MDPs and, more generally, linear MDPs in an infinite dimensional RKHS. Furthermore, PC-PG also has strong guarantees under model misspecification that go beyond the standard worst case `∞ assumptions; this includes approximation guarantees for state aggregation under an average case error assumption, along with guarantees under a more general assumption where the approximation error under distribution shift is controlled. We complement the theory with empirical evaluation across a variety of domains in both reward-free and reward-driven settings.

Motivation & Objective

• To address the exploration-exploitation trade-off in model-free policy gradient methods, which are locally optimized and often fail to explore effectively.
• To develop a method that ensures provable sample efficiency and runtime complexity in both tabular and linear MDPs.
• To provide robustness guarantees under model misspecification, particularly for function approximation and state aggregation.
• To extend theoretical guarantees beyond worst-case assumptions to average-case and distribution-shift settings.

Proposed method

• PC-PG constructs a policy cover—an ensemble of learned policies—that collectively explore the state-action space more effectively than single policies.
• It uses a policy gradient update that is guided by the coverage of the policy ensemble, ensuring that unexplored or under-explored regions are actively targeted.
• The algorithm operates within a reproducing kernel Hilbert space (RKHS), enabling generalization in infinite-dimensional function spaces.
• It incorporates optimism in the face of uncertainty through the policy cover, promoting exploration in low-coverage regions.
• The method provides theoretical bounds on sample complexity and runtime, scaling polynomially in the relevant problem dimensions.
• It derives approximation guarantees under average-case error assumptions and controlled error under distribution shift.

Experimental results

Research questions

• RQ1 Can a policy gradient method achieve provable sample efficiency and runtime complexity while balancing exploration and exploitation in tabular and linear MDPs?
• RQ2 How can policy gradient methods be made robust to model misspecification, particularly under function approximation?
• RQ3 What theoretical guarantees can be provided when approximation errors are bounded under average-case or distribution-shift conditions?
• RQ4 Can a policy cover mechanism ensure that exploration is systematically directed toward unexplored regions without relying on explicit bonus functions?
• RQ5 How does the performance of PC-PG compare to standard policy gradient and Q-learning methods in both reward-free and reward-driven settings?

Key findings

• PC-PG achieves polynomial sample complexity and runtime for both tabular MDPs and linear MDPs in an infinite-dimensional RKHS.
• The algorithm provides strong guarantees under model misspecification, including approximation bounds for state aggregation under average-case error assumptions.
• It ensures robustness when the approximation error under distribution shift is bounded, extending beyond standard worst-case ∞ assumptions.
• Empirical evaluation demonstrates competitive performance across diverse environments in both reward-free and reward-driven settings.
• The policy cover mechanism effectively directs exploration, reducing reliance on heuristic bonus functions or explicit exploration strategies.
• The method maintains theoretical guarantees even when the true dynamics or value functions are not perfectly captured by the function approximator.