[Paper Review] Optimizing over a Restricted Policy Class in Markov Decision Processes

Ershad Banijamali, Yasin Abbasi-Yadkori+2 more|arXiv (Cornell University)|Feb 26, 2018

Reinforcement Learning in Robotics2 references3 citations

TL;DR

This paper proposes an efficient algorithm for optimizing over a restricted policy class defined as the convex hull of known base policies in Markov Decision Processes (MDPs). By reformulating the problem in the dual space of occupancy measures and using a stochastic subgradient method, the approach achieves near-optimal performance with linear dependence on states and polynomial time in the number of base policies—offering significant computational advantages over policy gradient methods, especially when policy evaluation is costly.

ABSTRACT

We address the problem of finding an optimal policy in a Markov decision process under a restricted policy class defined by the convex hull of a set of base policies. This problem is of great interest in applications in which a number of reasonably good (or safe) policies are already known and we are only interested in optimizing in their convex hull. We show that this problem is NP-hard to solve exactly as well as to approximate to arbitrary accuracy. However, under a condition that is akin to the occupancy measures of the base policies having large overlap, we show that there exists an efficient algorithm that finds a policy that is almost as good as the best convex combination of the base policies. The running time of the proposed algorithm is linear in the number of states and polynomial in the number of base policies. In practice, we demonstrate an efficient implementation for large state problems. Compared to traditional policy gradient methods, the proposed approach has the advantage that, apart from the computation of occupancy measures of some base policies, the iterative method need not interact with the environment during the optimization process. This is especially important in complex systems where estimating the value of a policy can be a time consuming process.

Motivation & Objective

• Address the challenge of finding an optimal policy within a restricted class defined as the convex hull of a set of known, reasonably good or safe base policies.
• Overcome the computational infeasibility of exact or approximate optimization in this class, which is proven to be NP-hard.
• Develop a practical and efficient algorithm that achieves near-optimal performance when the occupancy measures of the base policies exhibit sufficient overlap.
• Reduce reliance on environment interaction during optimization, enabling faster convergence in complex systems where policy evaluation is expensive.
• Provide a method that avoids the need for backward simulators and importance weighting, simplifying implementation and tightening theoretical bounds.

Proposed method

• Reformulate the policy optimization problem in the dual space of occupancy measures, where the search space consists of linear combinations of the occupancy measures of the base policies.
• Use a convex relaxation technique inspired by Abbasi-Yadkori et al. (2014) to handle constraint violations and enable efficient optimization.
• Apply a stochastic subgradient method to minimize the dual objective, which is non-convex but amenable to approximation via the relaxation.
• Normalize linear combinations of occupancy measures to construct valid policies, avoiding the need for forward simulation during optimization.
• Compute the stationary occupancy measures of base policies once offline, then perform all optimization in the dual space without further environment interaction.
• Leverage the overlap in occupancy measures to ensure that non-trivial, high-performing policies can be generated in the dual space, even when the primal space is NP-hard.

Experimental results

Research questions

• RQ1 Can we efficiently optimize over a restricted policy class defined as the convex hull of a finite set of base policies in MDPs?
• RQ2 Is it possible to achieve near-optimal performance in this restricted class with polynomial-time complexity in the number of base policies and linear dependence on the number of states?
• RQ3 How does the overlap in occupancy measures of base policies affect the performance and feasibility of dual-space optimization?
• RQ4 Can we eliminate the need for repeated environment interaction during optimization, especially when policy evaluation is expensive?
• RQ5 What are the theoretical and empirical advantages of using the dual space of occupancy measures over traditional policy gradient methods in this setting?

Key findings

• The problem of optimizing over the convex hull of base policies in MDPs is NP-hard to solve exactly and to approximate to arbitrary accuracy.
• When the occupancy measures of the base policies have significant overlap, the proposed dual-space algorithm finds a policy nearly as good as the best convex combination of base policies.
• The algorithm runs in time linear in the number of states and polynomial in the number of base policies, offering a strong computational advantage over policy gradient methods.
• In the 8-queue problem, the dual-space method achieved a cost of 20.59 with optimal dual weights θ = [1.1246, -0.1143, -0.0102], outperforming the base policies and approaching the policy gradient result of 19.14 with far less computational cost.
• The method avoids the need for importance weighting and backward simulators, simplifying implementation and tightening error bounds compared to prior dual-space approaches.
• Empirical results show that the dual-space stochastic subgradient method converges faster and with lower variance than policy gradient in environments with expensive policy evaluation.