[Paper Review] Provably Efficient Maximum Entropy Exploration

Elad Hazan, Sham M. Kakade+2 more|arXiv (Cornell University)|Dec 6, 2018

Image and Signal Denoising MethodsComputer Science94 citations

TL;DR

The paper provides a provably efficient algorithm to optimize intrinsic state- visitation objectives, such as maximum entropy over state distributions, in unknown MDPs using a Frank-Wolfe style approach with planning and density oracles.

ABSTRACT

Suppose an agent is in a (possibly unknown) Markov Decision Process in the absence of a reward signal, what might we hope that an agent can efficiently learn to do? This work studies a broad class of objectives that are defined solely as functions of the state-visitation frequencies that are induced by how the agent behaves. For example, one natural, intrinsically defined, objective problem is for the agent to learn a policy which induces a distribution over state space that is as uniform as possible, which can be measured in an entropic sense. We provide an efficient algorithm to optimize such such intrinsically defined objectives, when given access to a black box planning oracle (which is robust to function approximation). Furthermore, when restricted to the tabular setting where we have sample based access to the MDP, our proposed algorithm is provably efficient, both in terms of its sample and computational complexities. Key to our algorithmic methodology is utilizing the conditional gradient method (a.k.a. the Frank-Wolfe algorithm) which utilizes an approximate MDP solver.

Motivation & Objective

• Motivate exploration in unknown MDPs when rewards are unavailable or sparse.
• Define and optimize intrinsic objectives that depend on state-visitation distributions (e.g., entropy).
• Show how to achieve provable efficiency with an approximate planning oracle and a state-distribution oracle.
• Provide tabular and unknown-MDP results with sample and computational guarantees.

Proposed method

• Formulate exploration as maximizing a concave functional R(dπ) over the induced state distribution dπ.
• Represent the search space as a convex set K of feasible state distributions and reduce optimization to this space.
• Use a Frank-Wolfe (conditional gradient) style algorithm that iteratively adds policies to a policy mixture and updates weights.
• At each iteration, construct a reward r_t via the gradient of R at the estimated distribution, then use an ApproxPlan oracle to obtain a near-optimal policy for r_t.
• Estimate the current state distribution with a DensityEst oracle, accounting for approximation errors.
• Provide guarantees that the number of oracle calls scales as O(1/ε log 1/ε), independent of the state space size, under smoothness assumptions on R.

Experimental results

Research questions

• RQ1 Can we efficiently optimize intrinsic objectives defined on state visitation distributions in unknown MDPs?
• RQ2 Does a Frank-Wolfe style method with planning and density oracles yield polynomial-time guarantees for entropy-based objectives?
• RQ3 How many calls to planning and density estimation oracles are needed to achieve ε-suboptimality for a given R?
• RQ4 What are the sample and computational complexities in the tabular vs unknown MDP settings for max-entropy exploration?

Key findings

• An efficient algorithm (Algorithm 1) achieves R(d_{π_mix_T}) within ε of the optimum after O(1/ε log 1/ε) calls to ApproxPlan & DensityEst.
• Maximizing entropy is framed as maximizing a concave functional over the induced state distribution with a convex reformulation in distribution space.
• Stationary policies suffice for the optimization over distributions (via π′(a|s) = dπ(s,a)/dπ(s)).
• In the tabular known-MDP setting, the method runs in polynomial time with standard planning methods; in the unknown-MDP setting, a sample-based construction (Algorithms 2 and 3) yields polynomial-time guarantees with specified episode complexity.
• The paper provides a smoothed entropy proxy H_σ to enable smooth optimization and relates its optimization guarantees back to the true entropy.