[Paper Review] Exponential Lower Bounds for Planning in MDPs With Linearly-Realizable Optimal Action-Value Functions

Gellért Weisz, Philip Amortila+1 more|arXiv (Cornell University)|Oct 3, 2020

Machine Learning and Algorithms16 references20 citations

TL;DR

This paper establishes exponential query complexity lower bounds for planning in fixed-horizon and discounted Markov Decision Processes (MDPs) with linearly-realizable optimal action-value functions. Despite the assumption that the optimal Q-function lies in the span of a d-dimensional feature map, it proves that any sound planner must make at least min(e^Ω(d), Ω(2^H)) queries in the fixed-horizon case and e^Ω(d) queries in the discounted case, showing that realizability alone is insufficient for query-efficient planning.

ABSTRACT

We consider the problem of local planning in fixed-horizon and discounted Markov Decision Processes (MDPs) with linear function approximation and a generative model under the assumption that the optimal action-value function lies in the span of a feature map that is available to the planner. Previous work has left open the question of whether there exist sound planners that need only poly(H,d) queries regardless of the MDP, where H is the horizon and d is the dimensionality of the features. We answer this question in the negative: we show that any sound planner must query at least $\min(\exp(Ω(d)), Ω(2^H))$ samples in the fized-horizon setting and $\exp(Ω(d))$ samples in the discounted setting. We also show that for any $δ>0$, the least-squares value iteration algorithm with $O(H^5d^{H+1}/δ^2)$ queries can compute a $δ$-optimal policy in the fixed-horizon setting. We discuss implications and remaining open questions.

Motivation & Objective

• To resolve whether realizability of the optimal action-value function enables query-efficient planning in MDPs with linear function approximation.
• To determine the worst-case query complexity of sound planners under the assumption that the optimal Q-function is linearly realizable via a known feature map.
• To establish tight lower bounds on the number of simulator queries required for planning in both fixed-horizon and discounted MDPs.
• To contrast the query complexity of planning under q*-realizability with other realizability assumptions, particularly in settings with large action spaces.

Proposed method

• Constructs a deterministic MDP family with exponentially many actions using Johnson-Lindenstrauss lemma to generate nearly orthogonal feature vectors.
• Designs a reward structure where the optimal action differs from others by an exponentially small gap at the final time step, which is amplified backward through the horizon.
• Uses backwards induction and concentration inequalities (Hoeffding's inequality) to bound estimation error in value function approximation under a generative model.
• Applies a least-squares value iteration algorithm with sample complexity Õ(H^5 d^{H+1} / δ^2) to achieve δ-optimal policies in the fixed-horizon setting.
• Employs a recursive error propagation argument to bound the difference between the estimated and true optimal value functions at each stage.
• Proves lower bounds via a reduction that shows any planner must query exponentially many times in d or H to distinguish between nearly identical value functions.

Experimental results

Research questions

• RQ1 Is realizability of the optimal action-value function sufficient to guarantee query-efficient planning in MDPs with linear function approximation?
• RQ2 What is the worst-case query complexity of sound planners in fixed-horizon MDPs when the optimal Q-function is linearly realizable?
• RQ3 How does the query complexity scale with the feature dimension d and planning horizon H under q*-realizability?
• RQ4 Does the exponential query complexity persist in the discounted MDP setting under the same realizability assumption?
• RQ5 Can the gap between known upper and lower bounds for query complexity in this setting be closed?

Key findings

• Any sound planner must make at least min(e^Ω(d), Ω(2^H)) queries in the fixed-horizon setting, showing that query complexity can be exponential in d or H.
• In the discounted MDP setting, the query complexity is at least e^Ω(d), indicating exponential dependence on the feature dimension.
• The least-squares value iteration algorithm with Õ(H^5 d^{H+1} / δ^2) queries computes a δ-optimal policy in the fixed-horizon setting.
• The lower bound construction is deterministic except for random rewards at the final time step, highlighting a stark contrast with deterministic MDPs where only d suboptimal actions may be chosen.
• The exponential lower bound arises not from the size of the action space alone, but from the structure of value function approximation under realizability, even when the action set is large.
• The results show a strong separation between q*-realizability and other realizability assumptions, such as realizability of q^π for all memoryless policies, especially for H > 1.