[论文解读] Is Q-Learning Minimax Optimal? A Tight Sample Complexity Analysis

Gen Li, Changxiao Cai+3 more|arXiv (Cornell University)|Feb 12, 2021

Reinforcement Learning in Robotics参考文献 58被引用 18

一句话总结

本文针对表格型马尔可夫决策过程在同步设置下的Q-learning，提供了紧致的样本复杂度分析，证明当|A| ≥ 2时，Q-learning在样本复杂度上为极小极大次优，其复杂度量级为O(|S||A|/(1−γ)^4ε²)，而TD学习（|A|=1）可达到极小极大最优，复杂度为O(|S|/(1−γ)^3ε²)。该工作将Q-learning的次优性归因于过估计问题，并将结果扩展至具有O(1/(1−γ)^4)有效时域依赖的异步Q-learning。

ABSTRACT

Q-learning, which seeks to learn the optimal Q-function of a Markov decision process (MDP) in a model-free fashion, lies at the heart of reinforcement learning. When it comes to the synchronous setting (such that independent samples for all state-action pairs are drawn from a generative model in each iteration), substantial progress has been made towards understanding the sample efficiency of Q-learning. Consider a $γ$-discounted infinite-horizon MDP with state space $\mathcal{S}$ and action space $\mathcal{A}$: to yield an entrywise $\varepsilon$-approximation of the optimal Q-function, state-of-the-art theory for Q-learning requires a sample size exceeding the order of $\frac{|\mathcal{S}||\mathcal{A}|}{(1-γ)^5\varepsilon^{2}}$, which fails to match existing minimax lower bounds. This gives rise to natural questions: what is the sharp sample complexity of Q-learning? Is Q-learning provably sub-optimal? This paper addresses these questions for the synchronous setting: (1) when $|\mathcal{A}|=1$ (so that Q-learning reduces to TD learning), we prove that the sample complexity of TD learning is minimax optimal and scales as $\frac{|\mathcal{S}|}{(1-γ)^3\varepsilon^2}$ (up to log factor); (2) when $|\mathcal{A}|\geq 2$, we settle the sample complexity of Q-learning to be on the order of $\frac{|\mathcal{S}||\mathcal{A}|}{(1-γ)^4\varepsilon^2}$ (up to log factor). Our theory unveils the strict sub-optimality of Q-learning when $|\mathcal{A}|\geq 2$, and rigorizes the negative impact of over-estimation in Q-learning. Finally, we extend our analysis to accommodate asynchronous Q-learning (i.e., the case with Markovian samples), sharpening the horizon dependency of its sample complexity to be $\frac{1}{(1-γ)^4}$.

研究动机与目标

• 解决在表格型MDP的同步设置下，Q-learning是否实现极小极大最优样本复杂度的开放问题。
• 通过将Q-learning的样本复杂度与极小极大下界进行比较，识别Q-learning的根本统计极限。
• 研究Q-learning中过估计的影响及其在存在多个动作时导致次优性的原因。
• 将分析扩展至使用马氏样本的异步Q-learning，优化其时域依赖性。
• 在ℓ∞-范数误差准则下，为同步与异步Q-learning建立紧致的上下界。

提出的方法

• 通过误差向量Δt的递推关系与集中不等式，推导出同步Q-learning的紧致上界。
• 引入辅助序列Q̂k(a)与Q̄k，以解耦最大值操作，并控制Q-learning中过估计的影响。
• 通过构造包含两个动作与单一状态的困难实例，推导出与上界仅相差对数因子的极小极大下界。
• 应用Freedman不等式与伯恩斯坦型集中不等式，控制经验转移概率的偏差。
• 通过将过程建模为具有混合时间约束的马氏抽样方案，将分析适配至异步Q-learning。
• 利用迭代误差传播与递归分解方法，界定了Q-learning估计的期望ℓ∞-误差。

Figure 1: The constructed hard MDP instance used in the analysis of Theorem 3 , where $p=\frac{4\gamma-1}{3\gamma}$ and the specifications are described in ( 46 ).

实验结果

研究问题

• RQ1 在|A| ≥ 2的表格型MDP中，Q-learning在样本复杂度上是否为极小极大最优？
• RQ2 在同步设置下，Q-learning的样本复杂度对有效时域1/(1−γ)的精确依赖关系为何？
• RQ3 Q-learning中的过估计如何导致其相比TD学习的次优性？
• RQ4 异步Q-learning的样本复杂度能否被进一步收紧，特别是对(1−γ)−1项的依赖？
• RQ5 Q-learning的极小极大下界是什么？其是否与上界仅相差对数因子？

主要发现

• 当|A| = 1（即TD学习）时，样本复杂度为极小极大最优，其量级为O(|S|/(1−γ)^3ε²)，与极小极大下界仅相差对数因子。
• 当|A| ≥ 2时，Q-learning的样本复杂度为O(|S||A|/(1−γ)^4ε²)，相比极小极大下界O(|S||A|/(1−γ)^3ε²)严格次优。
• Q-learning的次优性被严格归因于Q值更新中的过估计，其误差在迭代过程中累积。
• 分析表明，Q-learning中的有效时域依赖为O(1/(1−γ)^4)，劣于基于模型的方法可实现的最优O(1/(1−γ)^3)。
• 对于使用马氏样本的异步Q-learning，样本复杂度在有效时域上为O(1/(1−γ)^4)，与同步情况仅相差对数因子。
• 本文构造了一个困难的MDP实例，证明Q-learning的期望ℓ∞-误差下界为Ω(1/√(μ_min(1−γ)^4T log^3 T))，从而验证了上界紧致性。