[Paper Review] Provably Efficient Q-Learning with Low Switching Cost

Yu Bai, Tengyang Xie+2 more|arXiv (Cornell University)|Jan 1, 2019

Advanced Bandit Algorithms Research22 citations

TL;DR

This paper introduces Q-Learning with UCB2 exploration, a model-free reinforcement learning algorithm for H-step episodic MDPs that achieves sublinear regret with a local switching cost of $O(H^3SA\log K)$ over $K$ episodes. It establishes a matching lower bound of $\Omega(HSA)$, proving near-optimality in adaptivity, and extends naturally to concurrent settings with improved performance.

ABSTRACT

We take initial steps in studying PAC-MDP algorithms with limited adaptivity, that is, algorithms that change its exploration policy as infrequently as possible during regret minimization. This is motivated by the difficulty of running fully adaptive algorithms in real-world applications (such as medical domains), and we propose to quantify adaptivity using the notion of \emph{local switching cost}. Our main contribution, Q-Learning with UCB2 exploration, is a model-free algorithm for $H$-step episodic MDP that achieves sublinear regret whose local switching cost in $K$ episodes is $O(H^3SA\log K)$, and we provide a lower bound of $\Omega(HSA)$ on the local switching cost for any no-regret algorithm. Our algorithm can be naturally adapted to the concurrent setting \citep{guo2015concurrent}, which yields nontrivial results that improve upon prior work in certain aspects.

Motivation & Objective

• To study PAC-MDP algorithms with limited adaptivity, particularly in real-world applications where frequent policy updates are impractical.
• To quantify adaptivity using the notion of local switching cost, defined as the number of times the exploration policy is changed during learning.
• To design a model-free algorithm that achieves sublinear regret while minimizing policy switching frequency.
• To establish theoretical lower bounds on switching cost for any no-regret algorithm in episodic MDPs.
• To extend the proposed algorithm to the concurrent reinforcement learning setting, improving upon prior results.

Proposed method

• Proposes Q-Learning with UCB2 exploration, combining Q-learning updates with UCB2-based exploration to balance exploration and exploitation.
• Uses an exploration bonus based on uncertainty estimates, derived from UCB2, to guide policy selection with low switching frequency.
• Analyzes the local switching cost over $K$ episodes, showing it scales as $O(H^3SA\log K)$, where $H$ is horizon, $S$ states, $A$ actions.
• Applies a novel analysis technique to bound the number of policy switches by controlling the frequency of value function updates.
• Adapts the algorithm to the concurrent setting by allowing multiple agents to learn in parallel with shared experience, reducing effective switching cost.
• Employs a regret analysis framework to ensure sublinear regret while maintaining low adaptivity.

Experimental results

Research questions

• RQ1 What is the minimal possible local switching cost for any no-regret PAC-MDP algorithm in episodic MDPs?
• RQ2 Can a model-free Q-learning algorithm achieve sublinear regret with low switching cost in episodic MDPs?
• RQ3 How does the proposed algorithm perform in the concurrent reinforcement learning setting compared to prior work?
• RQ4 Is the proposed switching cost bound tight, or can it be improved?
• RQ5 Can the algorithm be naturally extended to concurrent settings while preserving theoretical guarantees?

Key findings

• The proposed Q-Learning with UCB2 exploration achieves sublinear regret in $H$-step episodic MDPs with a local switching cost of $O(H^3SA\log K)$.
• A lower bound of $\Omega(HSA)$ on the local switching cost is established for any no-regret algorithm, showing the proposed bound is nearly optimal.
• The algorithm can be naturally adapted to the concurrent setting, yielding nontrivial improvements over prior work in certain regimes.
• The switching cost bound scales logarithmically with the number of episodes $K$, indicating efficient adaptation over time.
• The theoretical analysis confirms that low adaptivity does not compromise regret performance, enabling practical deployment in sensitive domains.
• The results demonstrate that minimizing policy switches is compatible with achieving strong learning performance in model-free RL.