[Paper Review] Single-partition adaptive Q-learning

João Pedro Araújo, Mário A. T. Figueiredo+1 more|arXiv (Cornell University)|Jul 14, 2020

Reinforcement Learning in Robotics31 references4 citations

TL;DR

This paper proposes Single-partition adaptive Q-learning (SPAQL), an improved model-free reinforcement learning algorithm that adaptively partitions the state-action space using a single partitioning structure while learning a time-invariant policy. By combining upper confidence bounds (UCB) and Boltzmann exploration with an automatically tuned temperature, SPAQL achieves faster convergence and higher sample efficiency than adaptive Q-learning (AQL), requiring fewer arms and training iterations, especially in long-horizon tasks.

ABSTRACT

This paper introduces single-partition adaptive Q-learning (SPAQL), an algorithm for model-free episodic reinforcement learning (RL), which adaptively partitions the state-action space of a Markov decision process (MDP), while simultaneously learning a time-invariant policy (i. e., the mapping from states to actions does not depend explicitly on the episode time step) for maximizing the cumulative reward. The trade-off between exploration and exploitation is handled by using a mixture of upper confidence bounds (UCB) and Boltzmann exploration during training, with a temperature parameter that is automatically tuned as training progresses. The algorithm is an improvement over adaptive Q-learning (AQL). It converges faster to the optimal solution, while also using fewer arms. Tests on episodes with a large number of time steps show that SPAQL has no problems scaling, unlike AQL. Based on this empirical evidence, we claim that SPAQL may have a higher sample efficiency than AQL, thus being a relevant contribution to the field of efficient model-free RL methods.

Motivation & Objective

• To address the sample inefficiency and high memory cost of model-free RL in continuous state-action spaces.
• To improve upon adaptive Q-learning (AQL) by enabling faster convergence and reduced partition complexity.
• To balance exploration and exploitation using a hybrid UCB and Boltzmann exploration strategy with dynamic temperature tuning.
• To develop a time-invariant policy learning method that scales efficiently to long-horizon episodic tasks.
• To reduce dependency on manual hyperparameter tuning, particularly for the bonus scaling parameter.

Proposed method

• SPAQL uses a single adaptive partitioning structure that starts with one ball covering the entire state-action space and splits only when necessary.
• It applies a mixture of upper confidence bounds (UCB) and Boltzmann exploration to balance exploration and exploitation during training.
• The temperature parameter in Boltzmann exploration is dynamically adjusted via a cyclic schedule to improve learning efficiency.
• The algorithm automatically tunes the bonus scaling parameter ξ based on episode length H and confidence level δ, though experiments suggest lower values are preferable.
• Partition updates are performed using rollouts with a fixed number of evaluation episodes N, and the radius of each ball is adjusted based on Q-value variance and confidence bounds.
• The method avoids per-time-step partitioning by learning a time-invariant policy, reducing memory and computational overhead.

Experimental results

Research questions

• RQ1 Can a single-partition adaptive Q-learning algorithm achieve faster convergence and better sample efficiency than AQL in episodic, continuous MDPs?
• RQ2 How does combining UCB and Boltzmann exploration with adaptive temperature tuning affect learning performance and hyperparameter sensitivity?
• RQ3 To what extent does SPAQL scale to long-horizon tasks with many time steps, compared to AQL?
• RQ4 Can the bonus scaling parameter ξ be effectively tuned or eliminated without sacrificing convergence guarantees?
• RQ5 Does the use of a time-invariant policy reduce partition complexity while maintaining or improving performance?

Key findings

• SPAQL converges faster than AQL and requires significantly fewer arms (i.e., partition cells) to achieve comparable or better performance.
• In the oil discovery problem with a Laplace survey function, SPAQL outperforms AQL in cumulative reward, despite AQL having a higher bonus scaling parameter.
• SPAQL shows lower sensitivity to the bonus scaling parameter ξ than AQL, suggesting greater robustness and potential for automatic tuning.
• The algorithm maintains stable performance across long episodes (up to H=5), demonstrating scalability where AQL fails to scale.
• Empirical results indicate that setting ξ to a non-zero value below H/3 yields optimal performance, with values around 80 being excessive and counterproductive.
• The theoretical expression for ξ often yields values larger than H, which contradicts empirical findings and suggests a need for reformulating the bonus term to be bounded by H.