[Paper Review] Reinforcement Learning with Dynamic Boltzmann Softmax Updates

Ling Pan, Qingpeng Cai+4 more|arXiv (Cornell University)|Mar 14, 2019

Reinforcement Learning in Robotics20 references4 citations

TL;DR

This paper proposes the Dynamic Boltzmann Softmax (DBS) operator to resolve convergence issues in value function estimation by making the inverse temperature parameter β time-varying and state-independent. DBS ensures convergence in value iteration and Q-learning, and DBS-DQN outperforms DQN in 40 out of 49 Atari games, demonstrating improved stability and performance.

ABSTRACT

Value function estimation is an important task in reinforcement learning, i.e., prediction. The Boltzmann softmax operator is a natural value estimator and can provide several benefits. However, it does not satisfy the non-expansion property, and its direct use may fail to converge even in value iteration. In this paper, we propose to update the value function with dynamic Boltzmann softmax (DBS) operator, which has good convergence property in the setting of planning and learning. Experimental results on GridWorld show that the DBS operator enables better estimation of the value function, which rectifies the convergence issue of the softmax operator. Finally, we propose the DBS-DQN algorithm by applying dynamic Boltzmann softmax updates in deep Q-network, which outperforms DQN substantially in 40 out of 49 Atari games.

Motivation & Objective

• Address the lack of convergence in standard Boltzmann softmax due to its failure to satisfy the non-expansion property.
• Develop a value function update mechanism that maintains exploration-exploitation balance while ensuring convergence.
• Design a method applicable to both tabular and deep RL settings, particularly for deep Q-networks.
• Achieve better performance than DQN and soft Q-learning by reducing overestimation and gradient noise.
• Ensure the method requires no hyperparameter tuning and supports double Q-learning

Proposed method

• Propose the Dynamic Boltzmann Softmax (DBS) operator with a time-varying β parameter that increases over time, ensuring convergence.
• Prove that βt → ∞ is sufficient for convergence in value iteration, rectifying the non-convergence issue of fixed-β softmax.
• Establish a tighter error bound for the fixed-parameter softmax operator compared to prior work.
• Extend DBS to Q-learning, proving convergence of the DBS Q-learning algorithm.
• Integrate DBS into deep Q-networks via DBS-DQN, using function approximation in high-dimensional state spaces.
• Apply the DBS operator in a way that preserves Bellman optimality, enables overestimation reduction, and directly represents a policy

Experimental results

Research questions

• RQ1 Can a time-varying Boltzmann softmax operator ensure convergence in value function estimation where fixed-β softmax fails?
• RQ2 Does the DBS operator improve value function estimation accuracy and stability in tabular environments like GridWorld?
• RQ3 Can DBS-DQN achieve superior performance compared to DQN across a diverse set of Atari 2600 games?
• RQ4 Does the DBS operator maintain desirable properties such as overestimation reduction and compatibility with double Q-learning?
• RQ5 Is the DBS method robust and effective without requiring hyperparameter tuning?

Key findings

• DBS Q-learning with β = t² achieved the best performance in GridWorld, outperforming both Q-learning and soft Q-learning.
• The DBS operator ensures convergence in value iteration and Q-learning by satisfying the non-expansion property through time-varying β.
• DBS-DQN outperformed DQN in 40 out of 49 Atari games, with significant improvements in games like Atlantis, Road Runner, and Video Pinball.
• In the Atari suite, DBS-DQN achieved a relative human normalized score improvement of up to 3,768,100.0 in Atlantis, compared to DQN.
• The DBS operator achieved a tighter error bound than prior work on fixed-parameter softmax, improving theoretical guarantees.
• DBS-DQN maintained strong performance without requiring additional hyperparameter tuning, unlike other soft-max-based methods