[Paper Review] A New Softmax Operator for Reinforcement Learning.

Kavosh Asadi, Michael L. Littman|arXiv (Cornell University)|Dec 16, 2016

Reinforcement Learning in Robotics20 references7 citations

TL;DR

This paper proposes a novel softmax operator for reinforcement learning that ensures non-expansion and differentiability, addressing the instability and suboptimal behavior of the standard Boltzmann softmax. The new operator enables stable learning and gradient-based optimization, with empirical results showing improved convergence and performance over existing methods.

ABSTRACT

A softmax operator applied to a set of values acts somewhat like the maximization function and somewhat like an average. In sequential decision making, softmax is often used in settings where it is necessary to maximize utility but also to hedge against problems that arise from putting all of one's weight behind a single maximum utility decision. The Boltzmann softmax operator is the most commonly used softmax operator in this setting, but we show that this operator is prone to misbehavior. In this work, we study an alternative softmax operator that, among other properties, is both a non-expansion (ensuring convergent behavior in learning and planning) and differentiable (making it possible to improve decisions via gradient descent methods). We provide proofs of these properties and present empirical comparisons between various softmax operators.

Motivation & Objective

• To address the instability and non-convergent behavior of the standard Boltzmann softmax in reinforcement learning.
• To design a softmax operator that is both non-expanding and differentiable for stable learning and gradient-based optimization.
• To provide theoretical guarantees on convergence and performance for the new operator.
• To empirically compare the new operator with existing softmax variants in sequential decision-making tasks.

Proposed method

• Propose a new softmax operator that replaces the Boltzmann distribution with a modified form ensuring non-expansion under the L-infinity norm.
• Formalize the operator using a temperature-adjusted transformation that maintains monotonicity and smoothness.
• Prove that the new operator is a non-expansion, guaranteeing convergence in value iteration and policy learning.
• Demonstrate that the operator is differentiable, enabling end-to-end gradient-based policy optimization.
• Implement and evaluate the operator in standard RL benchmarks against Boltzmann and other softmax variants.
• Use theoretical analysis and empirical comparisons to validate stability and performance.

Experimental results

Research questions

• RQ1 Does the proposed softmax operator ensure non-expansive behavior in value and policy iteration?
• RQ2 Can the new operator be made differentiable to support gradient-based learning?
• RQ3 How does the new operator compare to the Boltzmann softmax in terms of convergence speed and stability?
• RQ4 Does the operator maintain utility maximization while providing sufficient exploration?
• RQ5 What are the empirical performance gains in standard reinforcement learning environments?

Key findings

• The proposed softmax operator is provably non-expanding, ensuring stable convergence in value iteration and policy learning.
• The operator is differentiable, enabling effective gradient-based optimization of policies.
• Empirical results show faster convergence and improved stability compared to the Boltzmann softmax in tested environments.
• The new operator maintains a balance between exploitation and exploration without sacrificing convergence guarantees.
• Theoretical analysis confirms that the operator does not amplify differences in value estimates, preventing divergence.
• The method outperforms the Boltzmann softmax in both learning stability and final policy performance in benchmark tasks.