[论文解读] On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning

Bolin Gao, Lacra Pavel|arXiv (Cornell University)|Apr 3, 2017

Mathematical and Theoretical Epidemiology and Ecology Models参考文献 38被引用 212

一句话总结

论文显示 softmax 是 log-sum-exp 函数的梯度，推导出受逆温度控制的 Lipschitz 与 co-coercivity 属性，并展示在无状态、博弈论强化学习方案中的应用。

ABSTRACT

In this paper, we utilize results from convex analysis and monotone operator theory to derive additional properties of the softmax function that have not yet been covered in the existing literature. In particular, we show that the softmax function is the monotone gradient map of the log-sum-exp function. By exploiting this connection, we show that the inverse temperature parameter determines the Lipschitz and co-coercivity properties of the softmax function. We then demonstrate the usefulness of these properties through an application in game-theoretic reinforcement learning.

研究动机与目标

• Using convex analysis and monotone operator theory to expand the mathematical understanding of the softmax function.
• Establish that softmax is the gradient of the log-sum-exp potential and study how the inverse temperature lambda affects its properties.
• Demonstrate how these properties can guarantee convergence aspects in a simple game-theoretic reinforcement learning setup.

提出的方法

• Show that the softmax is the gradient of the log-sum-exp function (Proposition 1).
• Compute the Hessian/Jacobian of the log-sum-exp function to obtain the softmax Jacobian (Proposition 2).
• Establish Lipschitz continuity of softmax with constant L = lambda (Proposition 4).
• Derive 1/L-co-coercivity of softmax via Baillon–Haddad theorem (Corollary 2).
• Discuss monotonicity properties and maximal monotonicity of softmax (Proposition 3 and Corollary 1).
• Apply these properties to a stateless continuous-time reinforcement learning scheme (EXP-D-RL) in a single-player game to illustrate convergence insights (Section VI).

实验结果

研究问题

• RQ1 What additional properties of the softmax function can be derived from convex analysis and monotone operator theory?
• RQ2 How does the inverse temperature lambda influence Lipschitz and co-coercivity properties of softmax?
• RQ3 Can the derived properties ensure convergence of learning dynamics in game-theoretic reinforcement learning?
• RQ4 How is softmax related to the log-sum-exp potential and its duality with negative entropy in this context?
• RQ5 What is the role of softmax in replicator-type dynamics and evolutionary game theory connections?

主要发现

• Softmax is the gradient of the log-sum-exp function (softmax = gradient of log-sum-exp).
• The Jacobian of softmax is lambda times (diag(sigma(z)) − sigma(z)sigma(z)^T).
• Softmax is lambda-Lipschitz and 1/lambda-co-coercive with respect to the Euclidean norm.
• Baillon–Haddad theorem implies 1/lambda-co-coercivity of softmax via Lipschitz gradient of the log-sum-exp.
• Softmax is monotone and maximal monotone (not strictly monotone) on R^n.
• These properties can be used to analyze convergence in a stateless, continuous-time reinforcement learning scheme (EXP-D-RL).