[论文解读] The Mechanics of n-Player Differentiable Games

David Balduzzi, Sébastien Racanière+4 more|arXiv (Cornell University)|Feb 15, 2018

Artificial Intelligence in Games参考文献 32被引用 81

一句话总结

本文将二阶博弈动力学分解为对称分量（势）和反对称分量（哈密顿量），并引入对称辛梯度调整（Symplectic Gradient Adjustment, SGA）以在一般可微博弈中找到稳定固定点，在GANs中的实验显示稳定性提升。

ABSTRACT

The cornerstone underpinning deep learning is the guarantee that gradient descent on an objective converges to local minima. Unfortunately, this guarantee fails in settings, such as generative adversarial nets, where there are multiple interacting losses. The behavior of gradient-based methods in games is not well understood -- and is becoming increasingly important as adversarial and multi-objective architectures proliferate. In this paper, we develop new techniques to understand and control the dynamics in general games. The key result is to decompose the second-order dynamics into two components. The first is related to potential games, which reduce to gradient descent on an implicit function; the second relates to Hamiltonian games, a new class of games that obey a conservation law, akin to conservation laws in classical mechanical systems. The decomposition motivates Symplectic Gradient Adjustment (SGA), a new algorithm for finding stable fixed points in general games. Basic experiments show SGA is competitive with recently proposed algorithms for finding stable fixed points in GANs -- whilst at the same time being applicable to -- and having guarantees in -- much more general games.

研究动机与目标

• Motivate the study of gradient dynamics in multi-objective and adversarial architectures where multiple losses interact.
• Introduce a decomposition of the Hessian of game dynamics into symmetric and antisymmetric components.
• Identify two solvable classes of games—potential and Hamiltonian—and develop a general method (SGA) to find stable fixed points.
• Demonstrate the effectiveness of SGA through experiments in GANs and discuss theoretical guarantees.

提出的方法

• Propose generalized Helmholtz decomposition of the game Hessian into symmetric S and antisymmetric A components (H = S + A).
• Define potential games (A = 0) and Hamiltonian games (S = 0) and relate them to gradient dynamics and conserved quantities.
• Introduce Symplectic Gradient Adjustment (SGA): update ξλ = ξ + λ · A⊤ξ to promote convergence to stable fixed points.
• Provide desiderata D1–D5 to ensure the adjustment preserves compatibility with potential and Hamiltonian dynamics and accelerates convergence to stable equilibria.
• Derive sign selection rules for λ to align updates toward stable fixed points and away from unstable ones.
• Show how to compute the adjustment with Hessian-vector products in practice (Appendix C).

实验结果

研究问题

• RQ1 Can the second-order dynamics of general n-player differentiable games be decomposed into tractable components?
• RQ2 When the antisymmetric vs. symmetric parts dominate, can we guarantee convergence to stable fixed points?
• RQ3 Does Symplectic Gradient Adjustment (SGA) improve convergence and stability in practical settings like GANs?
• RQ4 How should the adjustment sign be chosen to promote convergence to stable equilibria in general games?
• RQ5 What are the theoretical guarantees of SGA in potential and Hamiltonian games, and how does it perform empirically?

主要发现

• The Hessian of a game decomposes uniquely into symmetric and antisymmetric parts, yielding potential and Hamiltonian game classes.
• Potential games allow convergence of gradient descent to local minima of a potential function.
• Hamiltonian games admit a conserved quantity (Hamiltonian) and gradient descent on the Hamiltonian converges to a local Nash equilibrium.
• The Symplectic Gradient Adjustment (SGA) update ξλ = ξ + λ · A⊤ξ satisfies desirable properties and converges to stable fixed points in potential and Hamiltonian games.
• SGA enables faster and more robust convergence than standard gradient descent, especially in adversarial setups like GANs.
• Experiments show SGA mitigates mode collapse and mode hopping in a basic GAN setup and compares favorably to other methods in stability and convergence.