[Paper Review] Monte Carlo Neural Fictitious Self-Play: Approach to Approximate Nash equilibrium of Imperfect-Information Games

Li Zhang, Wei Wang+2 more|arXiv (Cornell University)|Mar 22, 2019

Artificial Intelligence in Games19 references5 citations

TL;DR

This paper proposes Monte Carlo Neural Fictitious Self-Play (MC-NFSP) and Asynchronous Neural Fictitious Self-Play (ANFSP), which combine Monte Carlo Tree Search (MCTS) with Neural Fictitious Self-Play (NFSP) to achieve faster and more stable convergence to approximate Nash equilibrium in large-scale imperfect-information games. MC-NFSP improves sample efficiency and enables convergence in deep, complex games like Othello where NFSP fails, while ANFSP accelerates training via parallel actor-learners, reducing memory and training time in poker games.

ABSTRACT

Researchers on artificial intelligence have achieved human-level intelligence in large-scale perfect-information games, but it is still a challenge to achieve (nearly) optimal results (in other words, an approximate Nash Equilibrium) in large-scale imperfect-information games (i.e. war games, football coach or business strategies). Neural Fictitious Self Play (NFSP) is an effective algorithm for learning approximate Nash equilibrium of imperfect-information games from self-play without prior domain knowledge. However, it relies on Deep Q-Network, which is off-line and is hard to converge in online games with changing opponent strategy, so it can't approach approximate Nash equilibrium in games with large search scale and deep search depth. In this paper, we propose Monte Carlo Neural Fictitious Self Play (MC-NFSP), an algorithm combines Monte Carlo tree search with NFSP, which greatly improves the performance on large-scale zero-sum imperfect-information games. Experimentally, we demonstrate that the proposed Monte Carlo Neural Fictitious Self Play can converge to approximate Nash equilibrium in games with large-scale search depth while the Neural Fictitious Self Play can't. Furthermore, we develop Asynchronous Neural Fictitious Self Play (ANFSP). It use asynchronous and parallel architecture to collect game experience. In experiments, we show that parallel actor-learners have a further accelerated and stabilizing effect on training.

Motivation & Objective

• To address the poor convergence of Neural Fictitious Self-Play (NFSP) in large-scale, deep imperfect-information games due to reliance on offline Deep Q-Networks (DQN).
• To improve sample efficiency and training stability in self-play by integrating Monte Carlo Tree Search (MCTS) into NFSP for online, online-offline hybrid learning.
• To reduce training time and memory usage in NFSP by introducing an asynchronous, parallel actor-learner architecture for experience collection.
• To evaluate the proposed methods in complex environments, including Othello, modified Leduc Hold’em, and a 10v10 FPS team combat game, to demonstrate superhuman performance.

Proposed method

• MC-NFSP integrates MCTS with NFSP, using MCTS to generate high-quality rollouts for training the best-response Q-network, replacing the offline DQN used in standard NFSP.
• The algorithm maintains two networks: a Q-network for learning best responses via MCTS-guided rollouts, and a supervised network for learning the average policy from historical actions.
• MCTS enables online planning and exploration, allowing the agent to adapt to changing opponent strategies and improve sample efficiency in deep game trees.
• ANFSP employs asynchronous, parallel actor-learners that interact with independent environments, share parameters, and update gradients in mini-batches to reduce memory and accelerate training.
• In ANFSP, gradient updates are performed on a shared Q-network and policy network using experience collected in parallel, with target networks updated periodically.
• The method uses a mixture of the average policy and best response via anticipatory dynamics, with $ε$-greedy exploration decaying over time.

Experimental results

Research questions

• RQ1 Can MCTS improve the convergence and performance of NFSP in large-scale imperfect-information games with deep search trees?
• RQ2 Does replacing offline DQN with online MCTS-based rollouts lead to faster and more stable learning in self-play?
• RQ3 Can an asynchronous, parallel actor-learner architecture reduce training time and memory usage in NFSP while maintaining or improving convergence?
• RQ4 Can MC-NFSP and ANFSP achieve approximate Nash equilibrium in games where standard NFSP fails to converge?
• RQ5 Can the combined approach produce superhuman policies in complex, real-time imperfect-information games like team-based FPS combat?

Key findings

• MC-NFSP successfully converged to an approximate Nash equilibrium in Othello, while standard NFSP failed to converge even after 700,000 episodes.
• In modified Leduc Hold’em, ANFSP achieved lower exploitability and faster convergence than NFSP, with training completing in less than 2 hours and achieving lower exploitability than NFSP’s 0.75.
• ANFSP reduced memory requirements and training time by enabling parallel experience collection and mini-batch gradient updates across multiple actors.
• In the FPS team combat game, the MC-NFSP agent achieved a win rate of 75% against 10 college students over 100 games, demonstrating superhuman performance.
• The training in the FPS game converged in under 150 episodes, with win rate exceeding 80% against a human-trained baseline (SL-Human), and training loss near zero.
• The combination of MCTS and NFSP enabled stable, high-performance policy learning in environments with high strategic depth and imperfect information.