[Paper Review] Double Neural Counterfactual Regret Minimization

Hui Li, Kailiang Hu+4 more|arXiv (Cornell University)|Dec 27, 2018

Artificial Intelligence in Games13 references15 citations

TL;DR

This paper proposes Double Neural Counterfactual Regret Minimization (DNCFR), a deep learning-based approach that uses two neural networks—one to track cumulative regret and another to represent average strategy—enabling end-to-end learning in large imperfect information games. The method achieves convergence comparable to tabular CFR while enabling continual improvement from poor initial strategies and generalizing well with minimal data.

ABSTRACT

Counterfactual Regret Minimization (CRF) is a fundamental and effective technique for solving Imperfect Information Games (IIG). However, the original CRF algorithm only works for discrete state and action spaces, and the resulting strategy is maintained as a tabular representation. Such tabular representation limits the method from being directly applied to large games and continuing to improve from a poor strategy profile. In this paper, we propose a double neural representation for the imperfect information games, where one neural network represents the cumulative regret, and the other represents the average strategy. Furthermore, we adopt the counterfactual regret minimization algorithm to optimize this double neural representation. To make neural learning efficient, we also developed several novel techniques including a robust sampling method, mini-batch Monte Carlo Counterfactual Regret Minimization (MCCFR) and Monte Carlo Counterfactual Regret Minimization Plus (MCCFR+) which may be of independent interests. Experimentally, we demonstrate that the proposed double neural algorithm converges significantly better than the reinforcement learning counterpart.

Motivation & Objective

• Address the limitations of tabular Counterfactual Regret Minimization (CRF) in large imperfect information games (IIGs), where memory and scalability are major constraints.
• Overcome the reliance on handcrafted features or game abstraction in prior deep reinforcement learning methods for IIGs by enabling end-to-end optimization.
• Develop a neural framework that can track cumulative regret and average strategy simultaneously, enabling convergence to Nash equilibrium without tabular representations.
• Enable continual strategy improvement from existing checkpoints, supporting iterative refinement from suboptimal policies.
• Demonstrate that purely neural methods can match the performance of tabular CFR in large games like Leduc Hold’em, resolving an open question in the field.

Proposed method

• Introduce a double neural architecture: one deep neural network learns the cumulative regret, and another learns the average strategy, both updated via counterfactual regret minimization.
• Use a novel robust sampling method that reduces variance compared to outcome sampling while maintaining lower memory usage than external sampling.
• Implement mini-batch Monte Carlo Counterfactual Regret Minimization (MCCFR) and MCCFR+ to improve training efficiency and stability.
• Train the two networks end-to-end using a differentiable objective that minimizes regret and approximates the Nash equilibrium through iterative strategy averaging.
• Employ a warm-start mechanism where the neural networks are initialized by cloning tabular regret and strategy values from existing CFR runs, enabling continual improvement.
• Use attention-enhanced RNNs (e.g., LSTM with attention) to improve generalization across unseen information sets, especially with small batch sizes.

Experimental results

Research questions

• RQ1 Can a purely neural, end-to-end approach match the performance of tabular CFR in large imperfect information games?
• RQ2 Does the double neural representation of cumulative regret and average strategy enable convergence to a Nash equilibrium comparable to tabular methods?
• RQ3 Can the method generalize to unseen information sets when only a small fraction of nodes are visited per iteration?
• RQ4 How effective is the method in continual improvement from an existing strategy checkpoint?
• RQ5 Does attention-based architecture enhance generalization and convergence speed in neural CFR?

Key findings

• The double neural method achieves exploitability of 0.02 after 1000 iterations in Leduc Hold’em, matching the performance of tabular CFR and significantly outperforming NFSP and XFP.
• With only 3.08% of information sets observed per iteration (b=50), the method still converges to exploitability below 0.1 within 1000 iterations, demonstrating strong generalization.
• The method generalizes effectively even with minimal parameters—embedding sizes of 8 and 16 (1048 and 2608 parameters) outperform tabular memory in compression while maintaining convergence.
• In a large game with over 2×10^7 states and 3.7×10^6 information sets, the method converges using only 0.53% of nodes per iteration (b=500), showing scalability.
• The use of attention in the RNN architecture (e.g., LSTM+attention) leads to faster convergence and better strategy quality than fully connected networks, RNNs, or GRUs.
• Warm-starting from tabular CFR or RS-MCCFR+ and continuing with the double neural method enables continual improvement, with the first 10 iterations from tabular methods and the rest from neural learning.