[Paper Review] Combining Deep Reinforcement Learning and Search for Imperfect-Information Games

Noam Brown, Anton Bakhtin+2 more|arXiv (Cornell University)|Jul 27, 2020

Artificial Intelligence in Games54 references62 citations

TL;DR

The paper introduces ReBeL, a self-play RL+search framework for two-player zero-sum imperfect-information games that converges to Nash equilibria and achieves superhuman poker performance with reduced domain knowledge.

ABSTRACT

The combination of deep reinforcement learning and search at both training and test time is a powerful paradigm that has led to a number of successes in single-agent settings and perfect-information games, best exemplified by AlphaZero. However, prior algorithms of this form cannot cope with imperfect-information games. This paper presents ReBeL, a general framework for self-play reinforcement learning and search that provably converges to a Nash equilibrium in any two-player zero-sum game. In the simpler setting of perfect-information games, ReBeL reduces to an algorithm similar to AlphaZero. Results in two different imperfect-information games show ReBeL converges to an approximate Nash equilibrium. We also show ReBeL achieves superhuman performance in heads-up no-limit Texas hold'em poker, while using far less domain knowledge than any prior poker AI.

Motivation & Objective

• Motivate the need for RL+Search in imperfect-information games and address convergence issues missing in prior methods.
• Develop a general framework (ReBeL) that expands states to public belief states (PBS) and enables RL+Search with theoretical guarantees.
• Show empirical success in large-scale imperfect-information games and poker benchmarks with reduced domain knowledge.

Proposed method

• Introduce public belief states (PBS) to convert imperfect-information games into a continuous-state perfect-information-like setting.
• Train a value network over infostates and a policy network, using self-play and search to solve depth-limited subgames.
• Use CFR-D (or FP) as the equilibrium-finding routine inside subgames, with leaf values provided by the learned value network.
• Prove that infostate values act as supergradients of PBS values, enabling convergent search.
• Demonstrate safe-test search: at test time, run the same training-time algorithm to preserve equilibrium play in expectation.

Experimental results

Research questions

• RQ1 Can RL+Search frameworks be made sound and convergent in two-player zero-sum imperfect-information games?
• RQ2 Does supervision via PBS-based value and policy networks enable convergence to Nash equilibria in large imperfect-information domains?
• RQ3 Can the approach achieve superhuman performance in real-world imperfect-information games like heads-up no-limit Texas hold’em poker with limited domain knowledge?

Key findings

• ReBeL converges to an approximate Nash equilibrium in imperfect-information benchmarks such as Liar’s Dice.
• In heads-up no-limit Texas hold’em poker, ReBeL achieves superhuman performance with far less domain knowledge than prior poker AIs.
• Compared to strong prior bots, ReBeL shows competitive exploitability metrics and fast decision times (often under a few seconds).
• The framework provides theoretical guarantees: infostate values correspond to supergradients, and training with T CFR iterations yields bounded error (O(1/√T)).
• At test time, safe search guarantees convergence to an approximate Nash equilibrium when using a well-trained PBS value network.