[Paper Review] DouZero: Mastering DouDizhu with Self-Play Deep Reinforcement Learning

Daochen Zha, Jingru Xie+5 more|arXiv (Cornell University)|Jun 11, 2021

Digital Games and MediaSocial Sciences21 citations

TL;DR

DouZero is a self-play deep reinforcement learning agent that masters DouDizhu, a complex three-player trick-taking card game with imperfect information and a massive action space. By combining Monte Carlo tree search with deep neural networks, action encoding via card matrices, and parallel self-play training, DouZero achieves superhuman performance—ranking first on the Botzone leaderboard among 344 agents—without relying on human-designed abstractions or domain-specific heuristics.

ABSTRACT

Games are abstractions of the real world, where artificial agents learn to compete and cooperate with other agents. While significant achievements have been made in various perfect- and imperfect-information games, DouDizhu (a.k.a. Fighting the Landlord), a three-player card game, is still unsolved. DouDizhu is a very challenging domain with competition, collaboration, imperfect information, large state space, and particularly a massive set of possible actions where the legal actions vary significantly from turn to turn. Unfortunately, modern reinforcement learning algorithms mainly focus on simple and small action spaces, and not surprisingly, are shown not to make satisfactory progress in DouDizhu. In this work, we propose a conceptually simple yet effective DouDizhu AI system, namely DouZero, which enhances traditional Monte-Carlo methods with deep neural networks, action encoding, and parallel actors. Starting from scratch in a single server with four GPUs, DouZero outperformed all the existing DouDizhu AI programs in days of training and was ranked the first in the Botzone leaderboard among 344 AI agents. Through building DouZero, we show that classic Monte-Carlo methods can be made to deliver strong results in a hard domain with a complex action space. The code and an online demo are released at https://github.com/kwai/DouZero with the hope that this insight could motivate future work.

Motivation & Objective

• To develop a strong AI agent for DouDizhu, a challenging three-player trick-taking game with imperfect information and a vast, dynamic action space.
• To overcome the limitations of traditional reinforcement learning algorithms like DQN and A3C, which struggle with overestimation and poor generalization in large action spaces.
• To eliminate reliance on human-designed heuristics or action abstractions, such as kicker networks or decomposition rules, which are error-prone and computationally expensive.
• To demonstrate that classic Monte Carlo methods can be significantly enhanced with deep learning to achieve strong performance in complex, high-dimensional game environments.
• To train an agent from scratch using only self-play and a single server with four GPUs, achieving superhuman performance in a fraction of the time required by prior methods.

Proposed method

• Employs a self-play reinforcement learning framework with parallel actors to generate diverse game trajectories efficiently.
• Enhances Monte Carlo tree search (MCTS) with a deep neural network that predicts policy and value functions using raw card states.
• Encodes actions as card matrices (e.g., representing pairs, sequences, and combinations), enabling the model to generalize across unseen legal actions.
• Uses a residual convolutional neural network to process card hands and encode action representations, improving generalization and reducing overestimation bias.
• Applies a weighted policy (WP) objective during self-play to stabilize learning and improve policy convergence.
• Trains the agent end-to-end from scratch using only game rules and self-play, without any human demonstrations or heuristic priors.

Experimental results

Research questions

• RQ1 Can a self-play deep reinforcement learning agent achieve superhuman performance in DouDizhu without relying on human-designed action abstractions?
• RQ2 How effective is a neural network-enhanced Monte Carlo tree search in handling the massive and dynamic action space of DouDizhu?
• RQ3 To what extent can action encoding via card matrices improve generalization over rare or unseen legal moves in complex card games?
• RQ4 Can a system trained from scratch on a single server with four GPUs outperform existing AI agents that use complex heuristics or Bayesian inference?
• RQ5 Does the absence of domain-specific abstractions lead to more robust and generalizable policies in imperfect-information, team-based card games?

Key findings

• DouZero outperformed all existing DouDizhu AI programs, ranking first on the Botzone leaderboard among 344 agents after only days of training.
• The agent achieved a winning rate of over 90% against strong baselines, including CQN and heuristic rule-based agents, within half a day of training.
• DouZero surpassed the performance of DeltaDou, a prior SOTA agent that required over two months of training and relied on Bayesian inference and heuristic kickers.
• The model demonstrated strong generalization, effectively handling rare or unseen card combinations through its action encoding mechanism.
• The system achieved superhuman performance without any human-designed abstractions, relying solely on self-play and neural network policy/value networks.
• The use of card matrix encoding enabled the agent to generalize across actions not frequently seen during training, reducing overestimation bias compared to standard DQN approaches.