[Paper Review] Q-DeckRec: A Fast Deck Recommendation System for Collectible Card Games

Zhengxing Chen, Christopher Amato+4 more|arXiv (Cornell University)|Jun 26, 2018

Artificial Intelligence in Games21 references4 citations

TL;DR

Q-DeckRec is a reinforcement learning-based deck recommendation system that learns a policy to efficiently build winning-effective decks in collectible card games. By training a Q-learning agent to generalize across opponent decks, it reduces real-time inference to under 10 seconds—dramatically outperforming genetic algorithms requiring hours—while matching or exceeding their win rates after a 3-day training phase.

ABSTRACT

Deck building is a crucial component in playing Collectible Card Games (CCGs). The goal of deck building is to choose a fixed-sized subset of cards from a large card pool, so that they work well together in-game against specific opponents. Existing methods either lack flexibility to adapt to different opponents or require large computational resources, still making them unsuitable for any real-time or large-scale application. We propose a new deck recommendation system, named Q-DeckRec, which learns a deck search policy during a training phase and uses it to solve deck building problem instances. Our experimental results demonstrate Q-DeckRec requires less computational resources to build winning-effective decks after a training phase compared to several baseline methods.

Motivation & Objective

• To address the high computational cost of existing deck recommendation methods in collectible card games (CCGs), especially those relying on repeated simulation of game matches.
• To develop a fast, scalable, and real-time deck recommendation system suitable for large-scale deployment or live gameplay.
• To reduce reliance on domain-specific heuristics or expensive evaluation functions by learning a generalizable deck-building policy through reinforcement learning.
• To enable efficient inference for new deck-building problem instances after an initial training phase, minimizing runtime computational cost.

Proposed method

• Q-DeckRec formulates deck building as a sequential decision-making problem, where each state represents a player's current deck and opponent's deck, and actions are card modifications (add, remove, replace).
• It employs deep Q-networks (DQN) with a multi-layer perceptron (MLP) to approximate the Q-value function, mapping state-action pairs to expected win rates.
• The system is trained using a large number of simulated episodes, where each episode starts from a random initial deck and applies actions to improve win rate against a fixed opponent deck.
• During training, the agent learns a policy by maximizing cumulative rewards through experience replay and target network updates, enabling stable learning.
• After training, the policy is deployed to generate decks in real time with minimal inference cost, requiring only 9.63 seconds of CPU time on average.
• The method uses a supervised learning baseline (MC-simulation) for comparison, which predicts win rates from card features but fails to generalize well due to noisy predictions.

Experimental results

Research questions

• RQ1 Can a reinforcement learning-based policy learn to generate winning-effective decks in collectible card games with minimal real-time computation?
• RQ2 How does the performance of Q-DeckRec compare to traditional metaheuristic methods like genetic algorithms in terms of win rate and computational cost?
• RQ3 To what extent can a pre-trained Q-network generalize across different opponent decks without retraining?
• RQ4 Does the use of function approximation via deep neural networks enable efficient generalization over the vast state space of possible decks?
• RQ5 Why does a supervised learning baseline using predicted win rates fail to produce effective decks despite high R² scores?

Key findings

• Q-DeckRec achieves a win rate comparable to a 25-minute genetic algorithm (GA) but with only 9.63 seconds of CPU time per inference, representing a 1000x reduction in computational cost.
• After a 3-day training phase, Q-DeckRec produces decks as effective as GA with a 25-minute wall time limit, with no statistically significant difference in win rate.
• The supervised learning baseline (MC-simulation) peaks at a win rate of 0.84, significantly lower than Q-DeckRec’s performance, due to overfitting to noisy predictions of win rates.
• The Q-DeckRec agent generalizes effectively across the state space, learning from only 62,000 state transitions—far fewer than the total possible states (~1.97×10^50).
• The method demonstrates robustness to noisy predictions because it learns a sequential policy that avoids selecting outlier decks with spuriously high predicted win rates.
• The model’s sample efficiency could be improved by initializing episodes from real player deck distributions rather than random states, reducing exploration in irrelevant regions of the state space.