[Paper Review] Testing match-3 video games with Deep Reinforcement Learning

Nicholas Napolitano|arXiv (Cornell University)|Jun 30, 2020

Artificial Intelligence in Games13 references4 citations

TL;DR

This paper proposes using a Dueling Deep Q-Network (DDQN) for automated testing of match-3 video games, demonstrating that the agent learns to play Jelly Juice effectively by extracting visual features and selecting optimal moves. The DDQN outperforms random players and closely matches human success rates across multiple levels, showing strong adaptability and progressive learning over time.

ABSTRACT

Testing a video game is a critical step for the production process and requires a great effort in terms of time and resources spent. Some software houses are trying to use the artificial intelligence to reduce the need of human resources using systems able to replace a human agent. We study the possibility to use the Deep Reinforcement Learning to automate the testing process in match-3 video games and suggest to approach the problem in the framework of a Dueling Deep Q-Network paradigm. We test this kind of network on the Jelly Juice game, a match-3 video game developed by the redBit Games. The network extracts the essential information from the game environment and infers the next move. We compare the results with the random player performance, finding that the network shows a highest success rate. The results are in most cases similar with those obtained by real users, and the network also succeeds in learning over time the different features that distinguish the game levels and adapts its strategy to the increasing difficulties.

Motivation & Objective

• To develop an automated synthetic player for match-3 games using deep reinforcement learning.
• To evaluate whether a deep reinforcement learning agent can simulate human-like behavior in gameplay.
• To assess the effectiveness of a Dueling Deep Q-Network in learning optimal strategies across varying levels of difficulty in Jelly Juice.
• To compare the performance of the DDQN agent against random players and real human users in terms of success rate.

Proposed method

• The study employs a Dueling Deep Q-Network (DDQN) to process visual game states and select optimal moves.
• Convolutional neural networks extract features from raw game frames, enabling the agent to interpret the game environment.
• Experience replay memory stores past experiences to improve training stability and sample efficiency.
• The network uses a double Q-learning update rule to reduce overestimation bias in Q-value predictions.
• A supervised learning block filters invalid moves and normalizes action probabilities before selection.
• Training is conducted on a custom dataset of game configurations and moves, with hyperparameters tuned for convergence.

Experimental results

Research questions

• RQ1 Can a Dueling Deep Q-Network effectively learn to play a match-3 game like Jelly Juice with minimal human supervision?
• RQ2 How does the performance of the DDQN agent compare to a random player and real human players across different levels?
• RQ3 Does the DDQN agent adapt its strategy over time to handle increasing level complexity?
• RQ4 To what extent can the DDQN replicate the decision-making patterns of human players?

Key findings

• The DDQN agent achieved a 91.3% success rate on Level 1, significantly outperforming the random player's 65%.
• On Level 21, the DDQN reached 74% success, compared to the random player’s 54% and human users’ 94.7%.
• On the most complex level (505), the DDQN achieved 9.33% success, surpassing the random player’s 0% and approaching human performance (13.05%).
• The agent required approximately 150 games to stabilize performance, showing progressive learning and adaptation to level-specific challenges.
• The DDQN demonstrated behavior similar to human players across most levels, particularly in move selection and strategic planning.
• The system showed robustness even on high-complexity levels, indicating potential generalization to other match-3 games.