[Paper Review] Towards Automated Video Game Testing: Still a Long Way to Go

Cristiano Politowski, Yann‐Gaël Guéhéneuc+1 more|arXiv (Cornell University)|Feb 25, 2022

Artificial Intelligence in Games4 citations

TL;DR

This study investigates the gap between academic research in automated video game testing and industry practitioners' needs, finding that developers remain skeptical about using AI agents for testing due to workflow disruption and high upfront costs. Despite rising research interest, most solutions prioritize ML model performance over practical testing goals, highlighting the need for lightweight, non-disruptive tools focused on testing objectives and oracles.

ABSTRACT

As the complexity and scope of game development increase, playtesting remains an essential activity to ensure the quality of video games. Yet, the manual, ad-hoc nature of playtesting gives space to improvements in the process. In this study, we investigate gaps between academic solutions in the literature for automated video game testing and the needs of video game developers in the industry. We performed a literature review on video game automated testing and applied an online survey with video game developers. The literature results show a rise in research topics related to automated video game testing. The survey results show that game developers are skeptical about using automated agents to test games. We conclude that there is a need for new testing approaches that did not disrupt the developer workflow. As for the researchers, the focus should be on the testing goal and testing oracle.

Motivation & Objective

• To identify the disconnect between academic research in automated video game testing and real-world developer needs.
• To evaluate the desirability, viability, and feasibility of academic solutions from the perspective of practicing game developers.
• To understand why automated testing using AI agents has not been adopted despite growing research interest.
• To identify key barriers such as workflow disruption, high development costs, and lack of general-purpose tools.
• To guide future research toward practical, developer-friendly testing approaches focused on testing goals and oracles.

Proposed method

• Conducted a literature review of 166 papers on automated video game testing, filtering to 53 promising solutions.
• Selected seven representative academic solutions for evaluation based on relevance and innovation in automated testing.
• Designed and deployed an online survey targeting game developers, including designers and managers, to assess the seven solutions.
• Focused survey questions on desirability, viability, and feasibility, simplifying technical details to improve developer comprehension.
• Used qualitative and quantitative analysis to interpret survey responses and identify recurring concerns.
• Addressed potential biases by acknowledging low response rates and limited technical expertise among respondents.

Experimental results

Research questions

• RQ1 How do academic solutions for automated video game testing compare to the practical needs of video game developers?
• RQ2 What are the key barriers preventing game developers from adopting AI-based automated testing agents?
• RQ3 To what extent do developers perceive academic solutions as feasible, viable, and desirable for real-world use?
• RQ4 How do workflow disruption and upfront development costs influence the adoption of automated testing tools?
• RQ5 What role should the testing goal and testing oracle play in future research on automated video game testing?

Key findings

• There has been a significant rise in academic research on automated video game testing in recent years, particularly in ML-driven testing approaches.
• Most academic solutions prioritize the performance of ML models over the actual testing objectives, such as bug detection or gameplay quality.
• Game developers are skeptical about using AI agents for testing, viewing them as costly, time-consuming, and disruptive to existing workflows.
• Indie developers and small studios found none of the seven evaluated solutions suitable due to high setup and customization costs.
• Respondents expressed strong demand for open-source, general-purpose testing tools that integrate directly with game engines like Unreal Engine.
• Despite advances in AI, developers believe human judgment remains essential for assessing fun, consistency, and usability—tasks AI cannot yet replicate effectively.