[Paper Review] Computer Games Are Serious Business and so is their Quality: Particularities of Software Testing in Game Development from the Perspective of Practitioners

Ronnie E. S. Santos, Cleyton Magalhães+4 more|arXiv (Cornell University)|Dec 12, 2018

Educational Games and GamificationPsychology23 references3 citations

TL;DR

This study investigates the unique challenges of software testing in computer game development through a mixed-method case study and survey of 38 practitioners. It reveals that game testing must extend beyond functional correctness to include evaluation of game balance, physics behavior, and entertainment quality, highlighting the need for specialized testing approaches tailored to gaming-specific non-functional requirements.

ABSTRACT

Over the last several decades, computer games started to have a significant impact on society. However, although a computer game is a type of software, the process to conceptualize, produce and deliver a game could involve unusual features. In software testing, for instance, studies demonstrated the hesitance of professionals to use automated testing techniques with games, due to the constant changes in requirements and design, and pointed out the need for creating testing tools that take into account the flexibility required for the game development process. Goal. This study aims to improve the current body of knowledge regarding software testing in game development and point out the existing particularities observed in software testing considering the development of a computer game. Method. A mixed-method approach based on a case study and an opinion survey was applied to collect quantitative and qualitative data from software professionals regarding the particularities of software testing in game development. Results. We analyzed over 70 messages posted on three well-established network of question-and-answer communities related to software engineering, software testing and game development and received answers of 38 professionals discussing differences between testing a computer game and a general software, and identified important aspects to be observed by practitioners in the process of planning, performing and reporting tests in this context. Conclusion. Considering computer games, software testing must focus not only on the common aspects of a general software, but also, track and investigate issues that could be related to game balance, game physics and entertainment related-aspects to guarantee the quality of computer games and a successful testing process.

Motivation & Objective

• To identify and analyze the specific challenges of software testing in computer game development as perceived by industry practitioners.
• To understand why traditional software testing practices often fall short in game development contexts.
• To explore the unique non-functional quality attributes—such as gameplay balance and realism—that significantly impact game testing.
• To propose a framework for improving testing practices by integrating game-specific quality dimensions into the testing lifecycle.
• To bridge the gap between general software testing methodologies and the unique demands of game development through practitioner-driven insights.

Proposed method

• Conducted a mixed-methods study combining qualitative analysis of 70+ messages from three Q&A communities (Stack Overflow, Software Engineering Stack Exchange, and GameDev Stack Exchange).
• Administered a survey to 38 software professionals with experience in game development and testing to gather insights on testing practices and challenges.
• Used thematic analysis to identify recurring patterns and unique concerns in game testing, focusing on differences from general software testing.
• Collected and analyzed practitioner opinions on automation limitations, test design, and reporting in game projects.
• Integrated findings into a conceptual model emphasizing non-functional testing dimensions such as game balance, physics, and player experience.
• Validated findings through cross-referencing with existing literature on software testing and game development.

Experimental results

Research questions

• RQ1 What are the primary differences between testing computer games and general-purpose software from a practitioner's perspective?
• RQ2 Why do software testing professionals hesitate to adopt automated testing techniques in game development?
• RQ3 Which non-functional quality attributes—beyond functionality—are most critical in evaluating the quality of a computer game?
• RQ4 How do game-specific factors such as balance, physics, and entertainment value influence test planning and execution?
• RQ5 What unique challenges do testers face in reporting and tracking issues related to gameplay and user experience in games?

Key findings

• Practitioners consistently emphasized that game testing must evaluate non-functional aspects such as game balance, physics behavior, and player engagement, which are not typically prioritized in general software testing.
• Automated testing is often avoided in game development due to the high volatility of game design and frequent changes in gameplay mechanics and content.
• The majority of testing efforts in games focus on exploratory and manual testing rather than scripted or automated test cases, especially in early development stages.
• Test reporting in game development frequently includes subjective assessments of fun, challenge, and immersion, which are difficult to quantify but critical to quality.
• Game-specific issues such as exploitability, progression pacing, and AI behavior are frequently identified as high-priority test concerns by practitioners.
• There is a significant gap in existing testing tools and methodologies that can effectively handle the dynamic, creative, and experiential nature of game development.