[Paper Review] A Systematic Literature Review of Automated Techniques for Functional GUI Testing of Mobile Applications

Yauhen Leanidavich Arnatovich, Lipo Wang|arXiv (Cornell University)|Dec 30, 2018

Software Testing and Debugging Techniques79 references4 citations

TL;DR

This systematic literature review evaluates 25 primary studies on automated functional GUI testing for mobile apps, analyzing effectiveness, efficiency, and practicality. It finds current techniques are only ~50% effective, inefficient (often >30 minutes per app), and only ~50% practical due to trade-offs between effectiveness and efficiency, highlighting the need for better model compression and intelligent input generation.

ABSTRACT

Context. Multiple automated techniques have been proposed and developed for mobile application GUI testing aiming to improve effectiveness, efficiency, and practicality. The effectiveness, efficiency, and practicality are 3 fundamental characteristics which testing techniques are built upon, and need to be continuously improved to deliver useful solutions for researchers and practitioners, and community as a whole. Objective. In this systematic review, we attempt to provide a broad picture of existing mobile testing tools by collating and analysing their conceptual, and also performance characteristics including an estimation of effectiveness, efficiency, and practicality. Method. To achieve our objective, we specify 3 primary, and 14 secondary review questions, and conducted an analysis of 25 primary studies. We first individually analyse each primary study, and next analyse the primary studies as a whole. We developed a review protocol which defines all the details of our systematic review. Results. From effectiveness, we conclude that testing techniques which implement model-checking, symbolic execution, constraint solving, and search-based test generation approach tend to be more effective than those implementing random test generation. From efficiency, we conclude that testing techniques which implement code search-based testing approaches tend to be more efficient than those implementing GUI model-based. From practicality, we conclude that the more effective a testing technique is, the less efficient it will be. Conclusion. For effectiveness, we observe that the existing automated testing techniques are not effective enough, and currently they achieve nearly half of the desired level of effectiveness. For efficiency, we observe that current automated testing techniques are not efficient enough.

Motivation & Objective

• To assess the current state of automated functional GUI testing techniques for mobile applications.
• To evaluate the effectiveness, efficiency, and practicality of existing automated testing tools.
• To identify gaps in test generation approaches, particularly in input generation and model complexity.
• To guide future research by highlighting underperforming areas such as input diversity and model scalability.

Proposed method

• Conducted a systematic literature review using a predefined protocol with 3 primary and 14 secondary review questions.
• Selected 25 primary studies through rigorous inclusion and exclusion criteria based on relevance and quality.
• Analyzed each study individually and then collectively to assess conceptual and performance characteristics.
• Used a custom quality assessment scale to evaluate study quality, minimizing human bias through multi-reviewer consensus.
• Mapped findings across effectiveness (e.g., model-checking vs. random testing), efficiency (e.g., code search vs. GUI model-based), and practicality (real-world usability).
• Evaluated threats to validity by analyzing research design and results discussion in primary studies.

Experimental results

Research questions

• RQ1 What are the most effective automated GUI testing techniques for mobile applications, and how do they compare in terms of test coverage and fault detection?
• RQ2 How efficient are existing automated testing techniques in terms of execution time and resource usage per application?
• RQ3 What factors contribute to the practicality (or impracticality) of current mobile GUI testing tools in real-world settings?
• RQ4 How do different test generation strategies (e.g., symbolic execution, random input, search-based) impact effectiveness and efficiency?
• RQ5 What are the key limitations in current approaches, particularly regarding input generation and model complexity?

Key findings

• Model-checking, symbolic execution, constraint solving, and search-based test generation achieve higher effectiveness than random test generation.
• Code search-based testing approaches are more efficient than GUI model-based approaches, which suffer from high overhead.
• Existing techniques achieve only about 50% of the desired effectiveness level, indicating a significant gap in fault detection capability.
• Most automated testing techniques require more than 30 minutes per application, with some taking several hours, indicating low efficiency.
• Only approximately half of the tested tools are practical for real-world use due to poor effectiveness and efficiency trade-offs.
• Automated textual input generation remains underdeveloped, with most tools relying on random text, which limits effectiveness.