[Paper Review] Scripted GUI Testing of Android Apps: A Study on Diffusion, Evolution and Fragility

Riccardo Coppola, Maurizio Morisio+1 more|arXiv (Cornell University)|Nov 9, 2017

Advanced Malware Detection Techniques10 references3 citations

TL;DR

This study investigates the adoption, evolution, and fragility of scripted GUI testing in open-source Android apps using six popular testing frameworks. It finds that only 8% of Android projects use GUI testing tools, with 60% of test code modifications driven by GUI fragility—highlighting a major maintenance burden that may deter widespread adoption of automated GUI testing.

ABSTRACT

Background. Evidence suggests that mobile applications are not thoroughly tested as their desktop counterparts. In particular GUI testing is generally limited. Like web-based applications, mobile apps suffer from GUI test fragility, i.e. GUI test classes failing due to minor modifications in the GUI, without the application functionalities being altered. Aims. The objective of our study is to examine the diffusion of GUI testing on Android, and the amount of changes required to keep test classes up to date, and in particular the changes due to GUI test fragility. We define metrics to characterize the modifications and evolution of test classes and test methods, and proxies to estimate fragility-induced changes. Method. To perform our experiments, we selected six widely used open-source tools for scripted GUI testing of mobile applications previously described in the literature. We have mined the repositories on GitHub that used those tools, and computed our set of metrics. Results. We found that none of the considered GUI testing frameworks achieved a major diffusion among the open-source Android projects available on GitHub. For projects with GUI tests, we found that test suites have to be modified often, specifically 5\%-10\% of developers' modified LOCs belong to tests, and that a relevant portion (60\% on average) of such modifications are induced by fragility. Conclusions. Fragility of GUI test classes constitute a relevant concern, possibly being an obstacle for developers to adopt automated scripted GUI tests. This first evaluation and measure of fragility of Android scripted GUI testing can constitute a benchmark for developers, and the basis for the definition of a taxonomy of fragility causes, and actionable guidelines to mitigate the issue.

Motivation & Objective

• To assess the diffusion of scripted GUI testing frameworks in open-source Android projects.
• To measure the frequency and causes of test suite modifications over time.
• To quantify the portion of test changes attributable to GUI test fragility.
• To provide a benchmark for developers and a foundation for mitigating fragility in automated GUI testing.

Proposed method

• The authors mined GitHub repositories of open-source Android projects using six popular GUI testing tools: Espresso, UI Automator, Selendroid, Robotium, Robolectric, and Appium.
• They analyzed version history at the release level, tracking changes in test classes and methods across tagged releases.
• Fragility was estimated using a proxy: any modification in a test method was considered indicative of GUI fragility.
• Metrics were computed to characterize test suite evolution, including percentage of modified lines in test code and the proportion of changes due to fragility.
• The study used precision evaluation (70%) to validate the proxy's reliability in linking test changes to GUI modifications.
• Statistical analysis was performed on a sample of 1,000 projects with at least one tagged release to ensure representativeness.

Experimental results

Research questions

• RQ1 What is the level of adoption of scripted GUI testing frameworks in open-source Android projects?
• RQ2 How frequently are test suites modified during the evolution of Android projects?
• RQ3 What proportion of test code modifications are caused by GUI test fragility?
• RQ4 How reliable is the proxy of test method modification for estimating GUI fragility?

Key findings

• Only 8% of open-source Android projects using at least one tagged release adopted any of the six GUI testing frameworks studied.
• On average, 7.5% of all code changes per release were in test code, with 3% of test code being modified per release.
• Approximately 60% of test code modifications were attributed to GUI test fragility, indicating a major maintenance burden.
• The proxy of test method modification correctly identified fragility in 70% of cases, supporting its use for estimating fragility.
• Test suites represented about 11% of total project code on average in projects that used GUI testing.
• One in five test classes, on average, required modifications due to GUI fragility in each release cycle.