[Paper Review] From Manual Android Tests to Automated and Platform Independent Test Scripts

Mattia Fazzini, Eduardo Noronha de Andrade Freitas+2 more|arXiv (Cornell University)|Aug 11, 2016

Software Testing and Debugging Techniques27 references3 citations

TL;DR

This paper presents BARISTA, a tool that enables testers to create platform-independent, automated test scripts for Android apps by recording user interactions and visually specifying expected outcomes (oracles) through an intuitive interface. The technique leverages Android's accessibility framework to capture actions and oracles without modifying the app, generating Espresso-compatible test scripts that can be reliably executed across diverse devices and OS versions, outperforming existing tools in portability and robustness.

ABSTRACT

Because Mobile apps are extremely popular and often mission critical nowadays, companies invest a great deal of resources in testing the apps they provide to their customers. Testing is particularly important for Android apps, which must run on a multitude of devices and operating system versions. Unfortunately, as we confirmed in many interviews with quality assurance professionals, app testing is today a very human intensive, and therefore tedious and error prone, activity. To address this problem, and better support testing of Android apps, we propose a new technique that allows testers to easily create platform independent test scripts for an app and automatically run the generated test scripts on multiple devices and operating system versions. The technique does so without modifying the app under test or the runtime system, by (1) intercepting the interactions of the tester with the app and (2) providing the tester with an intuitive way to specify expected results that it then encode as test oracles. We implemented our technique in a tool named Barista and used the tool to evaluate the practical usefulness and applicability of our approach. Our results show that Barista can faithfully encode user defined test cases as test scripts with built-in oracles, generates test scripts that can run on multiple platforms, and can outperform a state-of-the-art tool with similar functionality. Barista and our experimental infrastructure are publicly available.

Motivation & Objective

• To address the high human effort and error-proneness of manual Android app testing across fragmented devices and OS versions.
• To reduce the need for specialized programming skills in test automation by enabling non-developers to create test scripts through interaction.
• To support the creation of robust, reusable test cases with built-in oracles that are independent of specific UI layouts or device configurations.
• To enable automatic execution of generated test scripts on multiple platforms without re-encoding.
• To improve test portability and maintainability by generating platform-agnostic test scripts using standard frameworks like Espresso.

Proposed method

• Recording user interactions with the app via Android's accessibility framework to capture touch, input, and navigation events.
• Providing a visual interface for testers to specify expected results (oracles) by selecting UI elements and defining expected states.
• Encoding both recorded actions and specified oracles into platform-independent test scripts using the Espresso testing framework format.
• Translating the recorded execution traces into structured test code that can be executed on any device or emulator with compatible Android versions.
• Using accessibility identifiers and UI hierarchy information to ensure test scripts remain robust against minor UI changes not affecting the logical structure.
• Leveraging the Android accessibility API to intercept and log interactions without modifying the app or runtime system.

Experimental results

Research questions

• RQ1 Can a non-programmer create reliable, platform-independent test scripts for Android apps through interactive recording and visual oracle specification?
• RQ2 How does the portability of test scripts generated by BARISTA compare to existing record/replay tools across different devices and OS versions?
• RQ3 To what extent does BARISTA support the creation of robust oracles that remain valid despite minor UI changes?
• RQ4 How does BARISTA's performance in test execution and script generation compare to state-of-the-art tools like Espresso or UIAutomator?
• RQ5 Can BARISTA-generated test scripts be effectively reused for regression testing across app updates?

Key findings

• BARISTA successfully encodes user-defined test cases with oracles into platform-independent test scripts that are executable on multiple devices and Android OS versions.
• The generated test scripts are more portable than those from existing record/replay tools, as they rely on semantic UI identifiers rather than screen coordinates.
• BARISTA outperforms a state-of-the-art tool with similar functionality in terms of test script reliability and cross-platform execution.
• The tool enables testers to create test scripts with minimal training, without requiring knowledge of programming or testing frameworks.
• Test scripts generated by BARISTA are resilient to minor UI changes, such as repositioning of elements, as long as the logical structure and accessibility labels remain unchanged.
• The approach supports the record-once-run-anywhere principle, significantly reducing maintenance overhead for regression testing.