[Paper Review] GIFdroid: Automated Replay of Visual Bug Reports for Android Apps

Sidong Feng, Chunyang Chen|arXiv (Cornell University)|Dec 8, 2021

Advanced Malware Detection Techniques4 citations

TL;DR

GIFdroid is a lightweight, image-processing-based tool that automatically replays visual bug reports in Android apps by extracting keyframes from GIF recordings, mapping them to GUI states in a UI Transition Graph (UTG), and reconstructing a complete execution trace to reproduce the bug. It successfully reproduces 82% of 61 real-world video bug reports, significantly reducing manual debugging effort for developers.

ABSTRACT

Bug reports are vital for software maintenance that allow users to inform developers of the problems encountered while using software. However, it is difficult for non-technical users to write clear descriptions about the bug occurrence. Therefore, more and more users begin to record the screen for reporting bugs as it is easy to be created and contains detailed procedures triggering the bug. But it is still tedious and time-consuming for developers to reproduce the bug due to the length and unclear actions within the recording. To overcome these issues, we propose GIFdroid, a light-weight approach to automatically replay the execution trace from visual bug reports. GIFdroid adopts image processing techniques to extract the keyframes from the recording, map them to states in GUI Transitions Graph, and generate the execution trace of those states to trigger the bug. Our automated experiments and user study demonstrate its accuracy, efficiency, and usefulness of the approach.

Motivation & Objective

• To address the challenge of manually reproducing bugs from visual recordings in Android apps, which are time-consuming and error-prone for developers.
• To reduce the effort required by developers to interpret and replay video-based bug reports by automating the trace reconstruction process.
• To enable automated, efficient, and accurate replay of bugs directly from user-generated GIF recordings without requiring app instrumentation or framework integration.
• To evaluate the feasibility and usefulness of a lightweight, image-processing-based approach for visual bug report replay in real-world Android applications.

Proposed method

• Extract keyframes from video recordings by comparing frame similarity to identify fully rendered GUI states.
• Map extracted keyframes to corresponding GUI states in the app's existing UI Transition Graph (UTG) using pixel- and structural-based image similarity metrics.
• Reconstruct a complete execution trace from app launch to the bug-triggering state by auto-completing missing steps using the mapped sequence and graph traversal.
• Apply a novel algorithm to handle defective or incomplete mappings and ensure robust trace generation despite missing or ambiguous frames.
• Leverage publicly available Android app UTGs and pre-processed visual data to enable efficient, background execution without runtime instrumentation.
• Integrate the approach into a fully automated pipeline that processes user-uploaded GIFs and outputs executable test traces for reproduction.

Experimental results

Research questions

• RQ1 Can a lightweight, image-processing-based approach accurately map keyframes from user-generated GIF recordings to GUI states in an Android app's UI Transition Graph?
• RQ2 To what extent can the proposed method reconstruct a complete and executable execution trace from a partial video recording that does not start from app launch?
• RQ3 How effective is the automated replay approach in reproducing real-world visual bug reports compared to existing baselines?
• RQ4 How useful is the tool in practice, as perceived by developers during real-world debugging scenarios?

Key findings

• GIFdroid successfully reproduced 82% (50 out of 61) of real-world visual bug reports from 31 Android applications in automated evaluation.
• The keyframe extraction component achieved high accuracy in identifying fully rendered GUI states by detecting frame stability through similarity comparison.
• The GUI mapping module outperformed baseline methods in matching keyframes to UTG states, with improved robustness to visual noise and minor UI variations.
• The trace generation algorithm effectively reconstructed missing pre-bug steps, enabling full replay even when recordings started 2–7 steps before the bug occurred.
• A user study confirmed the tool's usefulness in real-world settings, with developers reporting reduced time and effort in reproducing bugs from visual reports.
• The approach demonstrated strong generalization across diverse apps and recording conditions, including low-resolution and non-instrumented recordings.