[Paper Review] Prompting Is All You Need: Automated Android Bug Replay with Large Language Models

Sidong Feng, Chunyang Chen|arXiv (Cornell University)|Jun 3, 2023

Software Engineering ResearchComputer Science10 citations

TL;DR

The paper presents AdbGPT, a lightweight, prompt-engineered approach using LLMs to extract S2R entities from bug reports and guide automated Android bug replay, achieving high accuracy and efficiency without training.

ABSTRACT

Bug reports are vital for software maintenance that allow users to inform developers of the problems encountered while using the software. As such, researchers have committed considerable resources toward automating bug replay to expedite the process of software maintenance. Nonetheless, the success of current automated approaches is largely dictated by the characteristics and quality of bug reports, as they are constrained by the limitations of manually-crafted patterns and pre-defined vocabulary lists. Inspired by the success of Large Language Models (LLMs) in natural language understanding, we propose AdbGPT, a new lightweight approach to automatically reproduce the bugs from bug reports through prompt engineering, without any training and hard-coding effort. AdbGPT leverages few-shot learning and chain-of-thought reasoning to elicit human knowledge and logical reasoning from LLMs to accomplish the bug replay in a manner similar to a developer. Our evaluations demonstrate the effectiveness and efficiency of our AdbGPT to reproduce 81.3% of bug reports in 253.6 seconds, outperforming the state-of-the-art baselines and ablation studies. We also conduct a small-scale user study to confirm the usefulness of AdbGPT in enhancing developers' bug replay capabilities.

Motivation & Objective

• Motivate automated bug replay by leveraging large language models to handle unclear and incomplete bug reports.
• Propose a two-phase approach (S2R entity extraction and guided replay) that requires no model training or hard-coding.
• Show that LLM-based prompting can outperform state-of-the-art baselines in S2R extraction and bug replay efficiency.
• Demonstrate, via a user study, the practical usefulness of AdbGPT for developers.

Proposed method

• Define S2R entity specifications and action primitives to anchor the extraction task.
• Employ few-shot learning with chain-of-thought reasoning to elicit structured S2R entities from bug reports.
• Encode GUI state as HTML-like prompts to enable LLMs to identify target components.
• Use chain-of-thought guided replay to map S2R entities to GUI components across dynamic screens.
• Integrate LLM prompts with Genymotion, UIAutomator, and ADB for end-to-end automated replay.

Figure 1. The process of prompt engineering.

Experimental results

Research questions

• RQ1 RQ1: How accurately can AdbGPT extract S2R entities from bug reports?
• RQ2 RQ2: How accurately can AdbGPT guide and reproduce bugs via GUI interactions?
• RQ3 RQ3: How efficient is AdbGPT compared with baselines in bug replay time?
• RQ4 RQ4: Is AdbGPT perceived as useful by developers for real-world bug replay?

Key findings

• AdbGPT achieves 90.4% step extraction accuracy and 91.1% entity extraction accuracy in the S2R task.
• AdbGPT reproduces 81.3% of bugs and saves 1105.17 seconds per bug on average compared with baselines.
• Prompt engineering with few-shot and chain-of-thought significantly improves performance over zero-shot and non-CoT variants.
• GUI encoding into HTML-like prompts enables LLMs to reason about dynamic Android screens for guided replay.
• A small user study indicates AdbGPT’s usefulness in supporting developers’ bug replay workflows.

Figure 2. The overview of AdbGPT .