[Paper Review] Automatic Web Testing using Curiosity-Driven Reinforcement Learning

Yan Zheng, Yi Liu+5 more|arXiv (Cornell University)|Mar 10, 2021

Software Testing and Debugging Techniques52 references4 citations

TL;DR

This paper proposes WebExplor, an end-to-end reinforcement learning framework for automatic web testing that uses curiosity-driven exploration to generate high-coverage, behavior-diverse test cases. By integrating a dynamically learned deterministic finite automaton (DFA) for high-level guidance, WebExplor achieves superior code coverage and failure detection—outperforming state-of-the-art techniques and uncovering 12 unknown bugs in a commercial SaaS application.

ABSTRACT

Web testing has long been recognized as a notoriously difficult task. Even nowadays, web testing still heavily relies on manual efforts while automated web testing is far from achieving human-level performance. Key challenges in web testing include dynamic content update and deep bugs hiding under complicated user interactions and specific input values, which can only be triggered by certain action sequences in the huge search space. In this paper, we propose WebExplor, an automatic end-to-end web testing framework, to achieve an adaptive exploration of web applications. WebExplor adopts curiosity-driven reinforcement learning to generate high-quality action sequences (test cases) satisfying temporal logical relations. Besides, WebExplor incrementally builds an automaton during the online testing process, which provides high-level guidance to further improve the testing efficiency. We have conducted comprehensive evaluations of WebExplor on six real-world projects, a commercial SaaS web application, and performed an in-the-wild study of the top 50 web applications in the world. The results demonstrate that in most cases WebExplor can achieve a significantly higher failure detection rate, code coverage, and efficiency than existing state-of-the-art web testing techniques. WebExplor also detected 12 previously unknown failures in the commercial web application, which have been confirmed and fixed by the developers. Furthermore, our in-the-wild study further uncovered 3,466 exceptions and errors.

Motivation & Objective

• Address the challenge of inefficient and unbalanced exploration in automated web testing due to the vast action space and dynamic content.
• Overcome limitations of model-based approaches that rely on static navigation models and require manual domain knowledge.
• Enable end-to-end, adaptive test case generation without requiring explicit models or predefined test objectives.
• Improve exploration efficiency and coverage in complex web applications with dynamic behavior and deep logical dependencies.

Proposed method

• Employ curiosity-driven reinforcement learning (CDRL) to encourage exploration of novel, unvisited states by maximizing prediction error in a world model.
• Design a custom state representation combining DOM structure, UI element properties, and action history to capture web application semantics.
• Integrate a curiosity-based intrinsic reward to guide agents toward exploring new and potentially buggy behaviors.
• Incrementally learn a deterministic finite automaton (DFA) from executed action sequences to model valid control flows and guide exploration.
• Use the DFA as a high-level constraint to prioritize paths that are semantically valid and more likely to trigger complex logic.
• Combine intrinsic curiosity reward with extrinsic rewards (e.g., code coverage, crash detection) to balance novelty and functional correctness.

Experimental results

Research questions

• RQ1 Can curiosity-driven reinforcement learning effectively explore complex web applications with large and dynamic action spaces?
• RQ2 How does integrating a learned DFA improve exploration efficiency and test coverage compared to curiosity-only RL?
• RQ3 To what extent can WebExplor outperform existing state-of-the-art automated web testing tools in failure detection and code coverage?
• RQ4 Can WebExplor discover previously unknown bugs in real-world, production-grade web applications?

Key findings

• WebExplor achieved significantly higher code coverage than all baseline methods across six real-world web applications and a commercial SaaS platform.
• The framework detected 12 previously unknown failures in a commercial SaaS web application, all confirmed and fixed by the development team.
• In an in-the-wild study of the top 50 global web applications, WebExplor uncovered 3,466 exceptions and errors, demonstrating broad real-world effectiveness.
• The integration of DFA-guided exploration led to a measurable improvement in test case quality and exploration efficiency compared to curiosity-only RL.
• WebExplor outperformed existing model-free and model-based web testing techniques in both failure detection rate and execution efficiency.
• The framework demonstrated strong adaptability to dynamic content and complex interaction patterns, including form submissions requiring multi-step action sequences.