[Paper Review] The Agent Web Model -- Modelling web hacking for reinforcement learning

László Erdődi, Fabio Massimo Zennaro|arXiv (Cornell University)|Sep 23, 2020

Advanced Malware Detection Techniques23 references4 citations

TL;DR

This paper introduces the Agent Web Model, a formal framework that models web penetration testing as a reinforcement learning (RL) problem across seven abstraction levels, from simple to complex web hacking challenges. It provides an OpenAI Gym-compliant implementation for the first three levels, enabling standardized training of RL agents for ethical red-teaming, with a focus on capturing flags through vulnerability exploitation in a safe, controlled environment.

ABSTRACT

Website hacking is a frequent attack type used by malicious actors to obtain confidential information, modify the integrity of web pages or make websites unavailable. The tools used by attackers are becoming more and more automated and sophisticated, and malicious machine learning agents seems to be the next development in this line. In order to provide ethical hackers with similar tools, and to understand the impact and the limitations of artificial agents, we present in this paper a model that formalizes web hacking tasks for reinforcement learning agents. Our model, named Agent Web Model, considers web hacking as a capture-the-flag style challenge, and it defines reinforcement learning problems at seven different levels of abstraction. We discuss the complexity of these problems in terms of actions and states an agent has to deal with, and we show that such a model allows to represent most of the relevant web vulnerabilities. Aware that the driver of advances in reinforcement learning is the availability of standardized challenges, we provide an implementation for the first three abstraction layers, in the hope that the community would consider these challenges in order to develop intelligent web hacking agents.

Motivation & Objective

• To formalize web penetration testing as a reinforcement learning problem to enable training of intelligent agents for ethical hacking.
• To address the lack of standardized, scalable challenges for training RL agents in web security.
• To develop a hierarchical abstraction model (7 levels) that captures increasing complexity in web vulnerabilities and attack surfaces.
• To provide a reusable, customizable, and ethical benchmark for training RL agents on realistic web hacking tasks.
• To promote collaboration between machine learning and cybersecurity researchers by aligning CTF-style challenges with RL formalism.

Proposed method

• The Agent Web Model formalizes web hacking as a Markov Decision Process (MDP), with states representing the website's configuration and actions representing HTTP requests.
• It defines seven abstraction levels, each increasing in complexity by adding more objects (e.g., files, parameters, sessions), actions, and attack surface elements.
• Each level models a specific class of vulnerabilities—such as SQL injection, XSS, file inclusion, and privilege escalation—within a CTF-style game framework.
• The model uses a state transition mechanism where each agent action (HTTP request) results in a new state and a reward signal based on flag capture or progress.
• The first three abstraction levels are implemented using the OpenAI Gym interface, enabling integration with existing RL algorithms and training pipelines.
• Challenges are parametrically configurable, allowing for the generation of diverse, randomized web hacking scenarios for robust agent training.

Experimental results

Research questions

• RQ1 How can web penetration testing be systematically formalized as a reinforcement learning problem across multiple abstraction levels?
• RQ2 What is the impact of increasing abstraction levels on the complexity of state space, action space, and learning difficulty for RL agents?
• RQ3 Can a standardized, customizable, and safe environment be created to train RL agents on realistic web vulnerabilities without real-world risk?
• RQ4 How do different vulnerability types (e.g., SQLi, XSS, RCE) map to distinct RL problem structures within the Agent Web Model?
• RQ5 To what extent can the proposed framework serve as a benchmark for advancing both RL and cybersecurity research?

Key findings

• The Agent Web Model successfully formalizes web hacking as a hierarchical reinforcement learning problem across seven abstraction levels, with increasing complexity in state and action spaces.
• The first three levels of the model have been implemented using the OpenAI Gym standard, enabling direct integration with existing RL training pipelines and algorithms.
• The implementation supports parametric configuration, allowing for the generation of a wide variety of randomized, yet consistent, web hacking challenges for agent training.
• The model captures a broad range of real-world vulnerabilities, including SQL injection, XSS, file inclusion, and privilege escalation, through structured state and action definitions.
• The framework provides a safe, ethical, and standardized environment for training RL agents on web security tasks, with a focus on proof-of-concept flag capture rather than destructive exploitation.
• The authors demonstrate that the model enables the creation of non-trivial learning challenges suitable for advancing RL research in cybersecurity, particularly in automated penetration testing.