[Paper Review] Deriving Semantics-Aware Fuzzers from Web API Schemas

Zac Hatfield-Dodds, Dmitry Dygalo|arXiv (Cornell University)|Dec 20, 2021

Software Testing and Debugging Techniques4 citations

TL;DR

This paper presents Schemathesis, a property-based fuzzer that automatically derives semantics-aware, structure-aware fuzzers from OpenAPI and GraphQL schemas using the Hypothesis testing library. It outperforms existing tools by handling 100% of real-world web services without fatal errors and discovering 1.4× to 4.5× more unique defects than the next-best fuzzer, while enabling customizable input generation and semantic validation.

ABSTRACT

Fuzzing -- whether generating or mutating inputs -- has found many bugs and security vulnerabilities in a wide range of domains. Stateful and highly structured web APIs present significant challenges to traditional fuzzing techniques, as execution feedback is usually limited to a response code instead of code coverage and vulnerabilities of interest include silent information-disclosure in addition to explicit errors. Our tool, Schemathesis, derives structure- and semantics-aware fuzzers from web API schemas in the OpenAPI or GraphQL formats, using property-based testing tools. Derived fuzzers can be incorporated into unit-test suites or run directly, with or without end-user customisation of data generation and semantic checks. We construct the most comprehensive evaluation of web API fuzzers to date, running eight fuzzers against sixteen real-world open source web services. OpenAPI schemas found in the wild have a long tail of rare features and complex structures. Of the tools we evaluated, Schemathesis was the only one to handle more than two-thirds of our target services without a fatal internal error. Schemathesis finds 1.4 times to 4.5 times more unique defects than the respectively second-best fuzzer for each target, and is the only fuzzer to find defects in four targets.

Motivation & Objective

• To develop a scalable, reusable method for generating custom, semantics-aware fuzzers from web API schemas.
• To address the limitations of traditional fuzzing in stateful, structured web APIs where feedback is limited to HTTP status codes.
• To improve defect detection by incorporating semantic checks such as response conformity, missing headers, and silent information disclosure.
• To evaluate and compare the effectiveness of web API fuzzers in real-world settings using a comprehensive benchmark suite.
• To provide a reusable, open-source evaluation corpus of real web services, schemas, and fuzzers for future research.

Proposed method

• Leverages the Hypothesis property-based testing library to generate structured, valid inputs based on OpenAPI or GraphQL schema definitions.
• Uses schema-defined types, constraints, and format annotations to generate semantically valid test inputs and request sequences.
• Integrates semantic validation checks—such as response body conformity, header presence, and status code correctness—into the fuzzer workflow.
• Supports in-process fuzzing via ASGI/WSGI integration to enable code coverage feedback and deeper instrumentation.
• Employs shrinking and deduplication techniques from Hypothesis to minimize and report unique, actionable defects.
• Allows user customization of data generation strategies and semantic checks through extensible test harnesses.

Figure 1. Sorting unique defects by the number of runs detecting them, we see that Schemathesis (black) is less consistent but discovers more than other fuzzers. Defect IDs are consistent between runs but not between fuzzers.

Experimental results

Research questions

• RQ1 Can property-based testing techniques be effectively adapted to derive semantics-aware fuzzers from web API schemas?
• RQ2 How does schema-aware fuzzer generation compare to traditional fuzzing in detecting real-world defects in production web services?
• RQ3 To what extent can a fuzzer handle the full diversity of real-world OpenAPI and GraphQL schemas, including rare or complex features?
• RQ4 How does the integration of semantic validation improve defect detection compared to purely syntactic or structural fuzzing?
• RQ5 What is the impact of property-based testing workflows on defect consistency and report quality in web API fuzzing?

Key findings

• Schemathesis is the only fuzzer in the evaluation to handle all 16 real-world web services without fatal internal errors, outperforming others on schema robustness.
• It discovered 1.4× to 4.5× more unique defects than the second-best fuzzer across all targets, with the only tool to find defects in four of the 16 services.
• The fuzzer reported a single minimal input per defect in most cases, with high report quality due to Hypothesis’s shrinking and deduplication capabilities.
• Schemathesis demonstrated higher consistency in defect detection than state-of-the-art tools, especially when run for longer durations.
• The evaluation corpus, including 16 real web services and 8 fuzzers, is open-sourced and reusable for future research.
• The study confirms that integrating semantic checks and schema-aware input generation significantly improves defect detection over syntactic or black-box approaches.