[Paper Review] A Survey of Modern Compiler Fuzzing

Haoyang Ma|arXiv (Cornell University)|Jun 12, 2023

Software Testing and Debugging Techniques4 citations

TL;DR

This survey provides a comprehensive analysis of modern compiler fuzzing techniques, focusing on test program generation and test oracle design to detect bugs in compilers. It synthesizes 58 studies on mainstream and domain-specific compilers, identifying key challenges and future research directions in validity, diversity, extensibility, and evaluation of fuzzing tools.

ABSTRACT

Most software that runs on computers undergoes processing by compilers. Since compilers constitute the fundamental infrastructure of software development, their correctness is paramount. Over the years, researchers have invested in analyzing, understanding, and characterizing the bug features over mainstream compilers. These studies have demonstrated that compilers correctness requires greater research attention, and they also pave the way for compiler fuzzing. To improve compilers correctness, researchers have proposed numerous compiler fuzzing techniques. These techniques were initially developed for testing traditional compilers such as GCC/LLVM and have since been generalized to test various newly developed, domain-specific compilers, such as graphics shader compilers and deep learning (DL) compilers. In this survey, we provide a comprehensive summary of the research efforts for understanding and addressing compilers defects. Specifically, this survey mainly covers two aspects. First, it covers researchers investigation and expertise on compilers bugs, such as their symptoms and root causes. The compiler bug studies cover GCC/LLVM, JVM compilers, and DL compilers. In addition, it covers researchers efforts in designing fuzzing techniques, including constructing test programs and designing test oracles. Besides discussing the existing work, this survey outlines several open challenges and highlights research opportunities.

Motivation & Objective

• To systematically analyze the state of the art in compiler fuzzing for modern and domain-specific compilers.
• To identify core challenges in generating valid and diverse test programs that trigger latent compiler bugs.
• To examine innovative test oracle designs that detect subtle semantic bugs in compiled binaries.
• To highlight open issues in extensibility, generalizability, and evaluation frameworks for compiler fuzzing tools.
• To guide researchers and practitioners by outlining future research opportunities in compiler correctness and fuzzing methodology.

Proposed method

• Surveying 58 high-quality research papers on compiler fuzzing, focusing on GCC, LLVM, JVM, and domain-specific compilers like DL and graphics shader compilers.
• Categorizing and analyzing test program generation techniques, including grammar-based generation (e.g., Csmith), mutation-based approaches (e.g., EMI, Athena), and LLM-assisted synthesis.
• Evaluating test oracle mechanisms that detect semantic discrepancies between source and compiled code, including coverage-guided and semantics-aware oracles.
• Assessing diversity metrics such as code coverage and feature-based coverage to quantify test program variety and effectiveness.
• Examining extensibility challenges in existing tools, including hard-coded grammars and lack of modular design for cross-compiler portability.
• Reviewing evaluation practices and advocating for standardized benchmarks and consensus evaluation protocols to enable fair comparison of new fuzzing techniques.

Experimental results

Research questions

• RQ1 What are the primary challenges in generating valid and diverse test programs for modern compilers?
• RQ2 How do existing test oracles detect subtle, semantics-preserving bugs in compiled binaries?
• RQ3 What are the limitations in the extensibility and generalizability of current compiler fuzzing tools across different compiler components and languages?
• RQ4 How can evaluation frameworks for compiler fuzzing be standardized to ensure reproducibility and fair comparison?
• RQ5 What role can large language models play in reducing manual effort for grammar learning and test program synthesis in compiler fuzzing?

Key findings

• Compiler fuzzing is effective in uncovering critical bugs, including those causing assertion failures, segmentation faults, and miscompilation of well-formed programs.
• Many compiler bugs stem from undefined behaviors in languages like C, and fuzzing tools that avoid such constructs achieve higher reliability in test generation.
• Coverage-guided testing and feature-based diversity metrics are effective in increasing the likelihood of triggering latent compiler bugs.
• Mutation-based tools like Athena and EMI significantly outperform earlier approaches in diversity and bug-finding capability, especially when enhanced with feedback-driven strategies.
• Current tools suffer from poor extensibility due to hard-coded grammars and lack of modular design, making porting to new compilers time-consuming.
• While LLMs show promise in generating valid programs and learning language grammars, their ability to produce complex, type-safe, and semantically correct programs remains limited and requires further research.