[論文レビュー] On the Challenges of Fuzzing Techniques via Large Language Models

Long Huang, Peizhou Zhao+2 more|arXiv (Cornell University)|Feb 1, 2024

Speech and dialogue systems被引用数 5

ひとこと要約

A systematic survey of how large language models are integrated with fuzzing to improve automation, coverage, and vulnerability discovery across AI and non-AI software, up to 2024.

ABSTRACT

In the modern era where software plays a pivotal role, software security and vulnerability analysis are essential for secure software development. Fuzzing test, as an efficient and traditional software testing method, has been widely adopted across various domains. Meanwhile, the rapid development in Large Language Models (LLMs) has facilitated their application in the field of software testing, demonstrating remarkable performance. As existing fuzzing test techniques are not fully automated and software vulnerabilities continue to evolve, there is a growing interest in leveraging large language models to generate fuzzing test. In this paper, we present a systematic overview of the developments that utilize large language models for the fuzzing test. To our best knowledge, this is the first work that covers the intersection of three areas, including LLMs, fuzzing test, and fuzzing test generated based on LLMs. A statistical analysis and discussion of the literature are conducted by summarizing the state-of-the-art methods up to date of the submission. Our work also investigates the potential for widespread deployment and application of fuzzing test techniques generated by LLMs in the future, highlighting their promise for advancing automated software testing practices.

研究の動機と目的

• Assess how LLMs are used to enhance fuzzing for AI and non-AI software systems.
• Identify advantages of LLM-based fuzzers over traditional fuzzers.
• Summarize methods, benchmarks, and evaluation metrics used in LLMs-based fuzzing.
• Discuss future research directions, data needs, and automation challenges.

提案手法

• Review and categorize LLM-based fuzzing approaches across AI and non-AI software.
• Analyze prompt engineering and seed mutation as core techniques.
• Compare performance against traditional fuzzers using metrics like code coverage and bug counts.
• Provide a consolidated taxonomy of models, benchmarks, and evaluation practices.
• Propose an agenda for standardized evaluation frameworks and automation.

実験結果

リサーチクエスチョン

• RQ1 How are LLMs employed in fuzzing for AI and non-AI software systems?
• RQ2 What advantages do LLM-based fuzzers offer over traditional fuzzers?
• RQ3 What are the future potentials and challenges for LLM-based fuzzing research?

主な発見

• LLMs are used to enhance prompt engineering and seed mutation to improve fuzzing effectiveness.
• LLM-based fuzzers show higher API and code coverage in examples like TitanFuzz and CHATAFL compared to some traditional or non-LLM fuzzers.
• Fuzzers based on LLMs detect more complex bugs and enable greater automation, albeit with higher time costs.
• Multiple categories of metrics (code, performance, time) are used to evaluate LLM-based fuzzers, with no universal framework yet.
• There is a trend toward two fuzzer types: learning from historical data to train specialized fuzzers, and integrating LLMs into specific fuzzing steps.