[Paper Review] CorrectBench: Automatic Testbench Generation with Functional Self-Correction using LLMs for HDL Design

Ruidi Qiu, Grace Li Zhang+3 more|arXiv (Cornell University)|Nov 13, 2024

Real-time simulation and control systems4 citations

TL;DR

CorrectBench is a novel LLM-based framework for automatic HDL testbench generation that integrates functional self-validation and self-correction to improve reliability. By validating testbenches against natural language specifications using a scenario-based validator and correcting errors via an LLM-based corrector, it achieves a 70.13% pass ratio—significantly outperforming prior methods at 52.18% and direct LLM generation at 33.33%.

ABSTRACT

Functional simulation is an essential step in digital hardware design. Recently, there has been a growing interest in leveraging Large Language Models (LLMs) for hardware testbench generation tasks. However, the inherent instability associated with LLMs often leads to functional errors in the generated testbenches. Previous methods do not incorporate automatic functional correction mechanisms without human intervention and still suffer from low success rates, especially for sequential tasks. To address this issue, we propose CorrectBench, an automatic testbench generation framework with functional self-validation and self-correction. Utilizing only the RTL specification in natural language, the proposed approach can validate the correctness of the generated testbenches with a success rate of 88.85%. Furthermore, the proposed LLM-based corrector employs bug information obtained during the self-validation process to perform functional self-correction on the generated testbenches. The comparative analysis demonstrates that our method achieves a pass ratio of 70.13% across all evaluated tasks, compared with the previous LLM-based testbench generation framework's 52.18% and a direct LLM-based generation method's 33.33%. Specifically in sequential circuits, our work's performance is 62.18% higher than previous work in sequential tasks and almost 5 times the pass ratio of the direct method. The codes and experimental results are open-sourced at the link: https://github.com/AutoBench/CorrectBench

Motivation & Objective

• Address the instability of LLMs in generating correct testbenches for hardware design, particularly in sequential circuits.
• Overcome the limitations of prior LLM-based testbench frameworks that lack functional validation and correction mechanisms.
• Enable fully automated testbench generation using only natural language RTL specifications as input.
• Improve pass rates in functional simulation by introducing a self-validation and self-correction loop driven by LLMs.
• Achieve higher reliability and scalability in testbench generation without human intervention.

Proposed method

• Utilizes natural language RTL specifications as the sole input for testbench generation.
• Employs a scenario-based self-validator that evaluates testbench correctness using a 70%-wrong threshold, achieving 88.85% validation accuracy.
• Introduces a functional self-validation mechanism that checks testbench outputs against expected behavior derived from the specification.
• Applies an LLM-based corrector that uses bug reports from the validator to iteratively refine and correct testbench code.
• Integrates the self-validation and self-correction loop into a feedback-driven workflow to enhance reliability.
• Employs a hybrid testbench structure combining Verilog drivers and Python checkers for end-to-end functional verification.

Experimental results

Research questions

• RQ1 Can functional self-validation significantly improve the reliability of LLM-generated testbenches in HDL design?
• RQ2 How effective is an LLM-based self-correction mechanism in reducing functional errors in automatically generated testbenches?
• RQ3 What validation threshold (e.g., 50%-wrong, 70%-wrong) maximizes the accuracy and convergence of testbench validation?
• RQ4 To what extent does the CorrectBench framework outperform existing LLM-based testbench generation methods in both combinational and sequential circuits?
• RQ5 Can the framework be generalized across different LLMs without sacrificing performance or requiring manual tuning?

Key findings

• CorrectBench achieves a 70.13% pass ratio across all 156 evaluated Verilog tasks, representing a 17.95 percentage point improvement over the previous state-of-the-art AutoBench framework (52.18%).
• In sequential circuits, CorrectBench outperforms AutoBench by 62.18 percentage points and achieves nearly five times the pass rate of direct LLM-based generation (33.33%).
• The scenario-based validator achieves 88.85% global validation accuracy using the 70%-wrong criterion, which was selected as optimal due to its balance between precision and recall.
• The LLM-based corrector contributes 34.33% to the total performance improvement over the previous method, demonstrating its critical role in error correction.
• The framework maintains consistent performance across different LLMs, including GPT-4o, GPT-4o-mini, and Claude-3.5-Sonnet, indicating generalizability.
• The 70%-wrong validation threshold outperforms 50%-wrong and 90%-wrong in both validation accuracy and overall system performance, with lower token costs and higher convergence rates.