[Paper Review] Revisiting VerilogEval: A Year of Improvements in Large-Language Models for Hardware Code Generation

Nathaniel Pinckney, Christopher Batten+3 more|arXiv (Cornell University)|Aug 20, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper enhances the VerilogEval benchmark to support specification-to-RTL tasks, in-context learning (ICL) examples, and detailed failure classification. It evaluates newer LLMs—including GPT-4 Turbo and Llama 3.1 405B—showcasing a 59% pass rate on specification-to-RTL tasks, with performance heavily dependent on prompt engineering and model-specific tuning.

ABSTRACT

The application of large-language models (LLMs) to digital hardware code generation is an emerging field, with most LLMs primarily trained on natural language and software code. Hardware code like Verilog constitutes a small portion of training data, and few hardware benchmarks exist. The open-source VerilogEval benchmark, released in November 2023, provided a consistent evaluation framework for LLMs on code completion tasks. Since then, both commercial and open models have seen significant development. In this work, we evaluate new commercial and open models since VerilogEval's original release-including GPT-4o, GPT-4 Turbo, Llama3.1 (8B/70B/405B), Llama3 70B, Mistral Large, DeepSeek Coder (33B and 6.7B), CodeGemma 7B, and RTL-Coder-against an improved VerilogEval benchmark suite. We find measurable improvements in state-of-the-art models: GPT-4o achieves a 63% pass rate on specification-to-RTL tasks. The recently released and open Llama3.1 405B achieves a 58% pass rate, almost matching GPT-4o, while the smaller domain-specific RTL-Coder 6.7B models achieve an impressive 34% pass rate. Additionally, we enhance VerilogEval's infrastructure by automatically classifying failures, introducing in-context learning support, and extending the tasks to specification-to-RTL translation. We find that prompt engineering remains crucial for achieving good pass rates and varies widely with model and task. A benchmark infrastructure that allows for prompt engineering and failure analysis is essential for continued model development and deployment.

Motivation & Objective

• To address limitations in existing Verilog code generation benchmarks, such as lack of in-context learning (ICL) support and failure analysis.
• To extend the VerilogEval benchmark to include specification-to-RTL translation tasks, aligning with instruction-tuned LLMs.
• To evaluate the performance of newer commercial and open-source LLMs, including GPT-4 Turbo and Llama 3.1, on both code completion and specification-to-RTL tasks.
• To introduce a failure classification system that enables granular analysis of compiler and runtime errors across models and prompting strategies.
• To demonstrate that prompt engineering significantly impacts model performance, with varying effects across different models and tasks.

Proposed method

• Extended the original VerilogEval benchmark to support specification-to-RTL prompting using a chat-style format with [BEGIN] and [DONE] tags, similar to MBPP.
• Integrated in-context learning (ICL) by injecting up to four example problem-solution pairs into the prompt for both code completion and specification-to-RTL tasks.
• Developed a Makefile-based evaluation environment to decouple dataset management from evaluation logic, enabling scalable and reproducible testing across models.
• Implemented a failure classification pipeline that categorizes model outputs into compiler errors, runtime errors, and logic bugs, with visual feedback via color-coding.
• Automated failure classification using static and dynamic analysis to report pass/fail rates and error type distributions across models and prompting configurations.
• Released the enhanced benchmark publicly at https://github.com/NVlabs/verilog-eval to support ongoing research and model development.

(a) Code Completion Task

Experimental results

Research questions

• RQ1 How do newer large language models, including GPT-4 Turbo and Llama 3.1, perform on specification-to-RTL hardware code generation compared to older models?
• RQ2 To what extent does in-context learning (ICL) improve pass rates across different LLMs and tasks, and how does its impact vary by model architecture?
• RQ3 What types of errors (e.g., compiler, runtime, logic) are most common in LLM-generated Verilog, and how do they vary across models and prompting strategies?
• RQ4 Can open-source and domain-specific LLMs achieve performance comparable to closed-source state-of-the-art models in hardware code generation?
• RQ5 How does prompt engineering influence model behavior, and what role does failure classification play in guiding effective prompt tuning?

Key findings

• GPT-4 Turbo achieved a 59% pass rate on specification-to-RTL tasks, significantly outperforming previous models and demonstrating strong generalization on instruction-tuned prompts.
• Llama 3.1 405B achieved a 58% pass rate on specification-to-RTL tasks, effectively matching GPT-4 Turbo and showing that open-source models are now competitive with commercial counterparts.
• The RTL-Coder 6.7B model achieved a 37% pass rate on specification-to-RTL tasks, demonstrating strong performance for a small, domain-specific model.
• In-context learning (ICL) improved performance for GPT-4 Turbo and Llama 3.1 405B on code completion, but had mixed effects on other models, with some showing degradation due to prompt-induced errors.
• Failure classification revealed that compiler errors were the primary failure mode in 0-shot settings, especially for Llama 3.1 70B, which frequently missed endmodule statements, indicating a need for targeted prompt tuning.
• Performance varied significantly across models and tasks, with Llama 3.1 70B showing degraded performance on code completion with ICL due to endmodule errors, while GPT-4 Turbo saw a slight performance drop in specification-to-RTL tasks with more ICL examples.

(b) Specification-to-RTL Task