[Paper Review] Evaluating LLMs for Hardware Design and Test

Jason Blocklove, Siddharth Garg+2 more|arXiv (Cornell University)|Apr 23, 2024

VLSI and Analog Circuit Testing4 citations

TL;DR

This paper evaluates state-of-the-art LLMs—particularly ChatGPT-4 and ChatGPT-3.5—for end-to-end hardware design and test, using natural language prompts to generate both functional Verilog modules and testbenches. It demonstrates that while LLMs can produce working designs (verified via silicon tapeout on a Skywater 130nm process), they consistently underperform in testbench generation, requiring frequent human feedback to achieve correctness.

ABSTRACT

Large Language Models (LLMs) have demonstrated capabilities for producing code in Hardware Description Languages (HDLs). However, most of the focus remains on their abilities to write functional code, not test code. The hardware design process consists of both design and test, and so eschewing validation and verification leaves considerable potential benefit unexplored, given that a design and test framework may allow for progress towards full automation of the digital design pipeline. In this work, we perform one of the first studies exploring how a LLM can both design and test hardware modules from provided specifications. Using a suite of 8 representative benchmarks, we examined the capabilities and limitations of the state-of-the-art conversational LLMs when producing Verilog for functional and verification purposes. We taped out the benchmarks on a Skywater 130nm shuttle and received the functional chip.

Motivation & Objective

• To investigate whether LLMs can autonomously generate both functional hardware designs and comprehensive testbenches from natural language specifications.
• To evaluate the reliability and correctness of LLM-generated Verilog code and testbenches through simulation and post-silicon validation.
• To assess the role of human feedback in improving LLM performance for hardware verification tasks.
• To provide open-source benchmarks, toolchain scripts, and conversation logs for reproducibility and future research in LLM-aided EDA.
• To identify key failure modes and limitations of current LLMs in hardware design and verification workflows.

Proposed method

• Designed a suite of 8 representative hardware benchmarks (e.g., 4-to-1 MUX, 8-bit adder, FSMs, dice roller) with plain-language specifications.
• Used a feedback loop with LLMs: prompt for Verilog design and testbench, compile with iverilog, simulate, and if errors occur, provide tool feedback or human feedback.
• Applied three feedback levels: No Feedback Needed (NFN), Tool Feedback (TF), and Simple Human Feedback (SHF) for persistent errors.
• Evaluated models including ChatGPT-4, ChatGPT-3.5, HuggingChat, and Google Bard using consistent prompts across multiple runs.
• Taped out the best-performing designs via Tiny Tapeout 3, a 130nm open-source fabrication shuttle, to validate functionality in silicon.
• Collected and open-sourced all conversation logs, Verilog files, and scripts on Zenodo for reproducibility and benchmarking.

Experimental results

Research questions

• RQ1 Can LLMs generate correct and synthesizable Verilog code for hardware modules directly from natural language specifications?
• RQ2 To what extent can LLMs generate comprehensive and correct testbenches that fully verify functional correctness?
• RQ3 How effective are different feedback mechanisms (tool feedback, human feedback) in correcting LLM-generated errors in design and testbench code?
• RQ4 What are the most common failure modes in LLM-generated hardware code and testbenches, and how do they vary across models?
• RQ5 Can LLM-generated designs be successfully taped out and function correctly in silicon, confirming end-to-end pipeline viability?

Key findings

• ChatGPT-4 successfully generated 12 out of 24 conversations that passed simulation and were compliant with specifications, with 11 of those requiring only tool feedback.
• ChatGPT-4 struggled significantly with testbench generation, with 12 out of 24 conversations requiring human feedback, and many testbenches failing to properly verify state transitions in FSMs.
• ChatGPT-3.5 performed worse, with most conversations failing, and many non-compliant passes due to inconsistent or incorrect testbench behavior.
• The Dice Roller benchmarks from ChatGPT-4 produced non-random outputs (e.g., repeating 1s or small cycles), indicating testbench flaws despite correct design code.
• Post-silicon testing confirmed that all taped-out designs functioned as expected in hardware, matching simulation results and validating the entire pipeline.
• Non-deterministic output across LLM runs and lack of consistent training data for testbench patterns were identified as key limitations affecting reliability.