[Paper Review] Unit Test Generation using Generative AI : A Comparative Performance Analysis of Autogeneration Tools

Shreya Bhatia, Tarushi Gandhi+2 more|arXiv (Cornell University)|Dec 17, 2023

Energy Load and Power Forecasting4 citations

TL;DR

This study evaluates ChatGPT's effectiveness in generating unit tests for Python code, comparing it to the search-based tool Pynguin across procedural, function-based, and class-based code. Results show ChatGPT achieves comparable coverage to Pynguin, with prompt iteration improving performance, but up to one-third of its assertions are incorrect; minimal overlap in missed statements suggests potential synergy between the two tools for enhanced test coverage.

ABSTRACT

Generating unit tests is a crucial task in software development, demanding substantial time and effort from programmers. The advent of Large Language Models (LLMs) introduces a novel avenue for unit test script generation. This research aims to experimentally investigate the effectiveness of LLMs, specifically exemplified by ChatGPT, for generating unit test scripts for Python programs, and how the generated test cases compare with those generated by an existing unit test generator (Pynguin). For experiments, we consider three types of code units: 1) Procedural scripts, 2) Function-based modular code, and 3) Class-based code. The generated test cases are evaluated based on criteria such as coverage, correctness, and readability. Our results show that ChatGPT's performance is comparable with Pynguin in terms of coverage, though for some cases its performance is superior to Pynguin. We also find that about a third of assertions generated by ChatGPT for some categories were incorrect. Our results also show that there is minimal overlap in missed statements between ChatGPT and Pynguin, thus, suggesting that a combination of both tools may enhance unit test generation performance. Finally, in our experiments, prompt engineering improved ChatGPT's performance, achieving a much higher coverage.

Motivation & Objective

• To evaluate the effectiveness of large language models, specifically ChatGPT, in generating unit tests for Python programs.
• To compare ChatGPT’s performance against Pynguin, a state-of-the-art search-based unit test generator, in terms of test coverage, correctness, and readability.
• To investigate whether iterative prompting improves the quality and coverage of test cases generated by ChatGPT.
• To assess the correctness of assertions generated by ChatGPT and determine their alignment with intended code functionality.
• To explore the potential of combining LLM-based and search-based approaches to enhance overall unit test generation performance.

Proposed method

• The study uses a curated dataset of 60 Python projects, 20 per category: procedural scripts, function-based modular code, and class-based modular code.
• For each project, a core module of 100–300 lines is selected based on cyclomatic complexity, function count, and interdependency metrics.
• Unit tests are generated by prompting ChatGPT with the code module and comparing them to tests produced by Pynguin.
• Iterative prompting is applied to ChatGPT, where missed statements from prior iterations are fed back to improve coverage.
• Test quality is evaluated using metrics including statement coverage, correctness of assertions, and readability.
• A combined approach is evaluated by merging test suites from both ChatGPT and Pynguin to assess synergistic coverage gains.

Experimental results

Research questions

• RQ1 RQ1: How does ChatGPT compare with Pynguin in generating unit tests in terms of coverage, correctness, and readability?
• RQ2 RQ2: How does the effectiveness of test cases generated by ChatGPT improve over multiple iterations of prompting?
• RQ3 RQ3: How correct are the assertions generated by ChatGPT, and what percentage align with the intended functionality of the code?
• RQ4 RQ4: Can combining ChatGPT and Pynguin enhance overall unit test generation performance in terms of coverage and effectiveness?

Key findings

• ChatGPT achieved comparable statement coverage to Pynguin across all code categories, with no statistically significant difference in average coverage.
• Iterative prompting improved ChatGPT’s coverage in function-based and class-based code, reaching saturation after four iterations, but no improvement was observed in procedural scripts.
• Approximately one-third of assertions generated by ChatGPT were incorrect, with correctness improving from category 1 (procedural) to category 3 (class-based) code.
• There was minimal overlap in the statements missed by ChatGPT and Pynguin, indicating complementary coverage and potential for combined use to increase overall test coverage.
• The study confirms that combining LLM-based and search-based approaches can yield higher coverage, supporting the integration of both tools in test generation pipelines.
• ChatGPT’s focus on natural language understanding may lead to high coverage but inaccurate assertions, highlighting a need for semantic-aware assertion generation in future LLM-based testing.