[Paper Review] Prompting Code Interpreter to Write Better Unit Tests on Quixbugs Functions

V.O.K. Li, Nick Doiron|arXiv (Cornell University)|Sep 30, 2023

Software Engineering Research4 citations

TL;DR

This study investigates how different prompting strategies affect the quality of unit tests generated by Code Interpreter, a GPT-4-based model, on functions from the Quixbugs dataset. Despite variations in prompt format, examples, or comments, test quality remained largely unchanged; however, the model frequently self-corrected its own code, suggesting that enabling executable feedback loops significantly improves reliability and output correctness.

ABSTRACT

Unit testing is a commonly-used approach in software engineering to test the correctness and robustness of written code. Unit tests are tests designed to test small components of a codebase in isolation, such as an individual function or method. Although unit tests have historically been written by human programmers, recent advancements in AI, particularly LLMs, have shown corresponding advances in automatic unit test generation. In this study, we explore the effect of different prompts on the quality of unit tests generated by Code Interpreter, a GPT-4-based LLM, on Python functions provided by the Quixbugs dataset, and we focus on prompting due to the ease with which users can make use of our findings and observations. We find that the quality of the generated unit tests is not sensitive to changes in minor details in the prompts provided. However, we observe that Code Interpreter is often able to effectively identify and correct mistakes in code that it writes, suggesting that providing it runnable code to check the correctness of its outputs would be beneficial, even though we find that it is already often able to generate correctly-formatted unit tests. Our findings suggest that, when prompting models similar to Code Interpreter, it is important to include the basic information necessary to generate unit tests, but minor details are not as important.

Motivation & Objective

• To evaluate how prompt variations impact the quality of unit tests generated by Code Interpreter on real-world Python functions.
• To determine whether minor prompt modifications—such as formatting, examples, or role instructions—affect test correctness and completeness.
• To assess the model’s ability to detect and correct errors in its own generated test code.
• To identify best practices for prompting LLMs in unit test generation that are both effective and practical for developers.
• To explore the feasibility of using executable feedback to improve test generation quality without iterative prompting.

Proposed method

• The study uses Code Interpreter, a GPT-4-based model, to generate unit tests for functions from the Quixbugs dataset, which are designed for automated testing.
• Prompts were systematically varied in format: code context was presented either as raw code or natural language descriptions, with or without formatting examples and role-based comments.
• Generated unit tests were executed in a controlled environment to evaluate correctness, compilation success, and assertion accuracy.
• The model’s ability to self-correct was assessed by running its own generated code and analyzing whether errors were detected and fixed in subsequent generations.
• A subset of functions was selected based on compatibility with a simple input/output testing framework to ensure executable evaluation.
• Multiple generations per prompt were used to increase the likelihood of obtaining a correct, compilable test suite.

Experimental results

Research questions

• RQ1 How do variations in prompt format—such as code vs. natural language context—impact the quality of generated unit tests?
• RQ2 Does including formatting examples or role-based comments (e.g., 'act as an expert coder') significantly improve test quality?
• RQ3 To what extent can Code Interpreter detect and correct errors in its own generated unit tests?
• RQ4 How does the inclusion of executable feedback (i.e., running the code) affect the reliability of test generation compared to static prompting?
• RQ5 What are the most effective prompting strategies for generating correct, compilable, and comprehensive unit tests using Code Interpreter?

Key findings

• The quality of unit tests generated by Code Interpreter is largely insensitive to minor prompt variations, such as formatting style, inclusion of examples, or role-based comments.
• Despite minimal prompt variation effects, Code Interpreter frequently self-corrects its own code, indicating strong internal consistency and error-detection capability.
• The model is already highly effective at generating syntactically correct and properly formatted unit tests without requiring additional prompting for structure.
• Enabling the model to execute its own output significantly improves reliability, suggesting that integrating runnable feedback loops enhances test quality.
• The model’s ability to generate correct tests is robust even when provided with minimal or ambiguous context, indicating strong generalization from training data.
• The study highlights that future improvements should focus on feedback mechanisms rather than prompt engineering for minor formatting details.