[Paper Review] Predicting Code Coverage without Execution

Michele Tufano, Shubham Chandel+3 more|arXiv (Cornell University)|Jul 25, 2023

Software Engineering Research4 citations

TL;DR

This paper introduces Code Coverage Prediction as a novel benchmark task to evaluate Large Language Models' (LLMs) understanding of code execution. By curating the CoverageEval dataset from HumanEval with actual code coverage annotations, the authors evaluate state-of-the-art LLMs—GPT-4, GPT-3.5, BARD, and Claude—showing that even GPT-4 achieves only 24.48% accuracy with few-shot prompting, indicating significant room for improvement in LLMs' comprehension of code execution dynamics.

ABSTRACT

Code coverage is a widely used metric for quantifying the extent to which program elements, such as statements or branches, are executed during testing. Calculating code coverage is resource-intensive, requiring code building and execution with additional overhead for the instrumentation. Furthermore, computing coverage of any snippet of code requires the whole program context. Using Machine Learning to amortize this expensive process could lower the cost of code coverage by requiring only the source code context, and the task of code coverage prediction can be a novel benchmark for judging the ability of models to understand code. We propose a novel benchmark task called Code Coverage Prediction for Large Language Models (LLMs). We formalize this task to evaluate the capability of LLMs in understanding code execution by determining which lines of a method are executed by a given test case and inputs. We curate and release a dataset we call COVERAGEEVAL by executing tests and code from the HumanEval dataset and collecting code coverage information. We report the performance of four state-of-the-art LLMs used for code-related tasks, including OpenAI's GPT-4 and GPT-3.5-Turbo, Google's BARD, and Anthropic's Claude, on the Code Coverage Prediction task. Finally, we argue that code coverage as a metric and pre-training data source are valuable for overall LLM performance on software engineering tasks.

Motivation & Objective

• To formalize Code Coverage Prediction as a new benchmark task to assess LLMs' understanding of code execution.
• To address the high computational cost and context dependency of traditional code coverage measurement, especially for code snippets or partial programs.
• To evaluate whether LLMs can predict which code lines are executed by a given test case without actual execution.
• To provide a new pre-training or fine-tuning objective that enhances LLMs' ability to reason about code behavior under different inputs.
• To enable practical applications in scenarios with expensive builds, limited code availability, or real-time feedback in IDEs.

Proposed method

• Curated the CoverageEval dataset by executing test cases from the HumanEval benchmark and collecting actual code coverage using instrumentation.
• Annotated each method-test pair with coverage labels: executed (‘>’), not executed (‘!’), or unreachable (‘-’), creating a structured prediction task.
• Formalized the task as a sequence-to-sequence prediction problem where the input is a method and test case, and the output is the coverage-annotated method.
• Evaluated four SOTA LLMs—GPT-4, GPT-3.5-Turbo, BARD, and Claude—using zero-shot and few-shot prompting on the coverage prediction task.
• Used exact match accuracy as the primary evaluation metric, measuring how often the model predicted the full coverage annotation correctly.
• Explored use cases such as expensive builds, limited code availability, and live unit testing to demonstrate the practical utility of prediction-based coverage.

Experimental results

Research questions

• RQ1 Can LLMs accurately predict which lines of code are executed by a given test case without running the code?
• RQ2 How does the performance of state-of-the-art LLMs vary in predicting code coverage across different few-shot prompting strategies?
• RQ3 To what extent do LLMs understand the dynamic behavior of code under different inputs and test conditions?
• RQ4 Can code coverage prediction serve as a meaningful pre-training or fine-tuning objective to improve LLMs’ reasoning about code execution?
• RQ5 In what real-world software engineering scenarios can LLM-based code coverage prediction provide practical advantages over traditional execution-based methods?

Key findings

• GPT-4 achieved the highest performance among the evaluated models, reaching 10.46% exact match accuracy with zero-shot prompting and 24.48% with few-shot prompting.
• No model achieved high accuracy—even GPT-4 predicted the correct coverage for less than a quarter of the method-test pairs, indicating limited understanding of code execution.
• The performance gap between models highlights that current LLMs still struggle with reasoning about control flow and conditional execution in code.
• The CoverageEval dataset enables systematic evaluation of LLMs on code execution understanding, offering a new benchmark for future research.
• The task has practical value in scenarios like expensive builds, partial code transmission, and live unit testing, where execution is infeasible or costly.
• Code coverage prediction can serve as a valuable training objective to improve LLMs’ ability to generate and test code effectively.