[Paper Review] Execution-based Evaluation for Data Science Code Generation Models

Junjie Huang, Chenglong Wang+7 more|arXiv (Cornell University)|Nov 17, 2022

Software Engineering Research4 citations

TL;DR

This paper introduces ExeDS, a new evaluation dataset of 534 data science problems from Jupyter Notebooks that enables execution-based evaluation of code generation models. It demonstrates that models with high surface-form scores (e.g., BLEU, CodeBLEU) often fail on execution correctness, proving execution-based metrics are more reliable for assessing functional code quality in data science tasks.

ABSTRACT

Code generation models can benefit data scientists' productivity by automatically generating code from context and text descriptions. An important measure of the modeling progress is whether a model can generate code that can correctly execute to solve the task. However, due to the lack of an evaluation dataset that directly supports execution-based model evaluation, existing work relies on code surface form similarity metrics (e.g., BLEU, CodeBLEU) for model selection, which can be inaccurate. To remedy this, we introduce ExeDS, an evaluation dataset for execution evaluation for data science code generation tasks. ExeDS contains a set of 534 problems from Jupyter Notebooks, each consisting of code context, task description, reference program, and the desired execution output. With ExeDS, we evaluate the execution performance of five state-of-the-art code generation models that have achieved high surface-form evaluation scores. Our experiments show that models with high surface-form scores do not necessarily perform well on execution metrics, and execution-based metrics can better capture model code generation errors. Source code and data can be found at https://github.com/Jun-jie-Huang/ExeDS

Motivation & Objective

• Address the lack of execution-based evaluation for data science code generation models.
• Identify the shortcomings of surface-form metrics like BLEU and CodeBLEU in capturing functional correctness.
• Provide a high-quality, executable benchmark dataset (ExeDS) with code context, natural language descriptions, reference code, and expected outputs.
• Evaluate state-of-the-art code generation models using execution-based metrics to reveal discrepancies between surface-form and functional performance.
• Demonstrate that execution-based evaluation can detect errors missed by surface-form metrics, guiding future model development.

Proposed method

• Crawled and reconstructed data dependencies from GitHub repositories to recover input data for Jupyter Notebook cells.
• Filtered out notebooks with runtime errors and curated 534 high-quality problems with sufficient context and human-annotated natural language descriptions.
• Collected reference code and desired execution outputs (including complex structures like dataframes and plots) for each problem.
• Evaluated five state-of-the-art code generation models on ExeDS using execution-based metrics, comparing results with surface-form metrics.
• Performed qualitative error analysis on execution failures, categorizing issues by exception type (e.g., undefined variables, API misuse) and output errors (e.g., incorrect, missing, or excessive outputs).
• Used a case study to illustrate how high BLEU scores can coexist with incorrect execution, proving the insufficiency of surface-form metrics.

Experimental results

Research questions

• RQ1 To what extent do high-surface-form metric scores (e.g., BLEU, CodeBLEU) correlate with execution correctness in data science code generation?
• RQ2 How do state-of-the-art code generation models perform on execution-based evaluation using the ExeDS benchmark?
• RQ3 What types of errors are most common in generated code that pass surface-form metrics but fail execution?
• RQ4 Can execution-based evaluation detect functional errors that surface-form metrics miss?
• RQ5 How do different error categories (e.g., undefined variables, schema misuse) impact model reliability in data science code generation?

Key findings

• Models with high BLEU and CodeBLEU scores do not necessarily generate execution-correct code; for example, Codex achieved low BLEU but high execution accuracy.
• 45% of execution failures were due to using undefined variables, indicating poor data-flow understanding in models.
• 16% of errors stemmed from API misuse, often due to incorrect method calls or missing imports, highlighting challenges in API reasoning.
• 22% of failures were caused by incorrect data schema handling, such as misinterpreting dataframe column names or structures.
• Only 8% of errors were due to syntax issues, showing that models generate syntactically correct code but fail functionally.
• 24% of generated outputs were excessive, indicating models often produce more output than required, even when the code is partially correct.