[Paper Review] L2CEval: Evaluating Language-to-Code Generation Capabilities of Large Language Models

Ansong Ni, Pengcheng Yin+12 more|arXiv (Cornell University)|Sep 29, 2023

Natural Language Processing TechniquesComputer Science3 citations

TL;DR

L2CEval presents a comprehensive, standardized benchmark for evaluating language-to-code (L2C) generation in large language models (LLMs), assessing 54 models across 7 tasks spanning semantic parsing, math reasoning, and Python programming. The study analyzes model performance, robustness, confidence calibration, and failure modes, revealing key impacts of model size, pretraining data, instruction tuning, and prompting strategies, while releasing all model outputs for future research.

ABSTRACT

Recently, large language models (LLMs), especially those that are pretrained on code, have demonstrated strong capabilities in generating programs from natural language inputs in a few-shot or even zero-shot manner. Despite promising results, there is a notable lack of a comprehensive evaluation of these models language-to-code generation capabilities. Existing studies often focus on specific tasks, model architectures, or learning paradigms, leading to a fragmented understanding of the overall landscape. In this work, we present L2CEval, a systematic evaluation of the language-to-code generation capabilities of LLMs on 7 tasks across the domain spectrum of semantic parsing, math reasoning and Python programming, analyzing the factors that potentially affect their performance, such as model size, pretraining data, instruction tuning, and different prompting methods. In addition to assessing model performance, we measure confidence calibration for the models and conduct human evaluations of the output programs. This enables us to identify and analyze the typical failure modes across various tasks and models. L2CEval offers a comprehensive understanding of the capabilities and limitations of LLMs in language-to-code generation. We also release the evaluation framework and all model outputs, hoping to lay the groundwork for further future research in this domain.

Motivation & Objective

• To address the lack of a systematic, standardized evaluation framework for language-to-code (L2C) generation in large language models (LLMs).
• To assess the impact of model size, pretraining data mixture, instruction tuning, and prompting methods on L2C performance across diverse tasks.
• To measure model robustness and confidence calibration, identifying common failure modes across different model capabilities.
• To provide a comprehensive, reproducible benchmark with consistent prompts and metrics for fair comparison across 54 models from 13 organizations.
• To release all model outputs and the evaluation framework to accelerate future research in L2C generation.

Proposed method

• Standardized evaluation protocol with consistent prompts and metrics across 7 L2C tasks: 2 semantic parsing (Spider, WikiTQ), 2 math reasoning (GSM8K, SVAMP), and 3 Python programming (MBPP, HumanEval, DS-1000).
• Evaluation of 54 LLMs from 13 organizations, including open-source and proprietary models, ranging from 1B to 170B+ parameters.
• Systematic analysis of scaling laws for model size and pretraining data (35B to 1T tokens), measuring performance gains across tasks.
• Comparison of few-shot, zero-shot, and instruction-tuned prompting strategies to assess their impact on generation quality.
• Confidence calibration analysis using logits and prediction confidence scores to evaluate model reliability.
• Human evaluation of model outputs to validate automatic metrics and identify qualitative failure patterns.

Experimental results

Research questions

• RQ1 How do model size and pretraining data scale in performance across diverse language-to-code tasks?
• RQ2 What is the relative impact of instruction tuning and different prompting strategies (zero-shot vs. few-shot) on L2C generation quality?
• RQ3 How well-calibrated are LLMs' confidence scores in generating correct code, and how does this vary across model types and tasks?
• RQ4 What are the most common failure modes in LLM-generated code, and how do they differ across model capabilities and task types?
• RQ5 To what extent do automatic metrics correlate with human judgments of code correctness and quality?

Key findings

• Larger models consistently outperform smaller ones across all L2C tasks, with performance gains following a predictable scaling law across model size and pretraining data size.
• Instruction tuning significantly improves zero-shot and few-shot performance, especially in complex reasoning and code generation tasks.
• Confidence calibration varies widely: larger models and instruction-tuned models show better calibration, reducing overconfidence in incorrect outputs.
• Common failure modes include incorrect arithmetic, misinterpretation of problem constraints, and logical errors in control flow, particularly in math reasoning and semantic parsing.
• Automatic metrics like exact match and pass@k correlate moderately with human judgments, but fail to capture subtle logical or syntactic errors.
• The release of all model outputs (texts and logits) enables future research to analyze failure patterns, improve evaluation metrics, and train more robust LLMs for code generation.