[Paper Review] A Comparative Study of Code Generation using ChatGPT 3.5 across 10 Programming Languages

Alessio Buscemi|arXiv (Cornell University)|Aug 8, 2023

Artificial Intelligence in Healthcare and Education16 citations

TL;DR

The paper investigates ChatGPT 3.5’s ability to generate executable code across 10 languages using 40 coding tasks, analyzing time, code length, and limitations.

ABSTRACT

Large Language Models (LLMs) are advanced Artificial Intelligence (AI) systems that have undergone extensive training using large datasets in order to understand and produce language that closely resembles that of humans. These models have reached a level of proficiency where they are capable of successfully completing university exams across several disciplines and generating functional code to handle novel problems. This research investigates the coding proficiency of ChatGPT 3.5, a LLM released by OpenAI in November 2022, which has gained significant recognition for its impressive text generating and code creation capabilities. The skill of the model in creating code snippets is evaluated across 10 various programming languages and 4 different software domains. Based on the findings derived from this research, major unexpected behaviors and limitations of the model have been identified. This study aims to identify potential areas for development and examine the ramifications of automated code generation on the evolution of programming languages and on the tech industry.

Motivation & Objective

• Assess ChatGPT 3.5’s code generation ability across 10 programming languages.
• Evaluate execution success rates and time performance for 40 coding tasks.
• Analyze code length, variability, and practical limitations of automated code generation.
• Identify language-dependent strengths, weaknesses, and ethical/technical concerns.

Proposed method

• Query ChatGPT 3.5 via OpenAI API (Turbo, role set to 'software developer', temperature 1).
• Use a fixed 40-task corpus spanning DS, Games, Security, and Algos categories.
• Test each task in 10 languages with 10 runs per task per language (4,000 tests total).
• Post-process outputs to extract code, tests, and language-specific formatting; classify outcomes into six statuses.
• Measure time per task per language relative to language-wide task means (P_l).
• Record LoC and NoC to assess code length and variability.

Figure 1: Status of the output generated by ChatGPT for the 4,000 tests, grouped by programming language and category.

Experimental results

Research questions

• RQ1 How does ChatGPT 3.5 perform in generating correct and executable code across different programming languages?
• RQ2 What are the language-dependent factors (abstraction level, training popularity) that influence code generation quality and success rates?
• RQ3 What are the time profiles and code-length characteristics of generated code across languages?
• RQ4 What limitations and ethical considerations arise in automated code generation across tasks and languages?

Key findings

• 1833 out of 4000 runs (45.8%) produced executable code, with results varying by language.
• Julia yielded the highest execution success (81.5%), while C++ had the lowest (7.3%).
• High-level, dynamically typed languages generally fared better than lower-level, statically typed ones; training corpus popularity also impacted performance.
• Time performance varied by language; e.g., palindromeInteger in C++ was fastest (4.83 s) while randomForest in C was slowest (140.7 s).
• Code length (LoC/NoC) did not correlate clearly with execution time and showed higher variability across languages.
• ChatGPT 3.5 exhibited notable limitations, including inconsistent task understanding, occasional non-compliance with instructions, and ethical concerns in certain tasks.

Figure 2: $P_{\ell}$ of each language.