[Paper Review] Programming with AI: Evaluating ChatGPT, Gemini, AlphaCode, and GitHub Copilot for Programmers

Md Kamrul Siam, Huanying Gu+1 more|arXiv (Cornell University)|Nov 14, 2024

Artificial Intelligence in Healthcare and Education4 citations

TL;DR

This study evaluates the code generation accuracy of leading AI programming assistants—ChatGPT, Gemini, GitHub Copilot, and AlphaCode—across Python, Java, and C++ using benchmarks like HumanEval and Natural2Code. It finds that Gemini 1.5 Pro and GPT-4-Turbo achieve top pass@100 rates, while GitHub Copilot and AlphaCode show strong but less consistent performance, highlighting the need for improved reliability and ethical deployment in AI-assisted development.

ABSTRACT

Our everyday lives now heavily rely on artificial intelligence (AI) powered large language models (LLMs). Like regular users, programmers are also benefiting from the newest large language models. In response to the critical role that AI models play in modern software development, this study presents a thorough evaluation of leading programming assistants, including ChatGPT, Gemini(Bard AI), AlphaCode, and GitHub Copilot. The evaluation is based on tasks like natural language processing and code generation accuracy in different programming languages like Java, Python and C++. Based on the results, it has emphasized their strengths and weaknesses and the importance of further modifications to increase the reliability and accuracy of the latest popular models. Although these AI assistants illustrate a high level of progress in language understanding and code generation, along with ethical considerations and responsible usage, they provoke a necessity for discussion. With time, developing more refined AI technology is essential for achieving advanced solutions in various fields, especially with the knowledge of the feature intricacies of these models and their implications. This study offers a comparison of different LLMs and provides essential feedback on the rapidly changing area of AI models. It also emphasizes the need for ethical developmental practices to actualize AI models' full potential.

Motivation & Objective

• To assess the code generation accuracy of state-of-the-art AI models—ChatGPT, Gemini, GitHub Copilot, and AlphaCode—across multiple programming languages.
• To identify the most effective metrics and benchmarks for evaluating LLM-generated code in real-world development scenarios.
• To analyze the strengths, weaknesses, and ethical implications of deploying AI models in software engineering workflows.
• To provide actionable feedback for improving model reliability and responsible use in programming contexts.

Proposed method

• Conducted empirical evaluation of code generation using standardized benchmarks such as HumanEval and Natural2Code across Python, Java, and C++.
• Measured model performance using metrics including pass@k and test case success rates to assess correctness and reliability.
• Evaluated models on diverse programming tasks, including natural language to code translation and algorithmic problem solving.
• Compared model outputs against human-written code to assess syntactic correctness, functionality, and semantic accuracy.
• Analyzed architectural and training differences across models to explain performance variations.
• Reviewed ethical considerations and responsible deployment practices based on observed model behaviors and limitations.

Experimental results

Research questions

• RQ1 RQ1: Which model provides the most accurate code for programmers across multiple programming languages?
• RQ2 RQ2: What are the key metrics used to evaluate the quality and correctness of LLM-generated code?
• RQ3 RQ3: Which benchmarks are most effective in measuring the real-world performance of AI coding assistants?

Key findings

• Gemini 1.5 Pro and Gemini-Ultra achieved high pass@100 rates on HumanEval and Natural2Code, demonstrating strong performance in code generation accuracy.
• GPT-4-Turbo models from OpenAI showed consistently high test case success rates, indicating strong reliability in generating functional code.
• GitHub Copilot demonstrated strong real-time code completion and feedback capabilities, though with variable correctness across complex tasks.
• AlphaCode, particularly AlphaCode 2, achieved competitive results in competitive programming, outperforming 85% of human participants on average in some benchmarks.
• ChatGPT exhibited strong natural language understanding and code generation, especially in translating natural language descriptions into working code.
• All models showed significant limitations in accuracy and reliability, requiring human review and validation before production use.