[Paper Review] Automatic Code Documentation Generation Using GPT-3

Junaed Younus Khan, Gias Uddin|arXiv (Cornell University)|Sep 6, 2022

Machine Learning and Data Classification7 citations

TL;DR

This paper evaluates Codex, a GPT-3-based model fine-tuned on code and natural language, for automatic code documentation generation across six programming languages. Using only one-shot prompting, Codex achieves a BLEU score of 20.6, significantly outperforming prior state-of-the-art methods by 11.2% in this task, demonstrating strong zero-shot and few-shot generalization for code documentation.

ABSTRACT

Source code documentation is an important artifact for efficient software development. Code documentation could greatly benefit from automation since manual documentation is often labouring, resource and time-intensive. In this paper, we employed Codex for automatic code documentation creation. Codex is a GPT-3 based model pre-trained on both natural and programming languages. We find that Codex outperforms existing techniques even with basic settings like one-shot learning (i.e., providing only one example for training). Codex achieves an overall BLEU score of 20.6 for six different programming languages (11.2% improvement over earlier state-of-the-art techniques). Thus, Codex shows promise and warrants in-depth future studies for automatic code documentation generation to support diverse development tasks.

Motivation & Objective

• To investigate the effectiveness of Codex, a GPT-3-based model, in generating code documentation without fine-tuning.
• To compare Codex’s performance against existing state-of-the-art models in automatic code documentation generation.
• To assess the quality of generated documentation using automatic metrics (BLEU) and human-readable examples.
• To explore the potential of few-shot and zero-shot prompting for code documentation tasks.

Proposed method

• Codex, a GPT-3-based model pre-trained on diverse programming and natural language data, was used for code documentation generation.
• The model was prompted using one-shot learning, where a single example of code-documentation pair was provided to guide generation.
• The evaluation was conducted on the CodeSearchNet dataset, covering six programming languages: Python, Java, PHP, Go, JavaScript, and Ruby.
• Generated documentation was evaluated using BLEU score for automatic metric comparison against actual human-written documentation.
• The study also included qualitative analysis of generated documentation, comparing clarity and completeness with ground-truth documentation.
• Parameter settings were selected based on official OpenAI documentation, with no model fine-tuning performed.

Experimental results

Research questions

• RQ1 Can Codex generate high-quality code documentation using only one-shot prompting without fine-tuning?
• RQ2 How does Codex’s performance in code documentation generation compare to existing state-of-the-art models like CodeBERT, PLBART, and CoTexT?
• RQ3 What is the impact of few-shot prompting on the quality and consistency of generated documentation?
• RQ4 How does the generated documentation from Codex compare in clarity and completeness to human-written documentation?
• RQ5 What are the limitations of using Codex for code documentation in real-world software engineering settings?

Key findings

• Codex achieved a BLEU score of 20.6 on the CodeSearchNet dataset, representing an 11.2% improvement over the previous state-of-the-art techniques.
• Even with minimal prompting (one-shot), Codex outperformed existing models that require task-specific fine-tuning or retraining.
• In qualitative analysis, Codex generated more comprehensible or semantically richer documentation than the actual human-written versions in several cases.
• For example, Codex correctly described a PHP function as converting a base-256 number to a base-10 number, while the original used the less precise term 'decimal'.
• In Ruby, Codex added the correct semantic detail that 'if the path is not present a noop is performed', which was absent in the original documentation.
• The study confirms that GPT-3-based models like Codex can serve as strong zero-shot or few-shot baselines for code documentation generation, warranting further investigation.