[Paper Review] Many bioinformatics programming tasks can be automated with ChatGPT

Stephen Piccolo, Paul Denny+3 more|arXiv (Cornell University)|Mar 7, 2023

Software Engineering ResearchComputer Science62 references19 citations

TL;DR

The paper evaluates OpenAI's ChatGPT on 184 introductory bioinformatics programming exercises and finds that it can solve a majority on first try and up to 97.3% within seven attempts with natural-language feedback.

ABSTRACT

Computer programming is a fundamental tool for life scientists, allowing them to carry out many essential research tasks. However, despite a variety of educational efforts, learning to write code can be a challenging endeavor for both researchers and students in life science disciplines. Recent advances in artificial intelligence have made it possible to translate human-language prompts to functional code, raising questions about whether these technologies can aid (or replace) life scientists' efforts to write code. Using 184 programming exercises from an introductory-bioinformatics course, we evaluated the extent to which one such model -- OpenAI's ChatGPT -- can successfully complete basic- to moderate-level programming tasks. On its first attempt, ChatGPT solved 139 (75.5%) of the exercises. For the remaining exercises, we provided natural-language feedback to the model, prompting it to try different approaches. Within 7 or fewer attempts, ChatGPT solved 179 (97.3%) of the exercises. These findings have important implications for life-sciences research and education. For many programming tasks, researchers no longer need to write code from scratch. Instead, machine-learning models may produce usable solutions. Instructors may need to adapt their pedagogical approaches and assessment techniques to account for these new capabilities that are available to the general public.

Motivation & Objective

• Motivate the need to assess whether AI can reduce the coding burden for life-science researchers and students.
• Assess ChatGPT's ability to complete basic- to moderate-level bioinformatics programming tasks using natural-language prompts.
• Quantify the improvement with iterative feedback to the model.

Proposed method

• Use 184 programming exercises from an introductory bioinformatics course as benchmarks.
• Test OpenAI's ChatGPT on first attempts and track success after natural-language feedback across up to seven attempts.
• Report success rates and discuss implications for education and research practice.

Experimental results

Research questions

• RQ1 Can ChatGPT solve basic- to moderate-level bioinformatics programming tasks from natural-language prompts?
• RQ2 How does iterative feedback impact ChatGPT's ability to complete the exercises?
• RQ3 What are the implications of AI-assisted code generation for bioinformatics education and research workflows?

Key findings

• 139 of 184 exercises (75.5%) were solved on the first attempt by ChatGPT.
• With natural-language feedback, the model could attempt different approaches and solve a total of 179 of 184 exercises (97.3%) within seven attempts.
• The results have implications for replacing or reducing manual coding in life-sciences research and for adjusting teaching and assessment approaches.
• The findings suggest machine-learning models may produce usable solutions for many programming tasks in bioinformatics.
• The study discusses how educators might adapt pedagogy to account for these capabilities.