[Paper Review] Exploring the Use of ChatGPT as a Tool for Learning and Assessment in Undergraduate Computer Science Curriculum: Opportunities and Challenges

Basit Qureshi|arXiv (Cornell University)|Apr 16, 2023

Artificial Intelligence in Healthcare and Education57 citations

TL;DR

The paper experimentally compares learning outcomes with and without ChatGPT for a sophomore Data Structures and Algorithms course, showing ChatGPT helps with scores but introduces inconsistencies and inaccuracies in code.

ABSTRACT

The application of Artificial intelligence for teaching and learning in the academic sphere is a trending subject of interest in the computing education. ChatGPT, as an AI-based tool, provides various advantages, such as heightened student involvement, cooperation, accessibility and availability. This paper addresses the prospects and obstacles associated with utilizing ChatGPT as a tool for learning and assessment in undergraduate Computer Science curriculum in particular to teaching and learning fundamental programming courses. Students having completed the course work for a Data Structures and Algorithms (a sophomore level course) participated in this study. Two groups of students were given programming challenges to solve within a short period of time. The control group (group A) had access to text books and notes of programming courses, however no Internet access was provided. Group B students were given access to ChatGPT and were encouraged to use it to help solve the programming challenges. The challenge was conducted in a computer lab environment using PC2 environment. Each team of students address the problem by writing executable code that satisfies certain number of test cases. Student teams were scored based on their performance in terms of number of successful passed testcases. Results show that students using ChatGPT had an advantage in terms of earned scores, however there were inconsistencies and inaccuracies in the submitted code consequently affecting the overall performance. After a thorough analysis, the paper's findings indicate that incorporating AI in higher education brings about various opportunities and challenges.

Motivation & Objective

• Explore opportunities and challenges of integrating AI tools (ChatGPT) into undergraduate CS education.
• Assess learning and assessment outcomes when students use ChatGPT to tackle programming challenges.
• Evaluate accuracy and reliability of student-submitted code produced with AI assistance.
• Provide insights for curriculum design and pedagogical strategies in AI-assisted learning.

Proposed method

• Two groups of Data Structures and Algorithms students (sophomore level) participated in programming challenges.
• Group A (control) used textbooks and notes with no Internet access.
• Group B had access to ChatGPT and were encouraged to use it to solve challenges.
• Challenges were performed in a PC2 computer lab environment with executable code judged by test cases.
• Teams were scored based on the number of test cases their code passed.
• Comparison focused on performance scores and observed code quality.

Experimental results

Research questions

• RQ1 Does access to ChatGPT improve student performance on programming challenges compared to traditional resources?
• RQ2 What are the benefits and drawbacks of using ChatGPT for learning and assessment in undergraduate CS?
• RQ3 How does AI-assisted coding affect code correctness and reliability in student submissions?

Key findings

• Students using ChatGPT earned higher scores than those without AI assistance.
• There were inconsistencies and inaccuracies in ChatGPT-generated code that affected overall performance.
• AI-assisted learning introduces opportunities (involvement, accessibility) and challenges (code reliability) in higher education.