[Paper Review] On the Educational Impact of ChatGPT: Is Artificial Intelligence Ready to Obtain a University Degree?

Kamil Malinka, Martin Perešíni+3 more|arXiv (Cornell University)|Mar 20, 2023

Artificial Intelligence in Healthcare and Education43 citations

TL;DR

The paper evaluates ChatGPT’s ability to tackle university-level computer security tasks, analyzes misuse vs assistive potential, and discusses necessary educational adaptations.

ABSTRACT

In late 2022, OpenAI released a new version of ChatGPT, a sophisticated natural language processing system capable of holding natural conversations while preserving and responding to the context of the discussion. ChatGPT has exceeded expectations in its abilities, leading to extensive considerations of its potential applications and misuse. In this work, we evaluate the influence of ChatGPT on university education, with a primary focus on computer security-oriented specialization. We gather data regarding the effectiveness and usability of this tool for completing exams, programming assignments, and term papers. We evaluate multiple levels of tool misuse, ranging from utilizing it as a consultant to simply copying its outputs. While we demonstrate how easily ChatGPT can be used to cheat, we also discuss the potentially significant benefits to the educational system. For instance, it might be used as an aid (assistant) to discuss problems encountered while solving an assignment or to speed up the learning process. Ultimately, we discuss how computer science higher education should adapt to tools like ChatGPT.

Motivation & Objective

• Assess the effectiveness of ChatGPT in solving university computer security tasks (exams, essays, programming, CTF).
• Examine varying levels of AI use from copying outputs to using AI as an assistant.
• Discuss positive and negative educational impacts and propose adaptations to curricula and assessment.
• Explore detection, prevention, and adoption strategies for AI-assisted learning in higher education.

Proposed method

• Define exam categories (written exams, term essays, programming assignments, CTF) and scoring schemes.
• Establish three AI usage modes (Copy&Paste, Interpretation, Assistant) and map them to task types.
• Apply ChatGPT to a baseline set of real Czech student assignments across four security-focused courses.
• Compare AI-generated results with historical student performance to assess relative merit.
• Use GPTZero as a detector and discuss language-specific detection limitations (English vs Czech).
• Evaluate overall pass-rate implications for courses with varying assessment weights.

Figure 1 . Trend in Google searches for chatbot and AI-related queries. Retrieved from Google Trends 2 2 2 Accessed 12.1.2023, https://trends.google.com/trends/explore?date=today%203-m&q=chatgpt,openai,chatbot,AI .

Experimental results

Research questions

• RQ1 Can ChatGPT achieve comparable or superior scores to average students across different exam formats?
• RQ2 How do different AI usage modes affect performance and learning outcomes?
• RQ3 What are the positive and negative impacts of AI-assisted learning on education quality and integrity?
• RQ4 What measures (detection, prevention, adoption) are needed to adapt higher education to AI tools?
• RQ5 Is it feasible to use AI as an educational assistant to accelerate learning while maintaining rigor?

Key findings

• ChatGPT can produce correct answers in fulltext exams but with broader, less specific responses and context sensitivity issues.
• In tests, Copy&Paste mode can yield variable results and harsh scoring can penalize incorrect answers; interpretation mode improves quality.
• AI performs worse on term essays than humans in some cases, though AI-assisted writing can save time for certain tasks (e.g., threat modeling, descriptions).
• For predefined code, ChatGPT can generate code and complete tasks quickly, but deep understanding and learning are reduced when simply copying outputs.
• Small projects and programming tasks show ChatGPT can implement algorithms and generate executable code, but learning depth may be compromised.
• Overall, ChatGPT can pass courses depending on weightings, with minor losses in some scenarios when used in Copy&Paste mode during tests.
• The study highlights significant educational risks (plagiarism, loss of foundational knowledge) alongside potential benefits (AI as a learning assistant, accelerated comprehension).
• Recommendations include detection/prevention tools, milestone-based assessments, and adopting AI as a curricular aid to foster critical thinking and problem-solving.

Figure 2 . The results obtained by scoring AI outputs in multiple evaluation categories with comparison to a genuine sample of student scores.