[Paper Review] Towards Mitigating ChatGPT's Negative Impact on Education: Optimizing Question Design through Bloom's Taxonomy

Saber Elsayed|arXiv (Cornell University)|Mar 31, 2023

Online Learning and Analytics4 citations

TL;DR

This paper proposes an evolutionary algorithm using genetic optimization to select Bloom’s Taxonomy keywords that generate programming questions which ChatGPT answers with low confidence. By targeting higher-order cognitive skills (e.g., analyze, evaluate, create), the method reduces ChatGPT’s average confidence from 95% to 45%, and generates questions with 0% confidence, helping educators design assessments that promote critical thinking and reduce AI dependency.

ABSTRACT

The popularity of generative text AI tools in answering questions has led to concerns regarding their potential negative impact on students' academic performance and the challenges that educators face in evaluating student learning. To address these concerns, this paper introduces an evolutionary approach that aims to identify the best set of Bloom's taxonomy keywords to generate questions that these tools have low confidence in answering. The effectiveness of this approach is evaluated through a case study that uses questions from a Data Structures and Representation course being taught at the University of New South Wales in Canberra, Australia. The results demonstrate that the optimization algorithm is able to find keywords from different cognitive levels to create questions that ChatGPT has low confidence in answering. This study is a step forward to offer valuable insights for educators seeking to create more effective questions that promote critical thinking among students.

Motivation & Objective

• To address the growing concern that generative AI tools like ChatGPT undermine student learning and assessment integrity by enabling over-reliance on AI-generated answers.
• To identify specific combinations of Bloom’s Taxonomy keywords that reduce AI confidence in answering educational questions, particularly in programming courses.
• To develop an evolutionary optimization approach that systematically evolves question variants to challenge AI reasoning capabilities.
• To support educators in designing assessments that promote deep learning and critical thinking by minimizing AI interference.
• To validate that AI-generated answers to optimized questions are not only less confident but also frequently incorrect or incomplete.

Proposed method

• An evolutionary algorithm based on genetic algorithms (GA) is used to evolve chromosomes representing sets of Bloom’s Taxonomy keywords.
• Each chromosome is used to generate a new question variant via a generative AI tool (e.g., ChatGPT), and the AI’s confidence score in answering the new question serves as the fitness function to be minimized.
• The algorithm applies selection, crossover, and mutation over multiple generations to evolve keyword combinations that minimize the AI’s confidence.
• The search space is optimized by avoiding brute-force enumeration, focusing instead on intelligent evolution of keyword sets across different cognitive levels.
• The method is evaluated on real programming questions from a Data Structures and Representation course at the University of New South Wales Canberra.
• Answers to the optimized questions are manually validated by the course instructor to assess correctness and completeness, ensuring AI limitations are exposed.

Experimental results

Research questions

• RQ1 Can an evolutionary algorithm identify specific combinations of Bloom’s Taxonomy keywords that significantly reduce ChatGPT’s confidence in answering programming questions?
• RQ2 To what extent can such keyword combinations challenge the reasoning and response quality of generative AI models like ChatGPT?
• RQ3 Do questions generated with optimized keyword sets lead to incorrect or incomplete AI responses, even when confidence is low?
• RQ4 Can this approach be used to design assessments that are less susceptible to AI-generated answers while promoting higher-order thinking?
• RQ5 How do different cognitive levels (e.g., analyze, evaluate, create) contribute to reducing AI confidence when combined in question design?

Key findings

• The optimization algorithm successfully reduced ChatGPT’s average confidence in answering questions from 95% to 45% across the test set.
• For several questions, the algorithm generated variants where ChatGPT’s confidence dropped to 0%, indicating complete inability to respond with certainty.
• The best-performing keyword combinations included high-level cognitive verbs such as 'analyze', 'evaluate', 'examine', and 'contrast', often from Bloom’s levels 4–6.
• Manual validation revealed that AI responses to the optimized questions were frequently incorrect, incomplete, or generic, even when confidence was high.
• The algorithm identified effective keyword pairs like 'analyze and design' or 'develop and differentiate' that significantly challenged AI reasoning in programming tasks.
• The results demonstrate that strategic use of diverse Bloom’s Taxonomy keywords can effectively disrupt AI-based answer generation and improve assessment validity.