[Paper Review] The end of multiple choice tests: using AI to enhance assessment

Michael W. Klymkowsky, Melanie M. Cooper|arXiv (Cornell University)|Jun 11, 2024

Artificial Intelligence in HealthcareHealth Professions3 citations

TL;DR

This paper proposes enhancing multiple-choice assessments by integrating AI to analyze student explanations of their answers, addressing the limitations of traditional MC tests that fail to reveal misconceptions. Using an AI bot trained on scientific concepts, the method identifies misconceptions and provides actionable feedback, demonstrating in a proof-of-concept study that AI-analyzed explanations yield rapid, insightful, and formative assessment data.

ABSTRACT

Effective teaching relies on knowing what students know-or think they know. Revealing student thinking is challenging. Often used because of their ease of grading, even the best multiple choice (MC) tests, those using research based distractors (wrong answers) are intrinsically limited in the insights they provide due to two factors. When distractors do not reflect student beliefs they can be ignored, increasing the likelihood that the correct answer will be chosen by chance. Moreover, making the correct choice does not guarantee that the student understands why it is correct. To address these limitations, we recommend asking students to explain why they chose their answer, and why "wrong" choices are wrong. Using a discipline-trained artificial intelligence-based bot it is possible to analyze their explanations, identifying the concepts and scientific principles that maybe missing or misapplied. The bot also makes suggestions for how instructors can use these data to better guide student thinking. In a small "proof of concept" study, we tested this approach using questions from the Biology Concepts Instrument (BCI). The result was rapid, informative, and provided actionable feedback on student thinking. It appears that the use of AI addresses the weaknesses of conventional MC test. It seems likely that incorporating AI-analyzed formative assessments will lead to improved overall learning outcomes.

Motivation & Objective

• To address the limitations of traditional multiple-choice tests in revealing student misconceptions and understanding.
• To develop a scalable method for formative assessment that captures student reasoning beyond correct/incorrect answers.
• To leverage AI to analyze student explanations and identify gaps in conceptual understanding.
• To provide instructors with actionable feedback based on student reasoning patterns.
• To demonstrate the feasibility and value of AI-enhanced MC assessments in real educational settings.

Proposed method

• Students are asked to select a multiple-choice answer and explain both their reasoning and why incorrect options are wrong.
• An AI-based bot, fine-tuned on discipline-specific scientific content, analyzes the explanations for conceptual accuracy and reasoning patterns.
• The AI identifies missing or misapplied scientific principles and classifies misconceptions using natural language processing.
• The system generates structured feedback for instructors on common misconceptions and learning gaps.
• The approach uses the Biology Concepts Instrument (BCI) as a testbed for validation in a proof-of-concept study.
• Results are aggregated to provide instructors with real-time insights into student thinking across a class.

Experimental results

Research questions

• RQ1 Can AI effectively analyze student explanations of multiple-choice responses to identify misconceptions?
• RQ2 Does incorporating explanation analysis improve the diagnostic value of multiple-choice assessments?
• RQ3 Can AI-generated feedback help instructors target specific conceptual misunderstandings more effectively?
• RQ4 How quickly and reliably can AI process student explanations in a classroom setting?
• RQ5 To what extent does this method enhance formative assessment compared to traditional MC testing?

Key findings

• The AI-based analysis of student explanations successfully identified misconceptions and conceptual gaps not detectable through correct/incorrect scoring alone.
• The system provided rapid, actionable feedback to instructors, enabling targeted instructional interventions.
• In the proof-of-concept study, the method revealed nuanced reasoning patterns that traditional MC testing would miss.
• Students' explanations of why wrong answers were incorrect proved particularly informative for diagnosing misconceptions.
• The integration of AI enhanced the diagnostic power of MC assessments without sacrificing scalability.
• The results suggest that AI-analyzed formative assessments can lead to improved learning outcomes by addressing misunderstandings early.