[Paper Review] AI-assisted Automated Short Answer Grading of Handwritten University Level Mathematics Exams

Tianyi Liu, Julia Chatain+4 more|arXiv (Cornell University)|Aug 21, 2024

Intelligent Tutoring Systems and Adaptive Learning4 citations

TL;DR

This paper evaluates GPT-4's effectiveness in grading handwritten short-answer responses on a university-level mathematics exam, demonstrating that it provides reliable, cost-effective initial grading with minimal prompt engineering. Despite limitations in recognizing handwritten mathematical expressions, GPT-4 shows strong performance in semantic understanding and equivalence detection, suggesting its potential as an AI assistant for human graders when combined with robust confidence estimation and improved OCR pipelines.

ABSTRACT

Effective and timely feedback in educational assessments is essential but labor-intensive, especially for complex tasks. Recent developments in automated feedback systems, ranging from deterministic response grading to the evaluation of semi-open and open-ended essays, have been facilitated by advances in machine learning. The emergence of pre-trained Large Language Models, such as GPT-4, offers promising new opportunities for efficiently processing diverse response types with minimal customization. This study evaluates the effectiveness of a pre-trained GPT-4 model in grading semi-open handwritten responses in a university-level mathematics exam. Our findings indicate that GPT-4 provides surprisingly reliable and cost-effective initial grading, subject to subsequent human verification. Future research should focus on refining grading rules and enhancing the extraction of handwritten responses to further leverage these technologies.

Motivation & Objective

• To assess the feasibility of using pre-trained large language models like GPT-4 for grading handwritten short-answer responses in university-level mathematics.
• To investigate the reliability and robustness of GPT-4 in identifying mathematically equivalent answers despite paraphrasing or different notational forms.
• To develop and evaluate a method for estimating the confidence of GPT-4’s grading decisions using probability-based grounding.
• To identify key challenges in integrating AI into high-stakes academic assessment, particularly in handwritten expression recognition and grading rule formulation.
• To propose a practical AI-assisted grading pipeline that reduces human grading workload while maintaining accuracy through human-in-the-loop verification.

Proposed method

• Utilized GPT-4 with minimal prompt engineering to grade student responses to six short-answer mathematics problems from a real university exam.
• Employed a two-stage workflow: first, scanned and transcribed entire exam sheets using OCR (via Mathpix API), then extracted answer fields using LaTeX and Python-based parsing.
• Applied a probability-theoretic approach to estimate the confidence of GPT-4’s grading decisions, enabling identification of uncertain or outlier predictions.
• Used a standardized grading rubric to evaluate GPT-4’s output against human-graded responses, focusing on correctness and equivalence detection.
• Conducted iterative refinement of grading prompts and rules based on observed model behavior and error analysis.
• Evaluated model robustness to paraphrasing by testing consistent sentence structure with varied phrasing in prompts.

Experimental results

Research questions

• RQ1 Can GPT-4 reliably grade handwritten short-answer mathematical responses with minimal prompt customization?
• RQ2 How robust is GPT-4’s grading to paraphrasing of the same mathematical concept in different notational forms?
• RQ3 To what extent does GPT-4’s performance depend on the quality of the OCR-transcribed input, particularly for handwritten mathematical expressions?
• RQ4 Can a probability-based confidence estimation method effectively identify unreliable or uncertain AI-generated grades?
• RQ5 What are the key challenges in deploying LLMs for high-stakes, human-in-the-loop grading in university mathematics?

Key findings

• GPT-4 demonstrated high reliability in grading short-answer mathematical responses, correctly identifying mathematically equivalent solutions even with significant notational variation.
• The model showed strong robustness to paraphrasing in grading prompts, maintaining consistent performance when sentence structure remained unchanged.
• GPT-4’s performance was significantly limited by the accuracy of the OCR pipeline for handwritten mathematical expressions, indicating that input quality is a critical bottleneck.
• The proposed probability-based confidence estimation method yielded promising results, offering a viable path toward identifying high-risk grading decisions for human review.
• Despite not being fine-tuned for this task, GPT-4 outperformed specialized models from five years prior, highlighting the potential of general-purpose LLMs in educational assessment.
• The study confirms that LLMs can serve as effective AI assistants in grading workflows, reducing human workload while maintaining accuracy when combined with human verification.