[Paper Review] NERIF: GPT-4V for Automatic Scoring of Drawn Models

Gyeong-Geon Lee, Xiaoming Zhaı|arXiv (Cornell University)|Nov 21, 2023

Genetics, Bioinformatics, and Biomedical Research9 citations

TL;DR

The paper introduces NERIF, a prompt-engineered method using GPT-4V to automatically score student-drawn scientific models viaFew-shot learning with instructional notes and rubrics, achieving moderate test accuracy and explainable scoring.

ABSTRACT

Scoring student-drawn models is time-consuming. Recently released GPT-4V provides a unique opportunity to advance scientific modeling practices by leveraging the powerful image processing capability. To test this ability specifically for automatic scoring, we developed a method NERIF (Notation-Enhanced Rubric Instruction for Few-shot Learning) employing instructional note and rubrics to prompt GPT-4V to score students' drawn models for science phenomena. We randomly selected a set of balanced data (N = 900) that includes student-drawn models for six modeling assessment tasks. Each model received a score from GPT-4V ranging at three levels: 'Beginning,' 'Developing,' or 'Proficient' according to scoring rubrics. GPT-4V scores were compared with human experts' scores to calculate scoring accuracy. Results show that GPT-4V's average scoring accuracy was mean =.51, SD = .037. Specifically, average scoring accuracy was .64 for the 'Beginning' class, .62 for the 'Developing' class, and .26 for the 'Proficient' class, indicating that more proficient models are more challenging to score. Further qualitative study reveals how GPT-4V retrieves information from image input, including problem context, example evaluations provided by human coders, and students' drawing models. We also uncovered how GPT-4V catches the characteristics of student-drawn models and narrates them in natural language. At last, we demonstrated how GPT-4V assigns scores to student-drawn models according to the given scoring rubric and instructional notes. Our findings suggest that the NERIF is an effective approach for employing GPT-4V to score drawn models. Even though there is space for GPT-4V to improve scoring accuracy, some mis-assigned scores seemed interpretable to experts. The results of this study show that utilizing GPT-4V for automatic scoring of student-drawn models is promising.

Motivation & Objective

• Motivate the need for automatic scoring of student-drawn models in science education to save time and provide timely feedback.
• Develop a prompt-based method (NERIF) that leverages GPT-4V’s image processing and language abilities to score drawn models.
• Assess GPT-4V performance against human expert scores across six modeling tasks.
• Demonstrate how instructional notes and rubrics enable interpretable, explainable scoring results.

Proposed method

• Use a few-shot learning approach with 9 example evaluations to prompt GPT-4V for trinomial categorization (Beginning, Developing, Proficient).
• Provide two attached images per query: problem context with scoring examples and student-drawn models; retrieve a random example from the prompts to guide scoring.
• Incorporate Notation-Enhanced Scoring Rubrics with three components: scoring aspects, proficiency rules, and instructional notes.
• Conduct validation (N=54) to iteratively refine prompts; then run test scoring (N=900) under greedy decoding (temperature 0, top_p 0.01).
• Evaluate using accuracy, precision, recall, F1, and Fleiss’ Kappa; analyze confusion matrices to understand misclassifications.

Experimental results

Research questions

• RQ1 How accurate is GPT-4V in automatically scoring student-drawn models?
• RQ2 How does GPT-4V automatically assign scores to student-drawn models using the provided rubrics and notes?

Key findings

• Average test scoring accuracy across six items: 0.51 (SD = 0.037).
• Average precision, recall, and F1 across items: 0.58, 0.51, and 0.49 respectively; Fleiss’ Kappa ranged from 0.32 to 0.51 (Fair to Moderate).
• Accuracy by category: Beginning 0.64, Developing 0.61, Proficient 0.26, indicating Proficient is more challenging for GPT-4V to score.
• Validation accuracy averaged 0.67 (Beginning 0.78, Developing 0.67, Proficient 0.56) across six items.
• GPT-4V can retrieve problem context and scoring exemplars from input images and generate natural-language rationales for scoring components.
• Results show that adding example demonstrations (few-shot prompts) and instructional notes improves scoring quality.