[Paper Review] Training Optimus Prime, M.D.: Generating Medical Certification Items by Fine-Tuning OpenAI's gpt2 Transformer Model

Matthias von Davier|arXiv (Cornell University)|Aug 22, 2019

Topic Modeling10 citations

TL;DR

This paper proposes fine-tuning OpenAI's GPT-2 transformer model on PubMed's open-access medical text to generate high-quality medical certification test items. The approach produces coherent, draft-ready case vignettes and effective distractors for multiple-choice questions, demonstrating that transformer-based language models can significantly support automated item generation in medical education with minimal human intervention.

ABSTRACT

This article describes new results of an application using transformer-based language models to automated item generation (AIG), an area of ongoing interest in the domain of certification testing as well as in educational measurement and psychological testing. OpenAI's gpt2 pre-trained 345M parameter language model was retrained using the public domain text mining set of PubMed articles and subsequently used to generate item stems (case vignettes) as well as distractor proposals for multiple-choice items. This case study shows promise and produces draft text that can be used by human item writers as input for authoring. Future experiments with more recent transformer models (such as Grover, TransformerXL) using existing item pools are expected to improve results further and to facilitate the development of assessment materials.

Motivation & Objective

• To investigate whether transformer-based language models can generate medically relevant, coherent test items suitable for certification exams.
• To evaluate the quality of generated text from a fine-tuned GPT-2 model compared to prior RNN-based approaches.
• To explore the feasibility of using prompted text generation for creating distractors in multiple-choice questions.
• To assess the potential of pre-trained language models as tools to support human item writers in medical assessment development.
• To demonstrate that fine-tuning on domain-specific medical corpora improves the relevance and structure of generated medical education content.

Proposed method

• Fine-tuning OpenAI’s GPT-2 language model using PubMed’s open-access medical text corpus as training data.
• Utilizing TensorFlow-GPU toolkits on a multi-GPU workstation to retrain the model for medical language generation.
• Employing prompt-based generation to produce structured case vignettes and distractor options for multiple-choice items.
• Evaluating generated text quality through qualitative assessment and comparison with human-authored items.
• Leveraging the transformer architecture’s attention mechanisms to model long-range dependencies in clinical narratives.
• Using transfer learning principles by adapting a general-purpose language model to the specialized domain of medical certification testing.

Experimental results

Research questions

• RQ1 Can a fine-tuned GPT-2 model generate medically coherent and structurally sound case vignettes suitable for certification exams?
• RQ2 How does the quality of text generated by a transformer model compare to that of prior RNN-based approaches in medical item generation?
• RQ3 To what extent can prompted generation from the fine-tuned model produce effective distractors for multiple-choice questions?
• RQ4 Can domain-specific fine-tuning on medical text improve the relevance and grammatical correctness of generated assessment items?
• RQ5 What is the potential of such models to support human item writers in the development of high-stakes medical certification exams?

Key findings

• The fine-tuned GPT-2 model generated higher-quality, more coherent medical text compared to earlier character-based RNN models trained on open-access items.
• Prompted text generation successfully produced plausible distractors suitable for use in multiple-choice questions for certification exams.
• The generated case vignettes were often grammatically correct and clinically plausible, resembling real patient scenarios commonly used in medical assessments.
• The model demonstrated the ability to produce text that could serve as a draft for human item writers, reducing manual authoring effort.
• Fine-tuning on medical corpora like PubMed significantly improved the domain relevance and structure of generated content.
• The results suggest that transformer-based language models, when properly fine-tuned, can become practical tools for automated item generation in medical education.