[Paper Review] Automatic Generation of Multiple-Choice Questions

Cheng Zhang|arXiv (Cornell University)|Mar 25, 2023

Topic Modeling4 citations

TL;DR

This paper presents two deep learning methods—TP3 (a T5-based end-to-end model with preprocessing/postprocessing) and MetaQA (a meta-sequence learning approach using semantic and syntactic tags)—for automatic generation of grammatically correct, contextually relevant multiple-choice questions (MCQs) with high-quality distractors. The system achieves 97% accuracy on SAT practice tests and generates distractors that are both semantically plausible and sufficiently distracting, with 96% providing sufficient distraction and 84% of MCQs being fully adequate.

ABSTRACT

Creating multiple-choice questions to assess reading comprehension of a given article involves generating question-answer pairs (QAPs) and adequate distractors. We present two methods to tackle the challenge of QAP generations: (1) A deep-learning-based end-to-end question generation system based on T5 Transformer with Preprocessing and Postprocessing Pipelines (TP3). We use the finetuned T5 model for our downstream task of question generation and improve accuracy using a combination of various NLP tools and algorithms in preprocessing and postprocessing to select appropriate answers and filter undesirable questions. (2) A sequence-learning-based scheme to generate adequate QAPs via meta-sequence representations of sentences. A meta-sequence is a sequence of vectors comprising semantic and syntactic tags. we devise a scheme called MetaQA to learn meta sequences from training data to form pairs of a meta sequence for a declarative sentence and a corresponding interrogative sentence. The TP3 works well on unseen data, which is complemented by MetaQA. Both methods can generate well-formed and grammatically correct questions. Moreover, we present a novel approach to automatically generate adequate distractors for a given QAP. The method is a combination of part-of-speech tagging, named-entity tagging, semantic-role labeling, regular expressions, domain knowledge bases, word embeddings, word edit distance, WordNet, and other algorithms.

Motivation & Objective

• To address the challenge of automatically generating grammatically correct, contextually relevant multiple-choice questions (MCQs) for reading comprehension assessment.
• To develop a method that generates distractors that are not only grammatically correct but also semantically related and sufficiently distracting to require understanding of the source article.
• To combine NLP techniques such as POS tagging, named-entity recognition, and semantic role labeling to improve the quality and relevance of generated distractors.
• To evaluate the generated MCQs on real SAT practice reading tests to ensure high accuracy and practical applicability in educational settings.
• To complement deep learning-based generation with a meta-sequence learning approach that enhances generalization and reduces incoherent question generation.

Proposed method

• TP3 uses a fine-tuned T5 Transformer model with a preprocessing pipeline to select appropriate answers and a postprocessing pipeline to filter out incoherent questions.
• MetaQA employs meta-sequences composed of semantic-role (SR), part-of-speech (POS), and named-entity (NE) tags to represent declarative sentences and their corresponding interrogative forms.
• The method learns meta-sequence-declarative-interrogative pairs (MSDIP) from training data to map a declarative sentence to its interrogative form via matching and transformation of meta-sequences.
• For distractor generation, the method classifies target words into three types—type-1 (quantitative), type-2 (person/location/organization), and type-3 (others)—and applies type-specific substitution strategies.
• The system integrates POS tagging, named-entity recognition, semantic-role labeling, and fastText embeddings to ensure grammatical correctness and semantic plausibility of distractors.
• Distractors are generated by substituting target words with semantically similar, grammatically consistent alternatives while preserving part-of-speech and syntactic structure.

Figure 3.1: TP3 Architecture

Experimental results

Research questions

• RQ1 Can a T5-based end-to-end model with preprocessing and postprocessing pipelines generate grammatically correct and contextually relevant multiple-choice questions with high accuracy?
• RQ2 Can a meta-sequence learning approach using semantic and syntactic tags effectively generate diverse and coherent questions from declarative sentences?
• RQ3 Can a combination of NLP tools and type-specific substitution strategies generate distractors that are both grammatically correct and sufficiently distracting?
• RQ4 To what extent do the generated MCQs match the quality and difficulty of real SAT practice test questions?
• RQ5 How effective are the generated distractors in requiring article-level understanding rather than just question-level parsing?

Key findings

• The MetaQA model achieved 97% accuracy on the official SAT practice reading tests, demonstrating strong performance on real-world, diverse sentence patterns.
• The TP3 model achieved over 95% accuracy on the Gaokao-EN dataset, showing strong generalization on unseen data.
• 98% of generated distractors (296 out of 303) were relevant to the question and provided sufficient distraction, indicating high semantic plausibility.
• 96% of distractors (291 out of 303) provided sufficient distraction, meaning the correct answer could only be selected with understanding of the article, not just the question.
• 84% of the generated MCQs were fully adequate, with all three distractors being grammatically correct, relevant, and sufficiently distracting.
• All generated distractors were grammatically correct, and no MCQ was rejected due to grammatical errors, confirming high linguistic quality.

Figure 4.1: MetaQA architecture and data flow