[Paper Review] Learning to Automatically Generate Fill-In-The-Blank Quizzes

Edison Marrese-Taylor, Ai Nakajima+2 more|arXiv (Cornell University)|Jun 12, 2018

Topic Modeling11 references3 citations

TL;DR

This paper proposes two deep learning approaches—sequence labeling and sequence classification—using bidirectional LSTMs to automatically generate fill-in-the-blank quizzes from natural language sentences. Evaluated on a real-world language learning platform dataset of ~1.5 million examples, both models achieve F1-scores and accuracy above 88%, demonstrating the feasibility of learning-based, adaptive quiz generation without hand-crafted rules or distractor engineering.

ABSTRACT

In this paper we formalize the problem automatic fill-in-the-blank question generation using two standard NLP machine learning schemes, proposing concrete deep learning models for each. We present an empirical study based on data obtained from a language learning platform showing that both of our proposed settings offer promising results.

Motivation & Objective

• To formalize automatic fill-in-the-blank quiz generation as a sequence labeling and sequence classification problem using deep learning.
• To replace rule-based and template-driven quiz generation with data-driven, end-to-end learning from user interaction data.
• To enable adaptive learning by generating quizzes tailored to user proficiency and style, based on real platform usage patterns.
• To evaluate the effectiveness of representation learning via word embeddings in generating contextually appropriate blanks.
• To explore the potential of unsupervised or weakly supervised learning from unlabeled, real-world user-generated quiz data.

Proposed method

• Formalize the task as a sequence labeling problem where each token in a sentence is assigned a label indicating whether it should be blanked.
• Formalize the task as a sequence classification problem where the model predicts the correct word to blank from a sentence.
• Use bidirectional LSTMs with 300-dimensional word embeddings and 300-dimensional hidden states for both models.
• Apply dropout with a rate of 0.2 before and after the LSTM layers to prevent overfitting.
• Train both models using the Adam optimizer with a base learning rate of 0.001 and gradient clipping to a maximum norm of 5.
• Use a 70/10/20 split for training, validation, and testing, with sentence-level tokenization and POS tagging via CoreNLP.

Experimental results

Research questions

• RQ1 Can bidirectional LSTM models effectively learn to identify the most suitable content to blank in a sentence for quiz generation?
• RQ2 How do sequence labeling and sequence classification approaches compare in performance for automatic fill-in-the-blank quiz generation?
• RQ3 To what extent can models trained on real user-generated quiz data from a language learning platform generalize to unseen sentences?
• RQ4 Can representation learning via word embeddings alone achieve high-quality blank selection without hand-engineered features?
• RQ5 Can such models support adaptive learning by generating quizzes of varying difficulty and style?

Key findings

• The sequence labeling model achieved an F1-score of 88.80% on the test set, with precision and recall of 88.56% and 88.34%, respectively.
• The sequence classification model reached a test accuracy of 89.31%, indicating strong performance in selecting the correct word to blank.
• Both models demonstrated robust generalization, with minimal performance drop between validation and test sets.
• The results suggest that end-to-end learning from user-generated data can effectively replace rule-based or template-driven approaches in quiz generation.
• The models' performance indicates that representation learning via word embeddings is sufficient for high-quality blank selection without relying on external features like POS tags or dependency parsing.
• The study establishes a foundation for adaptive quiz generation systems that can learn from real user interactions and tailor content to individual learning styles and difficulty levels.