[Paper Review] Large-scale Cloze Test Dataset Created by Teachers

Qizhe Xie, Guokun Lai+2 more|arXiv (Cornell University)|Nov 9, 2017

Natural Language Processing Techniques40 references4 citations

TL;DR

This paper introduces CLOTH, a large-scale cloze test dataset created by middle- and high-school teachers to evaluate language understanding. Unlike automatically generated datasets, CLOTH features carefully designed blanks and nuanced candidate options that test vocabulary, grammar, and reasoning, requiring deeper comprehension and longer context modeling. The key finding is that state-of-the-art language models significantly underperform humans, primarily due to limited long-context understanding, confirming that CLOTH poses a challenging benchmark for NLP systems.

ABSTRACT

Cloze tests are widely adopted in language exams to evaluate students' language proficiency. In this paper, we propose the first large-scale human-created cloze test dataset CLOTH, containing questions used in middle-school and high-school language exams. With missing blanks carefully created by teachers and candidate choices purposely designed to be nuanced, CLOTH requires a deeper language understanding and a wider attention span than previously automatically-generated cloze datasets. We test the performance of dedicatedly designed baseline models including a language model trained on the One Billion Word Corpus and show humans outperform them by a significant margin. We investigate the source of the performance gap, trace model deficiencies to some distinct properties of CLOTH, and identify the limited ability of comprehending the long-term context to be the key bottleneck.

Motivation & Objective

• To address the limitations of automatically generated cloze datasets, which often produce trivial or ambiguous questions due to random blank selection and poor candidate generation.
• To create a large-scale, human-curated cloze test dataset that reflects real educational assessments and tests deeper language understanding.
• To investigate why state-of-the-art language models underperform on human-designed cloze questions compared to human performance.
• To identify the root cause of the performance gap between models and humans in cloze-style reading comprehension tasks.
• To provide a robust benchmark for evaluating long-context modeling capabilities in language models and machine comprehension systems.

Proposed method

• CLOTH is constructed from real middle- and high-school English exams, with blanks created by teachers to test specific linguistic phenomena such as vocabulary, grammar, and inference.
• Each cloze question includes a context paragraph and a sentence with a blank, along with four candidate options—three distractors and one correct answer—carefully selected to be grammatically correct and semantically nuanced.
• The dataset is split into three subsets: CLOTH (full), CLOTH-M (medium), and CLOTH-H (hard), to evaluate model performance across varying difficulty levels.
• A representativeness-based averaging method is proposed to improve model predictions by weighting candidate answers based on their semantic relevance to the context.
• The authors train and evaluate a language model on the One Billion Word Corpus and compare its performance to human annotators on CLOTH, using ablation studies to isolate the impact of external data and representativeness features.
• Error analysis and human evaluation under restricted context conditions (e.g., one-sentence context) are used to validate the hypothesis that long-context modeling is the key bottleneck.

Experimental results

Research questions

• RQ1 Why do state-of-the-art language models underperform on human-created cloze tests compared to human performance?
• RQ2 What specific linguistic properties of CLOTH make it more challenging than automatically generated cloze datasets?
• RQ3 To what extent does the inability to model long-range dependencies contribute to the performance gap between models and humans on CLOTH?
• RQ4 How effective is the representativeness-based averaging method in improving model predictions on CLOTH?
• RQ5 Can human-created cloze questions serve as a more reliable benchmark for evaluating deep language understanding than automatically generated ones?

Key findings

• Humans outperform the best language model (1B-LM) on CLOTH by a significant margin, with a performance gap of approximately 10% on the full dataset.
• The performance gap is primarily attributed to the model’s limited ability to comprehend long-term context, as evidenced by similar performance drops when humans are restricted to one-sentence context.
• Ablation studies show that removing representativeness information reduces accuracy from 0.583 to 0.566, and removing human-created data entirely reduces it to 0.543, close to the baseline language model.
• The model performs substantially better on automatically generated cloze datasets than on CLOTH, confirming that human-created questions are more difficult and require deeper understanding.
• The representativeness-based averaging method improves performance over baseline models, demonstrating the value of incorporating semantic relevance in candidate selection.
• The F1 score of the prediction model on the test set is 36.5, indicating that multiple plausible questions can be generated per passage, highlighting the dataset’s complexity.