[Paper Review] ChID: A Large-scale Chinese IDiom Dataset for Cloze Test

Chujie Zheng, Minlie Huang+1 more|arXiv (Cornell University)|Jun 4, 2019

Natural Language Processing Techniques32 references4 citations

TL;DR

This paper introduces ChID, a large-scale Chinese idiom dataset with 729K cloze-style questions across news, novels, and essays, where idioms are replaced by blanks and models must select the correct idiom from candidates. Results show state-of-the-art models significantly underperform humans, highlighting the challenge of idiomatic meaning comprehension due to non-compositionality and metaphorical semantics.

ABSTRACT

Cloze-style reading comprehension in Chinese is still limited due to the lack of various corpora. In this paper we propose a large-scale Chinese cloze test dataset ChID, which studies the comprehension of idiom, a unique language phenomenon in Chinese. In this corpus, the idioms in a passage are replaced by blank symbols and the correct answer needs to be chosen from well-designed candidate idioms. We carefully study how the design of candidate idioms and the representation of idioms affect the performance of state-of-the-art models. Results show that the machine accuracy is substantially worse than that of human, indicating a large space for further research.

Motivation & Objective

• To address the lack of large-scale, high-quality Chinese cloze test datasets focused on idioms, a unique linguistic phenomenon in Chinese.
• To investigate how candidate idiom selection strategies and idiom representation methods affect model performance in cloze-style reading comprehension.
• To establish a benchmark for evaluating machine understanding of Chinese idioms, particularly their non-compositional and metaphorical meanings.
• To compare model performance against human performance and assess generalization across in-domain and out-of-domain data.
• To explore whether treating idioms as atomic semantic units or composing them from character embeddings yields better results in comprehension tasks.

Proposed method

• Constructed a large-scale dataset (ChID) with 581K passages and 729K blanks from news, novels, and essays, replacing idioms with blanks.
• Designed candidate idiom lists with varying difficulty, including synonyms, near-synonyms, and semantically similar distractors to assess model discrimination.
• Evaluated three state-of-the-art models—LM, AR, and SAR—on the dataset using different idiom representation methods: pre-trained idiom embeddings, average character embeddings, and MLP-composed embeddings.
• Used attention mechanisms to allow models to attend to context and blank representations simultaneously, improving context-aware prediction.
• Applied Fleiss’ kappa to measure inter-annotator agreement, confirming high annotation quality (kappa = 0.953).
• Compared model performance across in-domain (Dev, Test), out-of-domain (Out), and challenging (Ran, Sim) splits to assess generalization and difficulty.

Experimental results

Research questions

• RQ1 How does the design of candidate idiom sets—especially near-synonyms and semantically similar distractors—affect the difficulty of cloze-style comprehension tasks?
• RQ2 To what extent do different idiom representation methods (e.g., pre-trained embeddings vs. character-level composition) impact model performance on Chinese idiom cloze tasks?
• RQ3 How well do state-of-the-art models generalize to out-of-domain data, particularly when encountering low-frequency idioms?
• RQ4 What is the performance gap between human annotators and machine models in understanding figurative and non-compositional Chinese idioms?
• RQ5 Does treating idioms as atomic semantic units yield better results than composing them from character-level representations in cloze tasks?

Key findings

• Human performance on ChID is significantly higher than model performance, with a minimum gap of 14.6% (on Test) and a maximum of 23.3% (on Out), indicating substantial room for improvement.
• The AR model outperforms LM and SAR across all splits, achieving 72.4% on Test and 62.9% on Out, suggesting attention mechanisms that use blank representations twice improve performance.
• Using average character embeddings or MLP-composed embeddings for idioms leads to significant performance drops (p < 0.01) compared to pre-trained idiom embeddings, showing that treating idioms as atomic units is crucial.
• Models generalize poorly to out-of-domain data (Out), where performance drops from 72.4% (Test) to 62.9%, while humans remain stable (87.1% vs. 86.2%), indicating a generalization gap.
• The difficulty of cloze questions correlates positively with the similarity of candidate idioms, especially near-synonyms, which increases annotation difficulty and challenges model discrimination.
• Inter-annotator agreement (Fleiss’ kappa = 0.953) is high, confirming the reliability of the dataset and the consistency of human judgments on complex idiom choices.