[Paper Review] A Manually Annotated Chinese Corpus for Non-task-oriented Dialogue Systems

Jing Li, Yan Song+2 more|arXiv (Cornell University)|May 15, 2018

Speech and dialogue systems17 references4 citations

TL;DR

This paper introduces the first manually annotated Chinese corpus for non-task-oriented dialogue systems, comprising over 27,000 prompts and 82,000 response pairs rated on five quality levels (bad to excellent) for relevance, coherence, informativeness, and interestingness. The corpus enables superior performance in response selection, with supervised models outperforming unsupervised baselines and demonstrating robustness under stricter quality thresholds.

ABSTRACT

This paper presents a large-scale corpus for non-task-oriented dialogue response selection, which contains over 27K distinct prompts more than 82K responses collected from social media. To annotate this corpus, we define a 5-grade rating scheme: bad, mediocre, acceptable, good, and excellent, according to the relevance, coherence, informativeness, interestingness, and the potential to move a conversation forward. To test the validity and usefulness of the produced corpus, we compare various unsupervised and supervised models for response selection. Experimental results confirm that the proposed corpus is helpful in training response selection models.

Motivation & Objective

• To address the lack of high-quality, manually annotated training data for non-task-oriented Chinese dialogue systems.
• To reduce the prevalence of low-quality, generic responses (e.g., 'me too') in training data by introducing quality-aware annotations.
• To establish a benchmark dataset for training and evaluating response selection models in Chinese dialogue systems.
• To validate the corpus's utility through comparative experiments on diverse response selection models.
• To improve the reliability of dialogue systems by distinguishing responses of varying quality, especially excellent vs. mediocre or versatile responses.

Proposed method

• Collected over 27,000 distinct Chinese prompts and 82,000 response pairs from social media platforms including Baidu Tieba, Zhihu, Douban, and Sina Weibo.
• Defined a five-grade rating scheme—bad, mediocre, acceptable, good, excellent—based on relevance, coherence, informativeness, and conversational potential.
• Employed multiple annotators with inter-annotator agreement measured via Cohen’s kappa (80.0%), ensuring high annotation quality.
• Preprocessed text using Jieba for Chinese tokenization and split data into 80% training, 10% development, and 10% test sets.
• Evaluated a range of models: unsupervised (Cosine Sim, BM25), supervised (SVMRank, GBDT with handcrafted features), and neural models (BiLSTM, CNN) for response ranking.
• Used precision@1 (P@1), mean average precision (MAP), and mean reciprocal rank (MRR) as evaluation metrics, with gold-standard ratings used to define positive/negative response sets.

Experimental results

Research questions

• RQ1 Can a manually annotated Chinese corpus with multi-level quality ratings improve response selection in non-task-oriented dialogue systems?
• RQ2 How do supervised models trained on this corpus compare to unsupervised baselines in selecting high-quality responses?
• RQ3 To what extent can the corpus distinguish between excellent responses and generic or irrelevant ones?
• RQ4 Does the corpus support robust performance across varying quality thresholds (e.g., rating ≥3 vs. ≥5)?
• RQ5 How effective is the five-grade rating scheme in capturing nuanced differences in response quality?

Key findings

• The corpus contains 27,000 distinct prompts and 82,000 prompt-response pairs, with 48.9% of responses rated as 'acceptable' (rating 3), indicating a high prevalence of versatile or context-limited responses.
• 35.0% of responses received a rating ≤2.5, confirming the noisy and low-quality nature of raw social media data.
• Supervised models (SVMRank, GBDT, BiLSTM, CNN) outperformed unsupervised models (Cosine Sim, BM25) across all metrics, demonstrating the corpus’s value for training.
• As the quality threshold increased (e.g., from ≥3 to ≥5), unsupervised models suffered larger performance drops, while supervised models maintained robust performance, indicating better quality discrimination.
• The corpus enables effective ranking of responses, with excellent responses consistently assigned higher scores than good or acceptable ones, validating its utility for high-quality response selection.
• The inter-annotator agreement (Cohen’s kappa = 80.0%) confirms the reliability and consistency of the five-grade annotation scheme.