[Paper Review] COS960: A Chinese Word Similarity Dataset of 960 Word Pairs

Junjie Huang, Fanchao Qi+3 more|arXiv (Cornell University)|Jun 1, 2019

Natural Language Processing Techniques28 references10 citations

TL;DR

This paper introduces COS960, a Chinese word similarity dataset comprising 960 pairs of two-morpheme multiword expressions (MWEs) annotated for true similarity by 15 native speakers. The dataset, constructed using HowNet and GloVe embeddings, enables evaluation of word embeddings on semantic similarity in Chinese, with CBOW achieving the highest correlation (Spearman's ρ = 78.2) across all pairs.

ABSTRACT

Word similarity computation is a widely recognized task in the field of lexical semantics. Most proposed tasks test on similarity of word pairs of single morpheme, while few works focus on words of two morphemes or more morphemes. In this work, we propose COS960, a benchmark dataset with 960 pairs of Chinese wOrd Similarity, where all the words have two morphemes in three Part of Speech (POS) tags with their human annotated similarity rather than relatedness. We give a detailed description of dataset construction and annotation process, and test on a range of word embedding models. The dataset of this paper can be obtained from https://github.com/thunlp/COS960.

Motivation & Objective

• To address the lack of Chinese word similarity datasets focused on true similarity rather than relatedness.
• To create a benchmark for evaluating word embeddings on multiword expressions (MWEs) in Chinese, particularly two-morpheme compounds.
• To improve the evaluation of distributional semantics models in Chinese by providing a high-quality, human-annotated dataset.
• To assess the performance of various pre-trained word embedding models on Chinese semantic similarity using the proposed dataset.

Proposed method

• Constructed a dataset of 51,034 two-morpheme words from HowNet, filtered by part-of-speech (POS) tags: noun, verb, and adjective.
• Generated word pairs by pairing words within each POS group and filtering based on GloVe embedding cosine similarity in the range [0.4, 1.0].
• Annotated 960 word pairs (480 noun, 240 verb, 240 adjective pairs) using 30 native speakers, with similarity scores on a 0–4 scale based on true semantic similarity.
• Ensured annotation quality through a pre-annotation exam, discussion protocols, and Krippendorff’s alpha inter-annotator agreement checks.
• Used the average of 15 annotator scores as the final gold standard similarity score for each pair.
• Evaluated six word embedding models (Skip-Gram, CBOW, GloVe, CWE, fasttext, cw2vec) using Pearson and Spearman correlation with human scores.

Experimental results

Research questions

• RQ1 How well do existing word embedding models capture true semantic similarity in Chinese multiword expressions?
• RQ2 What is the performance gap between models on different POS categories (nouns, verbs, adjectives) in Chinese word similarity?
• RQ3 To what extent does the use of MWEs with two morphemes improve the evaluation of distributional semantics in Chinese?
• RQ4 How consistent are human annotations of semantic similarity across different word pairs and POS tags?
• RQ5 Does the inclusion of subword or character-level information in embeddings improve performance on this semantic similarity benchmark?

Key findings

• CBOW achieved the highest overall performance on COS960 with a Spearman’s rank correlation of 78.2, outperforming Skip-Gram by 2.1 percentage points.
• All six evaluated word embedding models showed high correlation with human judgments, indicating strong alignment with true semantic similarity in Chinese.
• Performance was highest on verb pairs (Spearman’s ρ = 84.8 for CBOW), followed by adjectives (ρ = 78.5 for CBOW), and lowest on nouns (ρ = 77.0 for CBOW).
• The evaluation protocol using Spearman’s rank correlation yielded the highest scores across all models, suggesting strong annotation consistency, especially within similarity bins.
• The dataset revealed that models like fasttext and cw2vec achieved competitive results (Spearman’s ρ > 75) despite not being specifically trained on Chinese, indicating transferability.
• The results contradict earlier claims (e.g., Hill et al., 2015) that word embeddings poorly capture true similarity, as high correlations were observed on this Chinese MWE-based dataset.