[Paper Review] Creation and Evaluation of Datasets for Distributional Semantics Tasks in the Digital Humanities Domain

Gerhard Wohlgenannt, Ariadna Barinova+2 more|arXiv (Cornell University)|Mar 7, 2019

Topic Modeling38 references4 citations

TL;DR

This paper introduces high-quality, manually curated datasets for distributional semantics tasks—word analogy and word intrusion—using two fantasy novel series, 'A Song of Ice and Fire' and 'Harry Potter'. It evaluates various word embedding models (word2vec, GloVe, fastText, LexVec) on these datasets, showing that even small-corpora-trained embeddings perform well on word intrusion, with CBOW models excelling on the hardest categories, and providing open-source resources for reproducibility and future benchmarking in digital humanities NLP.

ABSTRACT

Word embeddings are already well studied in the general domain, usually trained on large text corpora, and have been evaluated for example on word similarity and analogy tasks, but also as an input to downstream NLP processes. In contrast, in this work we explore the suitability of word embedding technologies in the specialized digital humanities domain. After training embedding models of various types on two popular fantasy novel book series, we evaluate their performance on two task types: term analogies, and word intrusion. To this end, we manually construct test datasets with domain experts. Among the contributions are the evaluation of various word embedding techniques on the different task types, with the findings that even embeddings trained on small corpora perform well for example on the word intrusion task. Furthermore, we provide extensive and high-quality datasets in digital humanities for further investigation, as well as the implementation to easily reproduce or extend the experiments.

Motivation & Objective

• To evaluate the performance of word embeddings trained on small corpora in the specialized digital humanities domain, particularly in literary texts.
• To create high-quality, expert-annotated test datasets for distributional semantics tasks in the context of fantasy literature.
• To establish reproducible baselines for future research on word embeddings in domain-specific, low-resource settings.
• To investigate the impact of hyperparameters, term frequency, and named entity handling on embedding performance.
• To provide open-source datasets, trained models, and evaluation code to support extensibility and benchmarking.

Proposed method

• Trained multiple word embedding models (word2vec, GloVe, fastText, LexVec) on two fantasy novel corpora: 'A Song of Ice and Fire' and 'Harry Potter'.
• Created two distinct task types: word analogy (e.g., husband:wife, sigil:house) and word intrusion (identifying the odd-one-out in a set of four words).
• Constructed datasets using domain experts to ensure high-quality ground truth annotations for both task types.
• Generated eight evaluation datasets by combining book series, task types, and unigram/n-gram models to capture named entities like 'Many Faced God'.
• Used baseline methods including PPMI and pre-trained embeddings for comparison, and performed grid search for hyperparameter tuning.
• Shared all datasets, trained models, and evaluation code via GitHub to ensure reproducibility and extensibility.

Experimental results

Research questions

• RQ1 How well do word embeddings trained on small literary corpora perform on word analogy and word intrusion tasks in the digital humanities domain?
• RQ2 Which word embedding technique (e.g., word2vec, GloVe, fastText) performs best on each task type, and how do hyperparameters like window size affect performance?
• RQ3 How does the frequency of task terms in the corpus influence the accuracy of word embedding models?
• RQ4 How do named entities and multi-word expressions affect embedding quality, and what role do n-grams play in capturing them?
• RQ5 To what extent do word embeddings outperform traditional distributional models like PPMI on these domain-specific tasks?

Key findings

• Even embeddings trained on relatively small corpora achieve strong performance on the word intrusion task, with CBOW-based models outperforming others on the most challenging difficulty categories.
• GloVe models showed strong performance on analogy tasks, while word2vec was more effective for word similarity, confirming prior findings in general-domain evaluations.
• Large word windows (potentially crossing sentence boundaries) were beneficial for analogical reasoning, whereas small windows improved performance on word intrusion tasks.
• Term frequency had a significant impact: embeddings of terms appearing more than 100 times in the text showed drastically improved quality, with the frequency of the predicted term being more important than the correct answer's frequency.
• PPMI performed comparably to some embedding models, particularly on the word intrusion task, but word embeddings generally outperformed PPMI across both book series and task types.
• The study provides a comprehensive, open-source resource suite including 31,474 evaluation questions across 16,220 (ASOIF) and 15,254 (HP) questions, with trained models and code for replication and extension.