[Paper Review] Diachronic Usage Relatedness (DURel): A Framework for the Annotation of Lexical Semantic Change

Dominik Schlechtweg, Sabine Schulte im Walde+1 more|arXiv (Cornell University)|Apr 17, 2018

Natural Language Processing Techniques4 citations

TL;DR

This paper introduces DURel, a language-independent framework for annotating diachronic lexical semantic change by measuring semantic relatedness between word uses across time periods. Using a 4-point relatedness scale and inter-annotator agreement, it distinguishes innovative from reductive meaning change, achieving high reliability and enabling evaluation of computational models with a publicly available German test set of 1,320 use pairs across 22 words.

ABSTRACT

We propose a framework that extends synchronic polysemy annotation to diachronic changes in lexical meaning, to counteract the lack of resources for evaluating computational models of lexical semantic change. Our framework exploits an intuitive notion of semantic relatedness, and distinguishes between innovative and reductive meaning changes with high inter-annotator agreement. The resulting test set for German comprises ratings from five annotators for the relatedness of 1,320 use pairs across 22 target words.

Motivation & Objective

• To address the lack of reliable, large-scale test sets for evaluating computational models of lexical semantic change.
• To extend synchronic polysemy annotation to diachronic meaning change using an intuitive notion of semantic relatedness.
• To distinguish between innovative and reductive meaning changes with high inter-annotator agreement.
• To develop a language-independent annotation framework applicable across languages.
• To provide a publicly available test set for German to support evaluation of semantic change models.

Proposed method

• The framework uses a 4-point semantic relatedness scale (4: identical, 3: closely related, 2: distantly related, 1: unrelated, 0: cannot decide) to rate word use pairs across two time periods.
• It defines Diachronic Usage Relatedness (DURel) as the mean relatedness of use pairs within each time period, enabling comparison across periods.
• Two measures are proposed: Δlater, which computes the change in mean relatedness from earlier to later period to detect innovative or reductive change.
• The compare measure evaluates the degree of change by comparing relatedness in a 'compare' group to the earlier period, capturing shifts even when Δlater fails.
• Normalization of the compare measure is proposed to account for baseline polysemy, using early-period relatedness as a reference.
• The annotation study involved five annotators rating 1,320 use pairs across 22 German target words, with high inter-annotator agreement.

Experimental results

Research questions

• RQ1 Can a framework based on semantic relatedness reliably annotate diachronic lexical semantic change?
• RQ2 Can the framework distinguish between innovative and reductive meaning change with high inter-annotator agreement?
• RQ3 How do the proposed measures Δlater and compare perform in detecting various semantic change constellations?
• RQ4 In what cases do the measures fail, and how can they be improved through normalization?
• RQ5 Can the framework be applied across languages and used to build reliable test sets for semantic change evaluation?

Key findings

• The DURel framework achieved high inter-annotator agreement, validating its reliability for semantic change annotation.
• The Δlater measure correctly predicted reductive change (mean value 0.39) and innovative change (mean value -0.18), confirming its ability to distinguish between the two types.
• The Δlater measure failed to detect change in cases like Presse, where a new meaning had high prevalence, due to insufficient sensitivity to high-frequency new uses.
• The compare measure successfully captured changes in such cases, correctly identifying strong change for Presse, where Δlater failed.
• The compare measure confused polysemy with change, as seen in Feder, where high polysemy led to high compare scores despite stable meaning.
• Normalization of the compare measure is necessary to account for baseline polysemy, ensuring that stable polysemous words do not trigger false positives.