[Paper Review] Using Information Content to Evaluate Semantic Similarity in a Taxonomy

Philip Resnik|ArXiv.org|Nov 29, 1995

Topic ModelingComputer Science18 references2,149 citations

TL;DR

This paper introduces an information-content-based measure of semantic similarity in is-a taxonomies, evaluates it against human judgments, and compares it to edge-counting and other baselines. It shows notable improvements over edge counting but discusses limitations and extensions.

ABSTRACT

This paper presents a new measure of semantic similarity in an IS-A taxonomy, based on the notion of information content. Experimental evaluation suggests that the measure performs encouragingly well (a correlation of r = 0.79 with a benchmark set of human similarity judgments, with an upper bound of r = 0.90 for human subjects performing the same task), and significantly better than the traditional edge counting approach (r = 0.66).

Motivation & Objective

• Motivate a robust, context-sensitive measure of semantic similarity in taxonomies beyond simple edge counting.
• Define an information-theoretic similarity metric using concept probabilities in a taxonomy.
• Evaluate the metric against human similarity judgments and benchmark it against baseline methods.

Proposed method

• Augment a taxonomy with a probability function p(c) for each concept based on corpus frequencies.
• Define information content as -log p(c) and similarity as the maximum information content among common subsumers of two concepts (and words).
• For words, compute similarity by maximizing over all sense pairs, w1 in s(w1) and w2 in s(w2).
• Compare with edge-based similarity and a p(c)-based similarity, using correlations with human judgments as the evaluation metric.

Experimental results

Research questions

• RQ1 Can an information-content-based measure of semantic similarity in an is-a taxonomy correlate with human judgments of word similarity?
• RQ2 How does the information-content method compare to edge-counting and other baselines in performance?
• RQ3 What are the limitations of single-sense maximal-information summaries, and can a multi-sense weighted approach mitigate them?

Key findings

• Information-content similarity achieves a correlation of 0.7911 with human judgments, outperforming edge counting (0.6645) and p(c) based similarity (0.6671).
• Edge counting is significantly less aligned with human judgments than the information-content approach.
• Replication with human subjects yielded an upper-bound correlation around 0.90 for humans performing the same task.
• A noted limitation is spuriously high similarity due to inappropriate senses, motivating a multi-sense weighted extension (Eq. 8).