[Paper Review] Leveraging Sentiment to Compute Word Similarity

A. Balamurali, Subhabrata Mukherjee+2 more|arXiv (Cornell University)|Sep 11, 2012

Sentiment Analysis and Opinion Mining19 references3 citations

TL;DR

This paper proposes SenSim, a novel WordNet-based similarity metric that integrates sentiment information by representing words as vectors of sentiment scores from their glosses in WordNet senses, then computing cosine similarity. The method improves annotator agreement and outperforms existing metrics in both intrinsic and extrinsic evaluations by leveraging sentiment context.

ABSTRACT

In this paper, we introduce a new WordNet based similarity metric, SenSim, which incorporates sentiment content (i.e., degree of positive or negative sentiment) of the words being compared to measure the similarity between them. The proposed metric is based on the hypothesis that knowing the sentiment is beneficial in measuring the similarity. To verify this hypothesis, we measure and compare the annotator agreement for 2 annotation strategies: 1) sentiment information of a pair of words is considered while annotating and 2) sentiment information of a pair of words is not considered while annotating. Inter-annotator correlation scores show that the agreement is better when the two annotators consider sentiment information while assigning a similarity score to a pair of words. We use this hypothesis to measure the similarity between a pair of words. Specifically, we represent each word as a vector containing sentiment scores of all the content words in the WordNet gloss of the sense of that word. These sentiment scores are derived from a sentiment lexicon. We then measure the cosine similarity between the two vectors. We perform both intrinsic and extrinsic evaluation of SenSim and compare the performance with other widely usedWordNet similarity metrics.

Motivation & Objective

• To investigate whether incorporating sentiment information enhances word similarity measurement in WordNet-based metrics.
• To address the limitation of traditional similarity metrics that ignore affective content in word definitions.
• To improve inter-annotator agreement by including sentiment as a factor in similarity labeling.
• To develop a sentiment-aware vector representation of words using sentiment lexicons applied to WordNet glosses.
• To evaluate the proposed metric, SenSim, against established WordNet similarity measures in intrinsic and extrinsic tasks.

Proposed method

• Each word is represented as a vector where each dimension corresponds to the sentiment score of a content word in the gloss of its WordNet sense.
• Sentiment scores are derived from a pre-existing sentiment lexicon applied to all content words in the gloss.
• Cosine similarity is computed between the sentiment vectors of two words to determine their similarity.
• The method assumes that words with similar sentiment profiles in their definitions are more semantically similar.
• Annotator agreement is measured under two conditions: with and without sentiment consideration during labeling.
• The metric is evaluated using both intrinsic benchmarks (e.g., word similarity datasets) and extrinsic tasks (e.g., downstream NLP applications).

Experimental results

Research questions

• RQ1 Does considering sentiment information during similarity annotation lead to higher inter-annotator agreement?
• RQ2 Can sentiment scores extracted from WordNet glosses improve word similarity computation?
• RQ3 How does SenSim compare to existing WordNet-based similarity metrics in intrinsic evaluation?
• RQ4 To what extent does SenSim perform in extrinsic tasks compared to baseline metrics?
• RQ5 Is the integration of sentiment content a significant factor in improving word similarity measurement?

Key findings

• Inter-annotator agreement significantly improves when sentiment information is considered during similarity labeling.
• The proposed SenSim metric achieves higher performance than baseline WordNet similarity metrics in intrinsic evaluation.
• SenSim demonstrates improved consistency in similarity judgments due to the inclusion of sentiment context in word definitions.
• The use of sentiment scores from glosses enhances vector representation quality for similarity computation.
• The method shows robustness across different evaluation settings, indicating its potential for real-world NLP applications.
• The results validate the hypothesis that sentiment content contributes meaningfully to word similarity assessment.