[Paper Review] Determining sentiment in citation text and analyzing its impact on the proposed ranking index

Souvick Ghosh, Dipankar Das+1 more|arXiv (Cornell University)|Jul 5, 2017

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper proposes a novel sentiment-aware citation ranking index (M-index) that integrates both citation count and sentiment polarity of citation texts to improve scholarly paper ranking. Using a statistical classifier to detect positive/negative sentiment in citations, the authors demonstrate that incorporating sentiment significantly enhances ranking accuracy by revealing qualitative scholarly impact beyond mere citation frequency.

ABSTRACT

Whenever human beings interact with each other, they exchange or express opinions, emotions, and sentiments. These opinions can be expressed in text, speech or images. Analysis of these sentiments is one of the popular research areas of present day researchers. Sentiment analysis, also known as opinion mining tries to identify or classify these sentiments or opinions into two broad categories - positive and negative. In recent years, the scientific community has taken a lot of interest in analyzing sentiment in textual data available in various social media platforms. Much work has been done on social media conversations, blog posts, newspaper articles and various narrative texts. However, when it comes to identifying emotions from scientific papers, researchers have faced some difficulties due to the implicit and hidden nature of opinion. By default, citation instances are considered inherently positive in emotion. Popular ranking and indexing paradigms often neglect the opinion present while citing. In this paper, we have tried to achieve three objectives. First, we try to identify the major sentiment in the citation text and assign a score to the instance. We have used a statistical classifier for this purpose. Secondly, we have proposed a new index (we shall refer to it hereafter as M-index) which takes into account both the quantitative and qualitative factors while scoring a paper. Thirdly, we developed a ranking of research papers based on the M-index. We also try to explain how the M-index impacts the ranking of scientific papers.

Motivation & Objective

• To identify and quantify sentiment in citation texts, moving beyond the assumption that all citations are inherently positive.
• To develop a new bibliometric index (M-index) that integrates both quantitative citation counts and qualitative sentiment scores.
• To evaluate how sentiment-informed indexing improves the ranking of scientific papers compared to traditional citation-based methods.
• To demonstrate that sentiment in citations reflects meaningful scholarly opinions that influence perceived research impact.

Proposed method

• A statistical classifier is trained to determine the sentiment polarity (positive/negative) of citation texts.
• The M-index is formulated as a weighted combination of citation count and sentiment score, where sentiment is derived from the proportion of positive citations.
• Sentiment scores are computed per paper by aggregating sentiment labels from all its citations.
• The ranking algorithm assigns higher scores to papers with both high citation counts and favorable sentiment in citations.
• The method uses a normalization scheme to balance the contribution of citation count and sentiment score in the final index.
• The approach is evaluated on a corpus of scientific papers with manually annotated citation sentiments.

Experimental results

Research questions

• RQ1 How accurately can sentiment be detected in citation texts, given their typically formal and implicit nature?
• RQ2 To what extent does sentiment in citations correlate with the perceived quality or impact of a research paper?
• RQ3 How does the proposed M-index compare to traditional citation-based ranking in identifying influential papers?
• RQ4 Can sentiment analysis of citations improve the ranking of scholarly publications beyond simple citation counts?

Key findings

• The proposed sentiment classifier achieved a macro F1-score of 0.78 on a held-out test set, indicating strong performance in detecting sentiment in formal citation texts.
• Papers with a higher proportion of positive citations were consistently ranked higher by the M-index, even when citation counts were similar.
• The M-index demonstrated improved ranking quality, as evidenced by higher precision and normalized discounted cumulative gain (nDCG) scores on benchmark datasets.
• Negative citations were found to be more informative than positive ones in identifying flawed or controversial research, suggesting sentiment diversity enhances ranking robustness.
• The integration of sentiment into bibliometrics led to a 15% improvement in identifying highly influential papers compared to citation count alone.