[Paper Review] Article citation study: Context enhanced citation sentiment detection

Vishal Vyas, Kumar Ravi+4 more|arXiv (Cornell University)|May 10, 2020

Topic Modeling28 references4 citations

TL;DR

This paper proposes a context-enhanced citation sentiment detection method using ensemble features from word embeddings, part-of-speech tags, and dependency parsing to improve sentiment classification in academic citations. Experiments show that deep learning models outperform traditional Bag-of-Words on larger datasets, while support vector machines perform better on smaller ones, and context-aware samples significantly improve performance over context-agnostic approaches across eight manually annotated datasets with three sentiment classes.

ABSTRACT

Citation sentimet analysis is one of the little studied tasks for scientometric analysis. For citation analysis, we developed eight datasets comprising citation sentences, which are manually annotated by us into three sentiment polarities viz. positive, negative, and neutral. Among eight datasets, three were developed by considering the whole context of citations. Furthermore, we proposed an ensembled feature engineering method comprising word embeddings obtained for texts, parts-of-speech tags, and dependency relationships together. Ensembled features were considered as input to deep learning based approaches for citation sentiment classification, which is in turn compared with Bag-of-Words approach. Experimental results demonstrate that deep learning is useful for higher number of samples, whereas support vector machine is the winner for smaller number of samples. Moreover, context-based samples are proved to be more effective than context-less samples for citation sentiment analysis.

Motivation & Objective

• To address the underexplored task of citation sentiment analysis in scientometric research.
• To develop eight manually annotated citation datasets with three sentiment polarities: positive, negative, and neutral.
• To investigate the impact of contextual information on citation sentiment classification performance.
• To compare deep learning and traditional machine learning (e.g., SVM) approaches for citation sentiment classification.
• To evaluate the effectiveness of ensemble features combining word embeddings, POS tags, and dependency parsing.

Proposed method

• The authors created eight citation sentiment datasets, three of which were annotated with full contextual information.
• They extracted ensemble features combining pre-trained word embeddings, part-of-speech (POS) tags, and dependency parsing relations.
• These features were used as input to deep learning models (e.g., BiLSTM, CNN) and compared against a Bag-of-Words baseline.
• The sentiment classification task was trained and evaluated using both deep learning and traditional machine learning models (e.g., SVM).
• The study employed a comparative evaluation framework to assess performance across varying dataset sizes and context availability.
• The final model architecture combined multiple linguistic features to enhance representation learning for citation sentiment.

Experimental results

Research questions

• RQ1 How does the inclusion of contextual information affect citation sentiment classification performance?
• RQ2 Which machine learning approach—deep learning or traditional models like SVM—performs better on citation sentiment classification across different dataset sizes?
• RQ3 To what extent do ensemble features combining word embeddings, POS tags, and dependency parsing improve classification accuracy?
• RQ4 How do the performance characteristics of deep learning and SVM models vary with training sample size in citation sentiment analysis?
• RQ5 What is the relative effectiveness of context-enhanced versus context-agnostic citation samples in sentiment classification?

Key findings

• Deep learning models achieve superior performance on larger citation datasets compared to the Bag-of-Words baseline.
• Support vector machines outperform deep learning models on smaller citation datasets, indicating a sample-size-dependent performance trade-off.
• Context-enhanced citation samples significantly improve classification accuracy over context-agnostic samples, demonstrating the importance of surrounding text in sentiment determination.
• The ensemble feature approach—combining word embeddings, POS tags, and dependency parsing—leads to better representation learning and improved sentiment classification.
• The proposed method achieves higher F1-scores on the manually annotated datasets, especially when full citation context is used.
• The study confirms that citation sentiment analysis is a viable and valuable task for scientometric and digital library applications when properly contextualized.