[Paper Review] A New Statistical Approach for Comparing Algorithms for Lexicon Based Sentiment Analysis

Mateus Tarcinalli Machado, Evandro Eduardo Seron Ruiz+1 more|arXiv (Cornell University)|Jun 20, 2019

Sentiment Analysis and Opinion Mining16 references4 citations

TL;DR

This paper proposes a statistical framework for comparing lexicon-based sentiment analysis algorithms without relying on human-annotated gold standards. Using marginal homogeneity tests and log-linear models with maximum likelihood estimation, it demonstrates that algorithm concordance varies significantly by label pair and lexicon, with neutral vs. positive/negative agreement being substantially lower than positive vs. negative, and LIWC showing significantly lower agreement than OpLex or SentiLex across all label pairs.

ABSTRACT

Lexicon based sentiment analysis usually relies on the identification of various words to which a numerical value corresponding to sentiment can be assigned. In principle, classifiers can be obtained from these algorithms by comparison with human annotation, which is considered the gold standard. In practise this is difficult in languages such as Portuguese where there is a paucity of human annotated texts. Thus in order to compare algorithms, a next best step is to directly compare different algorithms with each other without referring to human annotation. In this paper we develop methods for a statistical comparison of algorithms which does not rely on human annotation or on known class labels. We will motivate the use of marginal homogeneity tests, as well as log linear models within the framework of maximum likelihood estimation We will also show how some uncertainties present in lexicon based sentiment analysis may be similar to those which occur in human annotated tweets. We will also show how the variability in the output of different algorithms is lexicon dependent, and quantify this variability in the output within the framework of log linear models.

Motivation & Objective

• To develop a statistical method for comparing lexicon-based sentiment analysis algorithms without relying on human-annotated gold standards, which are scarce in resource-poor languages like Portuguese.
• To quantify the variability in algorithm outputs across different lexicons and label pairs, especially in the absence of known class labels.
• To assess whether observed differences in algorithm concordance are statistically significant or due to chance, using rigorous statistical inference.
• To extend existing comparisons of human annotators to machine algorithms, showing similar patterns of disagreement in label pairs involving neutral sentiment.
• To provide a methodological foundation for evaluating algorithm performance that is applicable beyond sentiment analysis, particularly in discrete, mutually exclusive labeling tasks.

Proposed method

• The authors use a contingency table to compare the outputs of two algorithms on the same text corpus, with each cell representing the number of texts assigned a specific label pair (e.g., algorithm 1 assigns 'n', algorithm 2 assigns 'p').
• They apply marginal homogeneity tests to assess whether the distribution of labels differs significantly between algorithms, testing for overall agreement beyond chance.
• Log-linear models with maximum likelihood estimation are used to model the joint distribution of algorithm outputs and estimate odds ratios between label pairs.
• The method computes 95% confidence intervals for log-odds ratios of agreement between label pairs (e.g., n vs. u, p vs. u, n vs. p) to assess statistical significance of differences in concordance.
• The approach is validated by comparing results with human annotator data from [16], showing consistent patterns in label pair disagreement.
• The method is applied across three lexicons—LIWC, OpLex, and SentiLex—to assess how algorithm concordance varies with lexicon choice.

Experimental results

Research questions

• RQ1 Is the concordance between two lexicon-based sentiment analysis algorithms significantly different from what would be expected by chance?
• RQ2 How does the degree of agreement between algorithms vary across different sentiment label pairs (e.g., n vs. p, n vs. u, p vs. u)?
• RQ3 Does the level of concordance between algorithms depend on the choice of lexicon used?
• RQ4 Are the observed differences in algorithm agreement statistically significant, or could they be due to sampling variability?
• RQ5 To what extent do the patterns of disagreement in algorithm outputs mirror those observed in human annotator disagreement?

Key findings

• The concordance between algorithms is significantly higher for the label pair n vs. p than for n vs. u or p vs. u, indicating that neutral sentiment is more ambiguous and harder to distinguish.
• For the (n,u) and (p,u) label pairs, the 95% confidence intervals for log-odds ratios of agreement do not overlap between LIWC and the other two lexicons (OpLex, SentiLex), indicating statistically significant differences in concordance.
• The log-odds ratio for agreement on (n,u) and (p,u) pairs is substantially lower for LIWC compared to OpLex and SentiLex, suggesting LIWC produces less consistent outputs on neutral sentiment.
• The results show that algorithm concordance is not only label-dependent but also lexicon-dependent, with LIWC consistently showing lower agreement than OpLex and SentiLex across all label pairs.
• The statistical patterns of disagreement between algorithms closely mirror those observed between human annotators in [16], particularly the higher disagreement on neutral-related pairs.
• The study confirms that even without gold-standard labels, statistical methods like marginal homogeneity tests and log-linear models can reliably detect and quantify algorithmic variability in sentiment analysis.