[Paper Review] A new ANEW: Evaluation of a word list for sentiment analysis in microblogs

Finn Årup Nielsen|arXiv (Cornell University)|Mar 15, 2011

Sentiment Analysis and Opinion Mining10 references739 citations

TL;DR

This paper introduces a new sentiment lexicon, AFINN-2477, specifically designed for microblog sentiment analysis, manually scored for valence on a -5 to +5 scale with inclusion of Internet slang and offensive words. Evaluated on 1,000 manually labeled Twitter posts via Amazon Mechanical Turk, it outperforms ANEW in correlation with human judgments (Pearson r = 0.564 vs. 0.525), though SentiStrength remains superior (r = 0.610), highlighting the benefit of domain-specific lexicons and advanced NLP techniques.

ABSTRACT

Sentiment analysis of microblogs such as Twitter has recently gained a fair amount of attention. One of the simplest sentiment analysis approaches compares the words of a posting against a labeled word list, where each word has been scored for valence, -- a 'sentiment lexicon' or 'affective word lists'. There exist several affective word lists, e.g., ANEW (Affective Norms for English Words) developed before the advent of microblogging and sentiment analysis. I wanted to examine how well ANEW and other word lists performs for the detection of sentiment strength in microblog posts in comparison with a new word list specifically constructed for microblogs. I used manually labeled postings from Twitter scored for sentiment. Using a simple word matching I show that the new word list may perform better than ANEW, though not as good as the more elaborate approach found in SentiStrength.

Motivation & Objective

• To develop a sentiment lexicon tailored for microblogging platforms like Twitter, where informal language, slang, and offensive terms are prevalent.
• To evaluate the performance of this new lexicon against established word lists such as ANEW, General Inquirer, and OpinionFinder in sentiment strength detection.
• To assess whether inclusion of Internet slang and strong negative words improves sentiment analysis accuracy on short, informal text.
• To investigate whether the performance difference between the new lexicon and ANEW stems from scoring quality or lexical coverage.

Proposed method

• Constructed a new sentiment lexicon (AFINN-2477) with 2,477 unique words and 15 phrases, manually scored on a -5 (very negative) to +5 (very positive) valence scale.
• Extended the lexicon iteratively using data from COP15 Twitter posts, public word lists (e.g., DeRose, Siegle), Urban Dictionary, Wiktionary, and Microsoft Web n-gram similarity service.
• Excluded ambiguous words (e.g., 'patient', 'mean') and high-arousal neutral words (e.g., 'surprise') to reduce noise.
• Evaluated performance using 1,000 Twitter posts manually labeled via Amazon Mechanical Turk, computing Pearson and Spearman correlations between automated scores and human labels.
• Compared the new lexicon against ANEW, General Inquirer, OpinionFinder, and SentiStrength using identical scoring methods and resampling techniques.
• Analyzed the intersection of ANEW and the new lexicon (299 words) to isolate the effect of scoring vs. lexical coverage by re-scoring with ANEW values.

Experimental results

Research questions

• RQ1 Does a sentiment lexicon specifically designed for microblogs outperform ANEW in sentiment strength detection on Twitter?
• RQ2 To what extent does the inclusion of Internet slang and offensive words improve sentiment analysis performance on informal text?
• RQ3 Is the performance gain of the new lexicon due to better word scoring or greater lexical coverage compared to ANEW?
• RQ4 How does the performance of the new lexicon evolve as it is incrementally expanded in size?
• RQ5 How does the new lexicon compare to SentiStrength, which uses advanced NLP techniques like negation and emoticon handling?

Key findings

• The new AFINN-2477 lexicon achieved a Pearson correlation of 0.564 with human judgments on 1,000 Twitter posts, outperforming ANEW’s 0.525.
• SentiStrength achieved the highest correlation (0.610), indicating that advanced NLP techniques still surpass simple word matching, even with a better lexicon.
• The General Inquirer word list performed poorly (Pearson r = 0.374), likely due to its polarity-only scoring, which lacks sentiment strength.
• The OpinionFinder word list performed better than General Inquirer (r = 0.458) but worse than both ANEW and the new lexicon, despite its larger size.
• Performance of the new lexicon improved steadily with expansion, as shown in Figure 4, suggesting further gains are possible with continued growth.
• Analysis of the 299-word intersection between ANEW and the new lexicon showed that ANEW’s scoring was superior, indicating that the performance gain of the new lexicon is due to lexical coverage, not scoring quality.