[Paper Review] Effect of Text Processing Steps on Twitter Sentiment Classification using Word Embedding

Manar D. Samad, Nalin D. Khounviengxay+1 more|arXiv (Cornell University)|Jul 25, 2020

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper investigates the impact of text processing steps on Twitter sentiment classification using skip-gram word embeddings trained on domain-specific Twitter data. It proposes a data-driven approach to word importance and tweet vector aggregation, showing that removing stop words and averaging word vectors harms performance; instead, summing word vectors and including emojis and context-aware word importance boosts AUC to 88.4±0.4.

ABSTRACT

Processing of raw text is the crucial first step in text classification and sentiment analysis. However, text processing steps are often performed using off-the-shelf routines and pre-built word dictionaries without optimizing for domain, application, and context. This paper investigates the effect of seven text processing scenarios on a particular text domain (Twitter) and application (sentiment classification). Skip gram-based word embeddings are developed to include Twitter colloquial words, emojis, and hashtag keywords that are often removed for being unavailable in conventional literature corpora. Our experiments reveal negative effects on sentiment classification of two common text processing steps: 1) stop word removal and 2) averaging of word vectors to represent individual tweets. New effective steps for 1) including non-ASCII emoji characters, 2) measuring word importance from word embedding, 3) aggregating word vectors into a tweet embedding, and 4) developing linearly separable feature space have been proposed to optimize the sentiment classification pipeline. The best combination of text processing steps yields the highest average area under the curve (AUC) of 88.4 (+/-0.4) in classifying 14,640 tweets with three sentiment labels. Word selection from context-driven word embedding reveals that only the ten most important words in Tweets cumulatively yield over 98% of the maximum accuracy. Results demonstrate a means for data-driven selection of important words in tweet classification as opposed to using pre-built word dictionaries. The proposed tweet embedding is robust to and alleviates the need for several text processing steps.

Motivation & Objective

• To evaluate the effect of common text processing steps on Twitter sentiment classification using word embeddings.
• To identify optimal text preprocessing strategies tailored to the Twitter domain, which features informal language, abbreviations, and emojis.
• To develop a data-driven method for selecting important words in tweets based on their learned word embedding representations.
• To propose robust tweet embedding techniques that reduce reliance on standard text processing steps.
• To achieve high-performance sentiment classification by optimizing word vector aggregation and feature space construction.

Proposed method

• Training skip-gram word embeddings on a large Twitter corpus to capture domain-specific semantics, including colloquial words, emojis, and hashtags.
• Measuring word importance using the magnitude of word embedding vectors, enabling data-driven stop word selection instead of relying on pre-built dictionaries.
• Aggregating individual word vectors into a tweet-level representation using sum, average, and weighted average operations.
• Applying 10x2 nested cross-validation to ensure unbiased evaluation of model performance.
• Constructing a linearly separable feature space from word embeddings to improve classifier generalization.
• Including non-ASCII emoji characters directly in the embedding process to preserve sentiment-relevant signals.

Experimental results

Research questions

• RQ1 How do common text processing steps—such as stop word removal and word vector averaging—affect sentiment classification performance on Twitter?
• RQ2 What is the optimal method for aggregating word vectors into a tweet-level representation for sentiment classification?
• RQ3 Can word embeddings trained on Twitter data be used to identify contextually important words, reducing reliance on fixed stop word lists?
• RQ4 How does the inclusion of emojis and non-ASCII characters impact sentiment classification accuracy?
• RQ5 To what extent can a data-driven, embedding-based approach reduce the need for aggressive text preprocessing in Twitter sentiment analysis?

Key findings

• Stop word removal significantly degrades sentiment classification performance, as even low-importance words contribute predictive information via their embeddings.
• Averaging word vectors for tweet representation leads to worse performance than summing them, with the sum yielding the highest AUC of 88.4±0.4.
• Including non-ASCII emoji characters in the word embedding process improves classification accuracy, as emojis carry sentiment-relevant signals.
• The ten most important words in a tweet, as determined by embedding magnitude, cumulatively account for over 98% of maximum classification accuracy.
• Word embeddings trained on Twitter data create a linearly separable feature space, enabling efficient and generalizable sentiment classification.
• The proposed method reduces dependency on standard text preprocessing steps, demonstrating robustness even when some words are out-of-vocabulary.