[Paper Review] Supervised Term Weighting Metrics for Sentiment Analysis in Short Text

Hussam Hamdan, Patrice Bellot+1 more|arXiv (Cornell University)|Oct 10, 2016

Text and Document Classification Technologies18 references3 citations

TL;DR

This paper evaluates 15 supervised global term weighting metrics and 4 local metrics for sentiment analysis in short text, using SVM on Twitter, restaurant, and laptop review datasets. It finds that most metrics improve binary baseline performance, with metrics like rf, kl, and wllr consistently effective; good metrics exhibit broad distributions and non-zero means, while poor ones show narrow, near-zero distributions.

ABSTRACT

Term weighting metrics assign weights to terms in order to discriminate the important terms from the less crucial ones. Due to this characteristic, these metrics have attracted growing attention in text classification and recently in sentiment analysis. Using the weights given by such metrics could lead to more accurate document representation which may improve the performance of the classification. While previous studies have focused on proposing or comparing different weighting metrics at two-classes document level sentiment analysis, this study propose to analyse the results given by each metric in order to find out the characteristics of good and bad weighting metrics. Therefore we present an empirical study of fifteen global supervised weighting metrics with four local weighting metrics adopted from information retrieval, we also give an analysis to understand the behavior of each metric by observing and analysing how each metric distributes the terms and deduce some characteristics which may distinguish the good and bad metrics. The evaluation has been done using Support Vector Machine on three different datasets: Twitter, restaurant and laptop reviews.

Motivation & Objective

• To understand why certain supervised term weighting metrics outperform others in sentiment analysis of short text.
• To analyze the distributional behavior of global and local term weighting metrics across different datasets.
• To identify characteristics that distinguish high-performing from low-performing term weighting metrics.
• To evaluate the impact of combining local and global weighting schemes on classification performance.
• To provide empirical insights into metric effectiveness beyond reported F1 scores, focusing on underlying distribution patterns.

Proposed method

• Empirically evaluates 15 global supervised term weighting metrics and 4 local weighting metrics on three benchmark datasets: Twitter, restaurant reviews, and laptop reviews.
• Uses Support Vector Machine (SVM) as the classifier for all experiments to ensure consistent evaluation across metrics.
• Applies standard vector space model (VSM) representation with binary, tf, atf, logtf, and sumstd/local normalization for local weighting.
• Measures performance using F1-score across all combinations of local and global metrics to assess relative effectiveness.
• Analyzes the distribution of term weights across the corpus for each global metric, focusing on mean, standard deviation, and skewness.
• Compares results across datasets to identify consistent performers and assess generalizability of metric behavior.

Experimental results

Research questions

• RQ1 Which global and local term weighting metrics consistently improve sentiment classification performance across diverse short-text datasets?
• RQ2 How do the statistical distributions (mean, variance, skewness) of term weights relate to the performance of a given metric?
• RQ3 What characteristics distinguish high-performing term weighting metrics from low-performing ones in short-text sentiment analysis?
• RQ4 To what extent do local weighting schemes influence the overall performance when combined with global metrics?
• RQ5 Can consistent patterns in metric behavior be identified across different domains (e.g., Twitter vs. product reviews)?

Key findings

• The binary baseline (bl*tp) achieved 69.33% F1 on the laptop dataset, with most global metrics improving upon this baseline.
• The metric dsidf achieved the highest F1 score of 72.02% on the laptop dataset when combined with atf local weighting.
• The metric ne achieved 71.71% F1 on the laptop dataset with atf, ranking among the top-performing combinations.
• Metrics such as rf, kl, and wllr showed consistently strong performance across all three datasets, indicating robustness.
• Poor-performing metrics were characterized by narrow distributions and means close to zero, suggesting limited discriminative power.
• Local metrics like tf and atf significantly improved performance when combined with effective global metrics, especially on Twitter and restaurant datasets.