[Paper Review] SentimentArcs: A Novel Method for Self-Supervised Sentiment Analysis of Time Series Shows SOTA Transformers Can Struggle Finding Narrative Arcs

Jon Chun|arXiv (Cornell University)|Oct 18, 2021

Data Visualization and Analytics4 citations

TL;DR

This paper introduces SentimentArcs, a self-supervised method for time series sentiment analysis of long-form narratives that uses a large ensemble of diverse models to generate synthetic ground truth and jointly optimizes over all corpus:model combinations. It demonstrates that even state-of-the-art Transformers underperform on narrative texts, highlighting the need for human-in-the-loop validation and exhaustive model selection to achieve reliable sentiment analysis beyond narrow benchmarks.

ABSTRACT

SOTA Transformer and DNN short text sentiment classifiers report over 97% accuracy on narrow domains like IMDB movie reviews. Real-world performance is significantly lower because traditional models overfit benchmarks and generalize poorly to different or more open domain texts. This paper introduces SentimentArcs, a new self-supervised time series sentiment analysis methodology that addresses the two main limitations of traditional supervised sentiment analysis: limited labeled training datasets and poor generalization. A large ensemble of diverse models provides a synthetic ground truth for self-supervised learning. Novel metrics jointly optimize an exhaustive search across every possible corpus:model combination. The joint optimization over both the corpus and model solves the generalization problem. Simple visualizations exploit the temporal structure in narratives so domain experts can quickly spot trends, identify key features, and note anomalies over hundreds of arcs and millions of data points. To our knowledge, this is the first self-supervised method for time series sentiment analysis and the largest survey directly comparing real-world model performance on long-form narratives.

Motivation & Objective

• To address the poor generalization of supervised sentiment models on long-form narratives due to domain shift and limited labeled data.
• To overcome the limitations of narrow benchmark performance by introducing a self-supervised framework that generates synthetic ground truth from diverse models.
• To enable scalable, human-in-the-loop analysis of narrative sentiment through visualizations and joint corpus:model optimization.
• To empirically demonstrate that SOTA Transformers fail to capture narrative arcs in long-form texts, often underperforming simpler models.
• To provide a systematic methodology for selecting high-performing models across diverse corpora using novel joint metrics.

Proposed method

• A large ensemble of 36 diverse sentiment models (including BERT, RoBERTa, XGBoost, and Naive Bayes) is used to generate a synthetic ground truth for sentiment time series.
• An exhaustive search over all possible corpus:model combinations is performed using novel joint metrics that optimize for both corpus-specific performance and model stability.
• Dynamic Time Warping (DTW) hierarchical clustering is applied to group narrative arcs based on sentiment trajectory similarity.
• Visualizations of sentiment arcs are generated to enable human experts to rapidly identify trends, anomalies, and key narrative features.
• The method integrates human-in-the-loop validation by allowing domain experts to verify and interpret results, especially in cases of ambiguity or artistic language.
• A time series processing pipeline processes long-form narratives into sentiment time series, enabling temporal analysis of narrative arcs.

Experimental results

Research questions

• RQ1 Can a self-supervised ensemble approach outperform state-of-the-art Transformers on sentiment analysis of long-form literary narratives?
• RQ2 To what extent do SOTA Transformers fail to detect narrative arcs compared to simpler models in open-domain, long-form texts?
• RQ3 How effective is an exhaustive joint optimization over corpus:model combinations in improving sentiment analysis generalization?
• RQ4 Can human-in-the-loop visualizations significantly reduce expert effort in identifying meaningful narrative features and anomalies?
• RQ5 What metrics best capture the stability and performance of sentiment models across diverse narrative corpora?

Key findings

• SOTA Transformers, despite 97.5% accuracy on narrow benchmarks like IMDB, show significantly reduced performance on long-form narratives, with accuracy drops up to 12% compared to training data.
• The SentimentArcs framework achieved superior performance by jointly optimizing over 840 corpus:model combinations, outperforming individual models including RoBERTa and BERT on narrative arcs.
• Human experts were able to identify key narrative features and anomalies 3–5× faster using SentimentArcs visualizations compared to raw model outputs.
• The Model Stability Metric successfully identified high-performing, consistent models across diverse corpora, revealing that model performance is highly sensitive to corpus-specific characteristics.
• DTW hierarchical clustering effectively grouped narrative arcs by sentiment trajectory, enabling the discovery of recurring structural patterns across novels.
• The study found that even high-performing models on short text benchmarks often fail to generalize to long-form narratives due to domain shift and linguistic complexity.