[Paper Review] PolyHope: Two-Level Hope Speech Detection from Tweets

Fazlourrahman Balouchzahi, Grigori Sidorov+1 more|arXiv (Cornell University)|Oct 25, 2022

Speech Recognition and Synthesis4 citations

TL;DR

This paper introduces PolyHope, the first two-level hope speech detection dataset for English tweets, classifying tweets into 'Hope' vs. 'Not Hope' and further into 'Generalized', 'Realistic', and 'Unrealistic Hope' subcategories. Using strict annotation guidelines and a diverse tweet collection from early 2022, the authors benchmark multiple models, finding that transformer-based models like BERT-base-uncased achieve the highest F1 scores of 0.85 (binary) and 0.72 (multiclass), outperforming traditional machine learning and deep learning baselines.

ABSTRACT

Hope is characterized as openness of spirit toward the future, a desire, expectation, and wish for something to happen or to be true that remarkably affects human's state of mind, emotions, behaviors, and decisions. Hope is usually associated with concepts of desired expectations and possibility/probability concerning the future. Despite its importance, hope has rarely been studied as a social media analysis task. This paper presents a hope speech dataset that classifies each tweet first into "Hope" and "Not Hope", then into three fine-grained hope categories: "Generalized Hope", "Realistic Hope", and "Unrealistic Hope" (along with "Not Hope"). English tweets in the first half of 2022 were collected to build this dataset. Furthermore, we describe our annotation process and guidelines in detail and discuss the challenges of classifying hope and the limitations of the existing hope speech detection corpora. In addition, we reported several baselines based on different learning approaches, such as traditional machine learning, deep learning, and transformers, to benchmark our dataset. We evaluated our baselines using weighted-averaged and macro-averaged F1-scores. Observations show that a strict process for annotator selection and detailed annotation guidelines enhanced the dataset's quality. This strict annotation process resulted in promising performance for simple machine learning classifiers with only bi-grams; however, binary and multiclass hope speech detection results reveal that contextual embedding models have higher performance in this dataset.

Motivation & Objective

• To address the lack of multiclass hope speech detection datasets in NLP by creating a fine-grained classification of hope types in social media.
• To improve the quality and reliability of hope speech annotation through a rigorous process involving expert annotators and detailed guidelines.
• To provide a benchmark dataset for researchers to explore hope as a psychological and linguistic phenomenon in online discourse.
• To evaluate the performance of diverse machine learning models, including traditional, deep learning, and transformer-based approaches, on this novel dataset.
• To enable future research in mental health, public sentiment analysis, and human behavior modeling using hope-related language on social media.

Proposed method

• The dataset was constructed from over 100,000 English tweets collected using keywords related to hope, desire, and expectation from the first half of 2022.
• A two-level annotation process was applied: first, classifying tweets as 'Hope' or 'Not Hope'; second, further categorizing 'Hope' tweets into 'Generalized', 'Realistic', or 'Unrealistic Hope'.
• A strict annotator selection process was used, with trained annotators applying detailed, psychology-informed guidelines to ensure consistency and high inter-annotator agreement (IAA) of 0.85 (binary) and 0.82 (multiclass).
• The final dataset contains 8,256 tweets: 4,175 labeled as 'Hope' and 4,081 as 'Not Hope', with hope tweets further split into the three subcategories.
• Multiple baseline models were trained and evaluated, including TF-IDF with n-grams, CNN and BiLSTM with GloVe/FastText embeddings, and transformer models such as BERT-base-uncased.
• Model evaluation used weighted and macro-averaged F1-scores, with additional analysis of misclassified examples using word count, character count, stopword frequency, and NER tags.

Figure 1: Workflow of annotation procedure

Experimental results

Research questions

• RQ1 How does the performance of different machine learning models compare on the novel two-level hope speech detection task?
• RQ2 To what extent does a rigorous annotation process with detailed guidelines improve the quality and reliability of hope speech datasets?
• RQ3 Can traditional machine learning models effectively distinguish between fine-grained types of hope (e.g., realistic vs. unrealistic) in social media text?
• RQ4 Why do certain tweets remain consistently misclassified across models, and what linguistic or structural features do they share?
• RQ5 How does the inclusion of contextual embeddings (e.g., BERT) improve performance on multiclass hope detection compared to non-contextual embeddings or n-gram features?

Key findings

• The PolyHope dataset achieved high inter-annotator agreement, with 0.85 F1-score for binary classification and 0.82 for multiclass classification, indicating strong annotation quality.
• Traditional machine learning models using only bi-grams achieved strong performance on binary hope detection, suggesting that simple features can be effective for coarse classification.
• Neural network models with contextual embeddings (e.g., BiLSTM, CNN) outperformed traditional models in multiclass hope detection, highlighting the importance of semantic context.
• The BERT-base-uncased transformer model achieved the highest performance, with a macro-averaged F1-score of 0.85 for binary classification and 0.72 for multiclass classification.
• Approximately 20% of tweets (±5%) were consistently misclassified across all models, indicating inherent ambiguity or complexity in hope expression that challenges current NLP models.
• Analysis of misclassified tweets revealed no strong correlation with word count, character count, or stopword frequency, but showed notable patterns in named entity recognition (NER) tags, suggesting entity-level features may play a role in misclassification.

Figure 2: Word cloud of hope keywords in the dataset