[Paper Review] Data Augmentation for Improving Emotion Recognition in Software Engineering Communication

Mia Mohammad Imran, Yashasvi Jain+2 more|arXiv (Cornell University)|Aug 10, 2022

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper proposes polarity-based data augmentation to improve emotion recognition in software engineering communication, using BART-based text generation to create synthetic training data that preserves emotional polarity. The approach increases micro F1-score by 9.3% on average across three existing emotion classifiers (ESEM-E, EMTk, SEntiMoji), significantly improving performance on GitHub issue and pull request comments.

ABSTRACT

Emotions (e.g., Joy, Anger) are prevalent in daily software engineering (SE) activities, and are known to be significant indicators of work productivity (e.g., bug fixing efficiency). Recent studies have shown that directly applying general purpose emotion classification tools to SE corpora is not effective. Even within the SE domain, tool performance degrades significantly when trained on one communication channel and evaluated on another (e.g, StackOverflow vs. GitHub comments). Retraining a tool with channel-specific data takes significant effort since manually annotating large datasets of ground truth data is expensive. In this paper, we address this data scarcity problem by automatically creating new training data using a data augmentation technique. Based on an analysis of the types of errors made by popular SE-specific emotion recognition tools, we specifically target our data augmentation strategy in order to improve the performance of emotion recognition. Our results show an average improvement of 9.3% in micro F1-Score for three existing emotion classification tools (ESEM-E, EMTk, SEntiMoji) when trained with our best augmentation strategy.

Motivation & Objective

• Address the poor performance of general-purpose emotion classifiers on software engineering (SE) communication due to domain-specific language and limited annotated data.
• Investigate the root causes of misclassification in existing SE emotion recognition tools through error analysis on a newly curated GitHub dataset.
• Develop and evaluate data augmentation strategies tailored to SE text to improve model generalization and performance without requiring extensive manual annotation.
• Demonstrate that augmentation focused on emotional polarity—rather than broad lexical substitution—yields the most effective improvements in emotion classification.
• Provide a scalable, automated solution to data scarcity in SE emotion recognition, enabling better tooling for developer well-being and team productivity analytics.

Proposed method

• Curated a new dataset of 2,000 GitHub issue and pull request comments, annotated with six primary emotions (Shaver et al., 1987) and secondary/tertiary categories from GoEmotions.
• Conducted error analysis on predictions from three existing tools (ESEM-E, EMTk, SEntiMoji) to identify misclassification patterns, revealing that lexical cue recognition failures were the primary cause of errors.
• Designed three data augmentation strategies: unconstrained random word replacement, constrained replacement using sentiment polarity, and emotion-specific word substitution using lexicons.
• Leveraged the BART language model for context-aware text generation, ensuring semantic coherence and emotional consistency in augmented samples.
• Applied augmentation to increase training data size by 10x, then fine-tuned the three emotion classifiers on the augmented data to evaluate performance improvements.
• Used stratified 80%-20% train-test splits and micro F1-score as the primary evaluation metric, with inter-annotator agreement >0.8 to ensure annotation reliability.

Experimental results

Research questions

• RQ1 How effective are existing emotion classifiers (ESEM-E, EMTk, SEntiMoji) in detecting emotions in GitHub comments, and what types of errors are most common?
• RQ2 What are the primary causes of misclassification in current SE emotion recognition tools, particularly in relation to lexical cues and emotional polarity?
• RQ3 Can data augmentation techniques improve the performance of existing emotion classifiers in SE communication, and which augmentation strategy yields the best results?
• RQ4 Does polarity-based augmentation—targeting positive/negative sentiment words—outperform generic or emotion-specific augmentation strategies in improving F1-score?

Key findings

• The three existing emotion classifiers (ESEM-E, EMTk, SEntiMoji) showed significantly reduced performance on GitHub comments compared to their original evaluation datasets, indicating domain shift issues.
• Error analysis revealed that the majority of misclassifications stemmed from the tools' inability to recognize clear lexical cues for emotions, not from implicit or complex emotional expressions.
• Polarity-based data augmentation—focusing on replacing words with synonyms of the same emotional polarity (e.g., positive words for Joy, negative for Anger)—yielded the highest performance gains.
• This approach improved the micro F1-score by an average of 9.3% across all three tools, demonstrating that targeted augmentation is more effective than generic or emotion-specific methods.
• The best-performing augmentation strategy did not rely on large SE-specific emotion lexicons, suggesting that polarity-aware generation via pre-trained models like BART is sufficient and scalable.
• The results indicate that data augmentation can effectively mitigate data scarcity and domain shift in SE emotion recognition, offering a practical path to improving existing tools without retraining from scratch.