[Paper Review] SoftMCL: Soft Momentum Contrastive Learning for Fine-grained Sentiment-aware Pre-training

Jin Wang, Liang-Chih Yu+1 more|arXiv (Cornell University)|May 3, 2024

Emotion and Mood Recognition4 citations

TL;DR

This paper proposes SoftMCL, a soft momentum contrastive learning framework for fine-grained sentiment-aware pre-training that uses continuous valence ratings as soft labels instead of hard sentiment classes. By integrating word- and sentence-level contrastive learning with a momentum queue to expand negative sample diversity, SoftMCL improves affective representation learning and achieves state-of-the-art performance on multiple sentiment analysis benchmarks, outperforming existing methods in handling nuanced sentiment distinctions.

ABSTRACT

The pre-training for language models captures general language understanding but fails to distinguish the affective impact of a particular context to a specific word. Recent works have sought to introduce contrastive learning (CL) for sentiment-aware pre-training in acquiring affective information. Nevertheless, these methods present two significant limitations. First, the compatibility of the GPU memory often limits the number of negative samples, hindering the opportunities to learn good representations. In addition, using only a few sentiment polarities as hard labels, e.g., positive, neutral, and negative, to supervise CL will force all representations to converge to a few points, leading to the issue of latent space collapse. This study proposes a soft momentum contrastive learning (SoftMCL) for fine-grained sentiment-aware pre-training. Instead of hard labels, we introduce valence ratings as soft-label supervision for CL to fine-grained measure the sentiment similarities between samples. The proposed SoftMCL is conducted on both the word- and sentence-level to enhance the model's ability to learn affective information. A momentum queue was introduced to expand the contrastive samples, allowing storing and involving more negatives to overcome the limitations of hardware platforms. Extensive experiments were conducted on four different sentiment-related tasks, which demonstrates the effectiveness of the proposed SoftMCL method. The code and data of the proposed SoftMCL is available at: https://www.github.com/wangjin0818/SoftMCL/.

Motivation & Objective

• To address the limitation of pre-trained language models in capturing fine-grained sentiment nuances due to reliance on hard sentiment labels.
• To overcome GPU memory constraints in contrastive learning by introducing a momentum queue for storing and reusing negative samples.
• To improve affective representation learning by replacing discrete sentiment labels with continuous valence ratings as supervision.
• To enhance model performance on sentiment-related tasks through joint word- and sentence-level pre-training with soft contrastive learning.
• To demonstrate the effectiveness of soft-label contrastive learning in mitigating latent space collapse and improving sentiment generalization.

Proposed method

• SoftMCL uses continuous valence ratings as soft supervision for contrastive learning, replacing hard sentiment labels to enable fine-grained similarity measurement between samples.
• A momentum encoder and momentum queue are introduced to store and retrieve historical negative samples, increasing the effective negative sample pool beyond batch size limits.
• The framework conducts contrastive learning simultaneously at word- and sentence-level, allowing multi-granularity affective representation learning.
• The training objective combines masked language modeling (MLM) with dual-branch contrastive loss, where the contrastive loss is computed using temperature-scaled cosine similarity between anchor and positive/negative pairs.
• The momentum encoder is updated with a momentum coefficient μ, allowing gradual evolution of the momentum network to stabilize the contrastive learning process.
• Hyperparameters such as temperature τ, balance coefficients λ₁ and λ₂, and queue size are tuned to optimize performance, with τ=0.1 and queue size=1024 yielding best results.

(b) Supervised contrastive learning with hard labels

Experimental results

Research questions

• RQ1 Can continuous valence ratings as soft labels improve the quality of affective representation learning compared to discrete hard labels?
• RQ2 Does the use of a momentum queue significantly enhance contrastive learning performance under GPU memory constraints?
• RQ3 How does joint word- and sentence-level pre-training affect sentiment classification performance compared to single-level pre-training?
• RQ4 What is the optimal configuration of hyperparameters such as temperature, momentum coefficient, and queue size for the SoftMCL framework?
• RQ5 To what extent does SoftMCL mitigate latent space collapse in contrastive learning for sentiment-aware pre-training?

Key findings

• SoftMCL achieves state-of-the-art performance on four sentiment-related benchmarks, including IMDB and MR, with the best results obtained at λ₁=0.25 and λ₂=0.25.
• The optimal temperature hyperparameter τ=0.1 consistently yields the best performance across IMDB and MR datasets, indicating improved contrastive learning stability and representation quality.
• A momentum coefficient μ=0.9 leads to superior performance, demonstrating that a slowly evolving momentum encoder enhances representation learning.
• A queue size of 1024 achieves the best performance, showing that larger queues improve negative sample diversity without introducing excessive noise.
• Ablation studies confirm that removing word-level or sentence-level pre-training leads to consistent performance degradation, proving the necessity of multi-granularity learning.
• The training loss converges within 40,000 steps for word-level and 20,000 steps for sentence-level pre-training, indicating efficient learning enabled by the momentum queue's persistent negative samples.

(c) Supervised contrastive learning with soft labels