[Paper Review] SNCSE: Contrastive Learning for Unsupervised Sentence Embedding with Soft Negative Samples

Hao Wang, Yangguang Li+4 more|arXiv (Cornell University)|Jan 16, 2022

Topic Modeling4 citations

TL;DR

SNCSE proposes a contrastive learning framework for unsupervised sentence embedding that introduces sentence negations as soft negative samples to alleviate feature suppression—where models conflate textual and semantic similarity. By using a bidirectional margin loss (BML), SNCSE improves semantic discrimination, achieving state-of-the-art performance with 79.23% average Spearman’s correlation on RoBERTa base and 81.77% on RoBERTa large across STS benchmarks.

ABSTRACT

Unsupervised sentence embedding aims to obtain the most appropriate embedding for a sentence to reflect its semantic. Contrastive learning has been attracting developing attention. For a sentence, current models utilize diverse data augmentation methods to generate positive samples, while consider other independent sentences as negative samples. Then they adopt InfoNCE loss to pull the embeddings of positive pairs gathered, and push those of negative pairs scattered. Although these models have made great progress on sentence embedding, we argue that they may suffer from feature suppression. The models fail to distinguish and decouple textual similarity and semantic similarity. And they may overestimate the semantic similarity of any pairs with similar textual regardless of the actual semantic difference between them. This is because positive pairs in unsupervised contrastive learning come with similar and even the same textual through data augmentation. To alleviate feature suppression, we propose contrastive learning for unsupervised sentence embedding with soft negative samples (SNCSE). Soft negative samples share highly similar textual but have surely and apparently different semantic with the original samples. Specifically, we take the negation of original sentences as soft negative samples, and propose Bidirectional Margin Loss (BML) to introduce them into traditional contrastive learning framework, which merely involves positive and negative samples. Our experimental results show that SNCSE can obtain state-of-the-art performance on semantic textual similarity (STS) task with average Spearman's correlation coefficient of 78.97% on BERTbase and 79.23% on RoBERTabase. Besides, we adopt rank-based error analysis method to detect the weakness of SNCSE for future study.

Motivation & Objective

• To address feature suppression in unsupervised sentence embedding, where models overestimate semantic similarity due to high textual similarity in positive pairs.
• To introduce soft negative samples—sentences with high textual similarity but distinct semantics—as a means to decouple textual and semantic similarity.
• To improve the ability of contrastive learning to distinguish semantic differences despite textual similarity.
• To develop a training framework that enhances semantic discrimination without requiring labeled data.

Proposed method

• SNCSE uses sentence negation as soft negative samples, generating pairs with high textual similarity but clearly different semantics.
• It introduces a bidirectional margin loss (BML) to explicitly model the semantic gap between positive samples and soft negative samples.
• The framework integrates BML into a standard contrastive learning setup that also uses InfoNCE loss for positive and standard negative pairs.
• BML ensures that the embedding of a sentence is pulled away from its negation, while still being pulled toward its augmented positive versions.
• The method is compatible with any pretrained encoder, such as BERT and RoBERTa, and operates in an unsupervised manner.
• The model is trained end-to-end using a combination of InfoNCE and BML losses to optimize semantic representation.

Experimental results

Research questions

• RQ1 Can introducing sentence negations as soft negative samples improve the disentanglement of textual and semantic similarity in unsupervised sentence embedding?
• RQ2 How effective is bidirectional margin loss (BML) in enhancing the model's ability to distinguish between semantically different but textually similar sentences?
• RQ3 Does the use of soft negative samples lead to measurable improvements in semantic textual similarity (STS) benchmark performance?
• RQ4 To what extent does SNCSE mitigate feature suppression compared to existing contrastive learning methods?
• RQ5 What are the primary failure modes of SNCSE, and how do they relate to linguistic phenomena like word order, typos, and negation logic?

Key findings

• SNCSE achieves state-of-the-art performance on the STS benchmark, with an average Spearman’s correlation of 79.23% on RoBERTa base and 81.77% on RoBERTa large.
• The ablation study confirms that BML loss significantly improves performance and contributes to alleviating feature suppression.
• SNCSE demonstrates improved ability to distinguish between a sentence and its negation, as evidenced by the model’s ability to assign lower similarity scores to such pairs.
• Rank-based error analysis reveals that the main failure modes include typos, implicit negation, textual independence, and word order changes—all of which stem from persistent feature suppression.
• The model underestimates similarity in pairs with low textual overlap but high semantic similarity, indicating ongoing challenges in capturing semantic meaning beyond surface-level text.
• Despite progress, SNCSE still struggles with modeling semantic differences in word order and implicit negation, highlighting the limitations of self-attention mechanisms in disentangling syntactic and semantic factors.