[Paper Review] ESimCSE: Enhanced Sample Building Method for Contrastive Learning of Unsupervised Sentence Embedding

Xing Wu, Chaochen Gao+4 more|arXiv (Cornell University)|Sep 9, 2021

Topic Modeling22 references69 citations

TL;DR

ESimCSE improves unsupervised sentence embeddings by (1) using a safer positive-pair length-extension via word/sub-word repetition and (2) increasing negative pairs with momentum contrast, yielding stronger STS performance than SimCSE on benchmarks.

ABSTRACT

Contrastive learning has been attracting much attention for learning unsupervised sentence embeddings. The current state-of-the-art unsupervised method is the unsupervised SimCSE (unsup-SimCSE). Unsup-SimCSE takes dropout as a minimal data augmentation method, and passes the same input sentence to a pre-trained Transformer encoder (with dropout turned on) twice to obtain the two corresponding embeddings to build a positive pair. As the length information of a sentence will generally be encoded into the sentence embeddings due to the usage of position embedding in Transformer, each positive pair in unsup-SimCSE actually contains the same length information. And thus unsup-SimCSE trained with these positive pairs is probably biased, which would tend to consider that sentences of the same or similar length are more similar in semantics. Through statistical observations, we find that unsup-SimCSE does have such a problem. To alleviate it, we apply a simple repetition operation to modify the input sentence, and then pass the input sentence and its modified counterpart to the pre-trained Transformer encoder, respectively, to get the positive pair. Additionally, we draw inspiration from the community of computer vision and introduce a momentum contrast, enlarging the number of negative pairs without additional calculations. The proposed two modifications are applied on positive and negative pairs separately, and build a new sentence embedding method, termed Enhanced Unsup-SimCSE (ESimCSE). We evaluate the proposed ESimCSE on several benchmark datasets w.r.t the semantic text similarity (STS) task. Experimental results show that ESimCSE outperforms the state-of-the-art unsup-SimCSE by an average Spearman correlation of 2.02% on BERT-base.

Motivation & Objective

• Identify biases in SimCSE arising from same-length positive pairs in Transformer encoders.
• Develop safe sentence-length extension methods that preserve meaning for positive pairs.
• Increase the number of informative negative pairs without prohibitive computation using momentum contrast.
• Evaluate the proposed enhancements on standard semantic textual similarity benchmarks.
• Demonstrate that the combined approach (ESimCSE) improves unsupervised sentence embeddings across models.

Proposed method

• Introduce word/sub-word repetition to modify one member of a positive pair while preserving meaning.
• Apply a momentum-contrast queue to enlarge the set of negative pairs with a momentum-updated encoder.
• Combine both enhancements with the standard SimCSE objective to form ESimCSE.
• Train on English Wikipedia (1M sentences) with a BERT/RoBERTa backbone and an MLP on CLS for sentence embeddings.
• Use a dropout-based positive pair generation as in SimCSE, but with independently constructed pairs for the positives.
• Evaluate using STS benchmarks and Spearman correlation, reporting average gains over SimCSE.

Experimental results

Research questions

• RQ1 Does the same-length bias in SimCSE negatively impact semantic similarity learning, and can it be mitigated with a safe length-extension method?
• RQ2 Can word/sub-word repetition provide a safer and effective positive-pair augmentation without distorting semantics?
• RQ3 Does incorporating momentum contrast to create more negative pairs improve unsupervised sentence embeddings without prohibitive computation?
• RQ4 What is the overall performance gain of combining these enhancements over the baseline SimCSE across standard STS datasets?

Key findings

• ESimCSE achieves an average Spearman correlation improvement of 2.02% over SimCSE on BERT-base.
• Word/sub-word repetition for positive pairs yields measurable STS gains (e.g., STS-B development improved by up to 1.64 points with word repetition, 1.53 points with momentum, and 2.40 points with the full ESimCSE configuration).
• Momentum contrast expands negative pairs via a queue and a momentum-updated encoder, improving learning without excessive memory cost.
• Across model variants (BERT base/large, RoBERTa base/large), ESimCSE consistently outperforms SimCSE on STS benchmarks.
• In transfer tasks, ESimCSE slightly increases average performance over SimCSE (e.g., 86.06 vs 85.81 on reported transfer tasks).
• Ablation studies show both positive-pair enhancement and negative-pair expansion contribute meaningfully, while sentence-length bucketing shows limited or negative impact.