[Paper Review] CLEAR: Contrastive Learning for Sentence Representation

Zhuofeng Wu, Sinong Wang+4 more|arXiv (Cornell University)|Dec 31, 2020

Topic Modeling28 references230 citations

TL;DR

CLEAR pre-trains a Transformer with a sentence-level contrastive loss using four augmentations (word/span deletion, reordering, synonym substitution) in addition to MLM, improving GLUE and SentEval performance. Different augmentations yield task-specific gains, and both larger batch size and contrastive loss contribute to gains.

ABSTRACT

Pre-trained language models have proven their unique powers in capturing implicit language features. However, most pre-training approaches focus on the word-level training objective, while sentence-level objectives are rarely studied. In this paper, we propose Contrastive LEArning for sentence Representation (CLEAR), which employs multiple sentence-level augmentation strategies in order to learn a noise-invariant sentence representation. These augmentations include word and span deletion, reordering, and substitution. Furthermore, we investigate the key reasons that make contrastive learning effective through numerous experiments. We observe that different sentence augmentations during pre-training lead to different performance improvements on various downstream tasks. Our approach is shown to outperform multiple existing methods on both SentEval and GLUE benchmarks.

Motivation & Objective

• Motivate learning better sentence representations beyond word-level objectives.
• Investigate sentence-level data augmentations suitable for contrastive learning in NLP.
• Pre-train a Transformer with a combined MLM and contrastive loss to learn noise-invariant sentence representations.
• Show how different augmentations affect downstream task performance and robustness.

Proposed method

• Propose CLEAR: an encoder f(·) with a projection head g(·) and a contrastive loss L_CL applied to augmented sentence pairs.
• Generate two augmented views of each sentence using augmentations in set A: word deletion, span deletion, synonym substitution, and reordering.
• Compute L_CL by comparing positive pairs within a minibatch using cosine similarity with temperature τ (SimCLR-style loss).
• Combine L_CL with MLM loss: L_total = L_MLM + L_CL for pre-training from scratch.
• Evaluate using GLUE and SentEval benchmarks to assess sentence representations.
• Perform ablations to separate effects of CL loss versus batch size and analyze which augmentations benefit which tasks.

Experimental results

Research questions

• RQ1 Does sentence-level contrastive learning with carefully designed augmentations improve sentence representations over word-level objectives alone?
• RQ2 Which augmentation strategies (deletion, span deletion, substitution, reordering) contribute to gains on different downstream tasks?
• RQ3 How does combining MLM with CL compare to MLM alone or CL alone in pre-training from scratch?
• RQ4 To what extent do batch size and the contrastive objective each contribute to observed improvements?
• RQ5 Do CLS-token representations or mean-pooling interact with contrastive pre-training in affecting downstream performance?

Key findings

• CLEAR-pretrained models outperform RoBERTa and BERT baselines on GLUE and SentEval benchmarks.
• Certain augmentations (e.g., MLM+del-word, MLM+del-span, MLM+del-span+reorder) yield +2.2% average on 8 GLUE tasks and +5.7% on 7 SentEval STS tasks over RoBERTa. (from abstract and results)
• Different augmentations benefit different tasks; e.g., MLM+subs+del-span excels on QQP and STS, while MLM+del-span improves inference tasks like MNLI, QNLI, RTE.
• Performance gains arise from both larger batch sizes and the contrastive loss, not solely batch size.
• CLS-token embeddings typically benefit more from CL training than mean-pooling in SentEval evaluations.
• Adding CL improves STS tasks significantly, suggesting better alignment of similar sentence pairs.