[Paper Review] ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer

Yuanmeng Yan, Rumei Li+4 more|arXiv (Cornell University)|May 25, 2021

Topic Modeling36 references47 citations

TL;DR

ConSERT fine-tunes BERT with a contrastive learning objective and data augmentation to fix collapsed sentence representations, achieving state-of-the-art STS performance in unsupervised and supervised settings and robustness in data-scarce scenarios.

ABSTRACT

Learning high-quality sentence representations benefits a wide range of natural language processing tasks. Though BERT-based pre-trained language models achieve high performance on many downstream tasks, the native derived sentence representations are proved to be collapsed and thus produce a poor performance on the semantic textual similarity (STS) tasks. In this paper, we present ConSERT, a Contrastive Framework for Self-Supervised Sentence Representation Transfer, that adopts contrastive learning to fine-tune BERT in an unsupervised and effective way. By making use of unlabeled texts, ConSERT solves the collapse issue of BERT-derived sentence representations and make them more applicable for downstream tasks. Experiments on STS datasets demonstrate that ConSERT achieves an 8\% relative improvement over the previous state-of-the-art, even comparable to the supervised SBERT-NLI. And when further incorporating NLI supervision, we achieve new state-of-the-art performance on STS tasks. Moreover, ConSERT obtains comparable results with only 1000 samples available, showing its robustness in data scarcity scenarios.

Motivation & Objective

• Motivate the need for high-quality sentence representations beyond native BERT embeddings due to collapse in semantic space.
• Propose a contrastive, self-supervised fine-tuning framework that reshapes sentence representations without extra inference cost.
• Explore data augmentation strategies to generate view pairs for contrastive learning.
• Demonstrate effectiveness with unsupervised fine-tuning and augmented supervision signals (e.g., NLI).
• Show robustness in few-shot data scenarios and analyze embedding space changes.

Proposed method

• Adopt a SimCLR-like contrastive objective (NT-Xent) on top of a shared BERT encoder.
• Generate two views per sentence via data augmentation applied at the token embedding layer.
• Obtain sentence representations by averaging token embeddings from the last layer, followed by contrastive loss to pull views of the same sentence together and push apart others in-batch.
• Explore four augmentation strategies: adversarial attack (supervised only), token shuffling, token/feature cutoff, and dropout.
• Optionally incorporate supervision signals through NLI by joint, sup-unsup, or joint-unsup training schemes.
• Report performance on STS datasets using Spearman correlation; provide ablations on augmentation and hyperparameters.

Experimental results

Research questions

• RQ1 Can contrastive self-supervised learning overcome BERT's sentence representation collapse for STS tasks?
• RQ2 How do different data augmentation strategies affect unsupervised sentence transfer performance?
• RQ3 Does incorporating supervision (e.g., NLI) plus unsupervised transfer yield state-of-the-art results on STS?
• RQ4 How robust is ConSERT in data-scarce scenarios (few unlabeled examples) for downstream STS tasks?

Key findings

• ConSERT large achieves the best unsupervised STS performance, with an 8% relative gain over BERT-flow on average.
• ConSERT large outperforms several supervised baselines and is competitive with SBERT-NLI in unsupervised settings.
• Joint and unsupervised transfers with NLI supervision yield new state-of-the-art STS performance in supervised settings.
• With only 1000 unlabeled samples, ConSERT remains robust and approaches full-data performance; few-shot results show large gains over baselines.
• Data augmentation strategies Shuffle and Token Cutoff are particularly effective; combination with Feature Cutoff enhances performance.
• Embedding space analysis shows ConSERT reduces dominance of frequent tokens, mitigating collapse.