[Paper Review] Virtual Augmentation Supported Contrastive Learning of Sentence Representations

Dejiao Zhang, Wei Xiao+3 more|arXiv (Cornell University)|Oct 16, 2021

Topic Modeling6 citations

TL;DR

This paper proposes Virtual Augmentation Supported Contrastive Learning (VaSCL), a novel method for unsupervised sentence representation learning that generates virtual data augmentations by leveraging the K-nearest in-batch neighbors of each instance in the representation space. By formulating an adversarial instance discrimination task on this neighborhood, VaSCL achieves state-of-the-art performance across multiple downstream NLP tasks, outperforming prior methods including SimCSE, especially when explicit text augmentations are used.

ABSTRACT

Despite profound successes, contrastive representation learning relies on carefully designed data augmentations using domain specific knowledge. This challenge is magnified in natural language processing where no general rules exist for data augmentation due to the discrete nature of natural language. We tackle this challenge by presenting a Virtual augmentation Supported Contrastive Learning of sentence representations (VaSCL). Originating from the interpretation that data augmentation essentially constructs the neighborhoods of each training instance, we in turn utilize the neighborhood to generate effective data augmentations. Leveraging the large training batch size of contrastive learning, we approximate the neighborhood of an instance via its K-nearest in-batch neighbors in the representation space. We then define an instance discrimination task regarding this neighborhood and generate the virtual augmentation in an adversarial training manner. We access the performance of VaSCL on a wide range of downstream tasks, and set a new state-of-the-art for unsupervised sentence representation learning.

Motivation & Objective

• To address the challenge of designing effective, general-purpose data augmentations in NLP due to the discrete nature of text.
• To overcome the limitations of existing explicit text augmentation strategies, which often rely on heuristic rules and yield suboptimal performance.
• To improve unsupervised sentence representation learning by leveraging neighborhood structure in the representation space to generate more effective virtual augmentations.
• To achieve state-of-the-art performance in zero-shot transfer learning on diverse NLP benchmarks without relying on task-specific annotations.

Proposed method

• VaSCL interprets data augmentation as constructing neighborhoods around training instances and inverts this process by using the neighborhood to generate augmentations.
• It approximates the neighborhood of a given instance using its K-nearest neighbors within the same training batch in the representation space.
• An adversarial training procedure is employed to generate virtual augmentations that are semantically close to the original instance but distinct enough to serve as positive pairs in contrastive learning.
• The method combines two contrastive objectives: one using standard dropout-based augmentation (as in SimCSE), and another using the generated virtual augmentations to enforce instance discrimination.
• The virtual augmentation is optimized end-to-end via a contrastive loss that encourages the model to distinguish between the original and virtual augmented representations.
• The framework is trained with a large batch size to ensure sufficient in-batch neighbor diversity, enabling robust neighborhood approximation.

Experimental results

Research questions

• RQ1 Can we improve unsupervised sentence representation learning by generating virtual data augmentations based on the neighborhood structure of instances in the representation space?
• RQ2 Why do explicit text augmentation strategies often underperform compared to simple dropout-based augmentation in contrastive learning?
• RQ3 Does leveraging in-batch neighbors to generate virtual augmentations lead to better semantic clustering and representation dispersion?
• RQ4 Can virtual augmentation mitigate the negative impact of noisy or weak explicit text augmentations on downstream transfer performance?
• RQ5 What role do linguistic diversity and semantic coherence play in the effectiveness of virtual versus explicit data augmentations?

Key findings

• VaSCL achieves state-of-the-art performance on a wide range of downstream NLP tasks, consistently outperforming previous unsupervised sentence representation models.
• Virtual augmentation via neighborhood-guided generation significantly outperforms all explicit text augmentation strategies (e.g., word deletion, synonym replacement, contextual substitution) across all evaluated tasks.
• The performance of both SimCSE and VaSCL degrades when combined with explicit text augmentations, suggesting that such augmentations introduce noise and limit representational diversity.
• The representation space learned by VaSCL is more dispersed, with better encoding of semantic semantics at multiple granularities, as evidenced by cosine similarity analysis.
• The model trained with virtual augmentations exhibits lower cosine similarity between original and augmented representations than models using explicit augmentations, indicating more effective and diverse positive pairs.
• The use of dropout as a baseline augmentation strategy outperforms explicit methods, highlighting the difficulty of designing effective text-level transformations in NLP.