[Paper Review] Self-Supervised Contrastive Learning with Adversarial Perturbations for Robust Pretrained Language Models

Meng Zhao, Yihan Dong+2 more|arXiv (Cornell University)|Jul 15, 2021

Topic Modeling34 references4 citations

TL;DR

This paper proposes a self-supervised contrastive learning method that enhances BERT's robustness against word substitution-based adversarial attacks by generating adversarial perturbations on-the-fly during training, using only unlabeled data. The approach improves robustness across four datasets without requiring labels or pre-defined adversarial examples.

ABSTRACT

This paper improves the robustness of the pretrained language model BERT against word substitution-based adversarial attacks by leveraging self-supervised contrastive learning with adversarial perturbations. One advantage of our method compared to previous works is that it is capable of improving model robustness without using any labels. Additionally, we also create an adversarial attack for word-level adversarial training on BERT. The attack is efficient, allowing adversarial training for BERT on adversarial examples generated on the fly during training. Experimental results on four datasets show that our method improves the robustness of BERT against four different word substitution-based adversarial attacks. Furthermore, to understand why our method can improve the model robustness against adversarial attacks, we study vector representations of clean examples and their corresponding adversarial examples before and after applying our method. As our method improves model robustness with unlabeled raw data, it opens up the possibility of using large text datasets to train robust language models.

Motivation & Objective

• To improve the robustness of pretrained language models like BERT against word substitution-based adversarial attacks.
• To develop a method that enhances robustness without relying on labeled data or precomputed adversarial examples.
• To design an efficient, on-the-fly adversarial attack generation mechanism compatible with BERT training.
• To understand the representation-level changes that lead to improved robustness through analysis of clean and adversarial example embeddings.

Proposed method

• The method employs self-supervised contrastive learning to align representations of clean input sentences with their adversarially perturbed counterparts.
• Adversarial examples are generated dynamically during training using a word substitution attack that replaces words with semantically similar but adversarial tokens.
• The contrastive loss encourages the model to produce similar representations for both clean and adversarial versions of the same input sentence.
• The approach is trained end-to-end on raw, unlabeled text, enabling scalability to large-scale datasets.
• A novel adversarial attack is designed to be efficient and compatible with online training, allowing real-time perturbation generation.
• The method improves robustness by learning invariant representations that are resilient to word-level perturbations.

Experimental results

Research questions

• RQ1 How can self-supervised contrastive learning improve the robustness of BERT against word substitution-based adversarial attacks without using labels?
• RQ2 What is the impact of on-the-fly adversarial example generation on model robustness and training efficiency?
• RQ3 How do the learned representations of clean and adversarial examples differ before and after applying the proposed method?
• RQ4 Can large-scale unlabeled text be effectively leveraged to train robust language models through this approach?

Key findings

• The proposed method improves BERT's robustness against four distinct word substitution-based adversarial attacks across four benchmark datasets.
• The model achieves improved robustness without requiring any labeled data, relying solely on self-supervision and adversarial perturbations.
• Representation analysis shows that the method reduces the distance between clean and adversarial example embeddings, indicating improved robustness at the representation level.
• The method enables efficient adversarial training by generating adversarial examples on-the-fly during the training process.
• The approach demonstrates scalability to large text datasets, opening avenues for training robust models on vast unlabeled corpora.