[Paper Review] InfoBERT: Improving Robustness of Language Models from An Information Theoretic Perspective

Boxin Wang, Shuohang Wang+5 more|arXiv (Cornell University)|Oct 5, 2020

Topic ModelingComputer Science43 references61 citations

TL;DR

InfoBERT introduces two mutual-information-based regularizers—Information Bottleneck and Anchored Feature—to fine-tune pre-trained language models for robustness against adversarial text attacks, achieving state-of-the-art robust accuracy on NLI and QA without sacrificing benign accuracy.

ABSTRACT

Large-scale language models such as BERT have achieved state-of-the-art performance across a wide range of NLP tasks. Recent studies, however, show that such BERT-based models are vulnerable facing the threats of textual adversarial attacks. We aim to address this problem from an information-theoretic perspective, and propose InfoBERT, a novel learning framework for robust fine-tuning of pre-trained language models. InfoBERT contains two mutual-information-based regularizers for model training: (i) an Information Bottleneck regularizer, which suppresses noisy mutual information between the input and the feature representation; and (ii) a Robust Feature regularizer, which increases the mutual information between local robust features and global features. We provide a principled way to theoretically analyze and improve the robustness of representation learning for language models in both standard and adversarial training. Extensive experiments demonstrate that InfoBERT achieves state-of-the-art robust accuracy over several adversarial datasets on Natural Language Inference (NLI) and Question Answering (QA) tasks. Our code is available at https://github.com/AI-secure/InfoBERT.

Motivation & Objective

• Motivate robust fine-tuning of large pre-trained language models against textual adversarial attacks.
• Propose an information-theoretic framework with regularizers to refine local (word-level) and global (sentence-level) representations.
• Theoretically analyze robustness improvements and empirically validate with adversarial datasets across NLP tasks.

Proposed method

• Propose Information Bottleneck (IB) regularizer to suppress noisy mutual information between input and local representations, yielding an approximate minimal sufficient statistic for downstream tasks.
• Introduce a localized IB objective that scales with sequence length to handle high-dimensional token-level features.
• Propose Anchored Feature regularizer to identify and align local stable features with global sentence representations, enhancing robustness.
• Utilize adversarial perturbation (one-step PGD-like) to identify nonrobust and unuseful local features; select local anchored features via Algorithm 1.
• Formulate a final objective combining I(Y;T), a penalty on I(X_i;T_i) for local features, and a MI-based term connecting anchored features to the global representation using InfoNCE as a tractable lower bound.

Experimental results

Research questions

• RQ1 How can information-theoretic regularizers improve robustness of pre-trained language models under textual adversarial attacks?
• RQ2 Do local (word-level) and global (sentence-level) representations benefit from coordinated regularization to resist adversarial manipulation?
• RQ3 Can adversarial training combined with IB-based regularization further enhance robustness while maintaining benign accuracy?

Key findings

• InfoBERT yields substantial improvements in robust accuracy on adversarial NLI and QA benchmarks compared to strong baselines.
• The Information Bottleneck regularizer helps prune noisy information and supports robustness under both standard and adversarial training.
• The Local Anchored Feature regularizer selects robust local features and aligns them with global representations, contributing significantly to robustness.
• Combining adversarial training with InfoBERT leads to better robustness than either approach alone, consistent with theoretical robustness bounds.
• InfoBERT achieves state-of-the-art robust accuracy on multiple adversarial datasets for NLI and QA while preserving benign accuracy and reducing training time relative to some baselines.