[Paper Review] BERT Rankers are Brittle: a Study using Adversarial Document Perturbations

Yumeng Wang, Lijun Lyu+1 more|arXiv (Cornell University)|Jun 23, 2022

Sentiment Analysis and Opinion Mining4 citations

TL;DR

This paper proposes white-box adversarial attacks on BERT-based ranking models by perturbing document text to drastically shift document ranks with minimal token changes. Using gradient-based optimization, the authors demonstrate that adding just 5 adversarial tokens—especially at document starts—can cause large rank drops or promotions, revealing model brittleness and topic-specific biases from pretraining and datasets.

ABSTRACT

Contextual ranking models based on BERT are now well established for a wide range of passage and document ranking tasks. However, the robustness of BERT-based ranking models under adversarial inputs is under-explored. In this paper, we argue that BERT-rankers are not immune to adversarial attacks targeting retrieved documents given a query. Firstly, we propose algorithms for adversarial perturbation of both highly relevant and non-relevant documents using gradient-based optimization methods. The aim of our algorithms is to add/replace a small number of tokens to a highly relevant or non-relevant document to cause a large rank demotion or promotion. Our experiments show that a small number of tokens can already result in a large change in the rank of a document. Moreover, we find that BERT-rankers heavily rely on the document start/head for relevance prediction, making the initial part of the document more susceptible to adversarial attacks. More interestingly, we find a small set of recurring adversarial words that when added to documents result in successful rank demotion/promotion of any relevant/non-relevant document respectively. Finally, our adversarial tokens also show particular topic preferences within and across datasets, exposing potential biases from BERT pre-training or downstream datasets.

Motivation & Objective

• To investigate the robustness of BERT-based ranking models under adversarial document perturbations.
• To develop gradient-based attack methods that generate adversarial tokens to demote relevant or promote non-relevant documents.
• To analyze the recurrence and topic preferences of adversarial tokens across queries and datasets.
• To uncover potential biases in BERT ranking models stemming from pretraining or downstream datasets.

Proposed method

• Propose a local ranking attack that uses gradient-based token search to identify adversarial tokens at high-gradient positions in document text.
• Design a global ranking attack that optimizes for rank shifts across an entire query workload, not per-query.
• Adapt gradient-based optimization techniques from NLP adversarial attacks (e.g., Ebrahimi et al., 2017; Wallace et al., 2019a) to the document ranking setting.
• Use BERT-style rankers as the victim model, fine-tuned on ClueWeb09 and TREC-DL datasets, to evaluate attack effectiveness.
• Perform post-hoc analysis on adversarial tokens using PCA visualization and topic frequency analysis to detect recurring themes.
• Evaluate attacks on both rank demotion (highly relevant documents) and rank promotion (lowly ranked documents), focusing on document start regions.

Experimental results

Research questions

• RQ1 Can small, targeted perturbations to document text cause significant rank shifts in BERT-based ranking models?
• RQ2 Are BERT rankers more vulnerable to adversarial attacks at the beginning of documents due to attention mechanisms or attention patterns?
• RQ3 Do adversarial tokens recur across different queries and datasets, and if so, what topics do they represent?
• RQ4 To what extent do the recurring adversarial tokens reflect biases from BERT's pretraining or the downstream ranking datasets?
• RQ5 How do the frequency and topic distribution of adversarial tokens compare between local and global attack strategies?

Key findings

• A local ranking attack using only five adversarial tokens caused an average rank drop of 83 positions across top-ranked documents in ClueWeb09.
• The global attack achieved a similar average rank drop of 79 positions using the same five most frequent adversarial tokens.
• Adversarial tokens frequently recurred across queries and datasets, with strong topic preferences for terms related to nature, religion, ethnicity, and medicine.
• The most frequently used adversarial tokens for rank demotion included 'acceptable', 'competition', and 'rayon', while those for promotion included 'tornadoes', 'hurricanes', and 'earthquakes'.
• BERT rankers were particularly sensitive to perturbations at the start of documents, indicating a strong reliance on initial tokens for relevance prediction.
• The recurring adversarial tokens were not frequent in the corpus and showed low mutual information with relevance labels, suggesting they exploit model biases rather than semantic relevance.