[Paper Review] FairFil: Contrastive Neural Debiasing Method for Pretrained Text Encoders

Pengyu Cheng, Weituo Hao+3 more|arXiv (Cornell University)|Mar 11, 2021

Adversarial Robustness in Machine Learning49 references50 citations

TL;DR

FairFil is a post hoc neural debiasing method for pretrained sentence encoders that uses a fair filter trained with contrastive learning to reduce bias while preserving semantic content, without retraining the encoder.

ABSTRACT

Pretrained text encoders, such as BERT, have been applied increasingly in various natural language processing (NLP) tasks, and have recently demonstrated significant performance gains. However, recent studies have demonstrated the existence of social bias in these pretrained NLP models. Although prior works have made progress on word-level debiasing, improved sentence-level fairness of pretrained encoders still lacks exploration. In this paper, we proposed the first neural debiasing method for a pretrained sentence encoder, which transforms the pretrained encoder outputs into debiased representations via a fair filter (FairFil) network. To learn the FairFil, we introduce a contrastive learning framework that not only minimizes the correlation between filtered embeddings and bias words but also preserves rich semantic information of the original sentences. On real-world datasets, our FairFil effectively reduces the bias degree of pretrained text encoders, while continuously showing desirable performance on downstream tasks. Moreover, our post-hoc method does not require any retraining of the text encoders, further enlarging FairFil's application space.

Motivation & Objective

• Motivate the need to reduce sentence-level social bias in pretrained text encoders beyond word-level debiasing.
• Propose a neural debiasing framework (FairFil) that outputs debiased embeddings from a fixed encoder.
• Develop a contrastive learning objective to preserve semantic information while removing bias.
• Incorporate a debiasing regularizer to further minimize information about sensitive words in debiased embeddings.

Proposed method

• Learn a fair filter network f on top of a pretrained encoder E to map z = E(x) to debiased d = f(z).
• Create augmented sentences x' by replacing sensitive words with semantically replaceable words in a different bias direction.
• Train with InfoNCE contrastive loss to maximize mutual information between d and d' for (x, x').
• Add a debiasing regularizer that minimizes mutual information between d and embeddings of sensitive words w^p via CLUB bounds, encouraging removal of bias information from d.
• Parameterize f as a one-layer neural net; g for InfoNCE scoring is a two-layer net; q_theta(w|d) as a Gaussian variational model for CLUB estimation.
• Use a batch-based training setup with 128 samples and a learning rate of 1e-5 for up to 10 epochs on a fixed training corpus.

Experimental results

Research questions

• RQ1 Can a neural post hoc fair filter reduce sentence-level bias in pretrained text encoders without retraining the encoder?
• RQ2 Does contrastive learning with semantically augmented sentences preserve semantic content while removing bias?
• RQ3 Does a debiasing regularizer targeting sensitive word information improve debiasing without harming downstream performance?

Key findings

• FairFil reduces SEAT bias measures on pretrained BERT and improves or preserves downstream task accuracy compared with Sent-Debias.
• FairFil achieves lower average SEAT effect sizes than Sent-Debias across multiple bias templates and topics.
• The debiasing regularizer further reduces bias but can trade off slightly with downstream performance, indicating a fairness-representativeness trade-off.
• FairFil demonstrates data efficiency, showing effective bias reduction with smaller training data partitions (as low as 20%).
• Compared to word-level debiasing baselines, FairFil achieves substantially lower bias, indicating sentence-level debiasing advantages.