[Paper Review] Adversarial Removal of Demographic Attributes from Text Data

Yanai Elazar, Yoav Goldberg|arXiv (Cornell University)|Aug 20, 2018

Topic Modeling32 references10 citations

TL;DR

This paper investigates adversarial training to remove demographic attributes (e.g., gender, race, age) from text representations in neural networks. Despite adversarial training reducing attribute predictability, a post-hoc classifier can still accurately recover demographic information from the encoded representations, revealing that adversarial performance alone is an unreliable proxy for true demographic invariance in text data.

ABSTRACT

Recent advances in Representation Learning and Adversarial Training seem to succeed in removing unwanted features from the learned representation. We show that demographic information of authors is encoded in -- and can be recovered from -- the intermediate representations learned by text-based neural classifiers. The implication is that decisions of classifiers trained on textual data are not agnostic to -- and likely condition on -- demographic attributes. When attempting to remove such demographic information using adversarial training, we find that while the adversarial component achieves chance-level development-set accuracy during training, a post-hoc classifier, trained on the encoded sentences from the first part, still manages to reach substantially higher classification accuracies on the same data. This behavior is consistent across several tasks, demographic properties and datasets. We explore several techniques to improve the effectiveness of the adversarial component. Our main conclusion is a cautionary one: do not rely on the adversarial training to achieve invariant representation to sensitive features.

Motivation & Objective

• To investigate whether demographic attributes (gender, race, age) are encoded in intermediate representations of text-based neural classifiers.
• To evaluate the effectiveness of adversarial training in removing such demographic information from learned representations.
• To assess whether the performance of the adversarial component reliably indicates successful removal of protected attributes.
• To explore methods for improving adversarial training's effectiveness in achieving demographic invariance in text representations.
• To caution against blind trust in adversarial training for fairness, advocating for external verification of representation invariance.

Proposed method

• The model uses a two-stream architecture: an encoder h(x) mapping input text x to a representation vector h_x, followed by a main classifier c(h(x)) for the primary task and an adversary adv(h(x)) for predicting protected attributes z.
• Adversarial training jointly optimizes for main task accuracy and adversarial failure using gradient reversal: the encoder is trained to minimize the main task loss while maximizing the adversary's loss.
• The gradient reversal layer (GRL) scales gradients from the adversary by −λ during backpropagation, enabling end-to-end training with opposing objectives.
• Post-hoc classifiers are trained on the encoded representations to evaluate whether demographic information remains after adversarial training.
• Experiments are conducted across multiple datasets and demographic attributes (gender, race, age), with adversarial performance and post-hoc accuracy compared.
• Model capacity, adversary weight, and ensemble methods are explored to improve adversarial robustness and representation invariance.

Experimental results

Research questions

• RQ1 To what extent are demographic attributes (gender, race, age) encoded in intermediate representations of text classifiers trained on seemingly unrelated tasks?
• RQ2 Does adversarial training effectively remove demographic information from text representations, as measured by the adversary's performance?
• RQ3 Can a post-hoc classifier trained on the encoded representations still accurately predict demographic attributes, even when the adversarial component performs well?
• RQ4 What modifications to the adversarial training setup (e.g., capacity, weight, ensembling) improve the removal of demographic signals?
• RQ5 Is the performance of the adversarial component a reliable indicator of true demographic invariance in the representation?

Key findings

• Demographic attributes such as gender, race, and age are significantly encoded in intermediate representations of text classifiers, even when training data is balanced and the main task is unrelated.
• Despite achieving chance-level accuracy on the development set, the adversarial component fails to fully eliminate demographic information, as post-hoc classifiers achieve substantially higher accuracy (e.g., 75.3% for gender, 65.4% for race, 58.1% for age).
• The performance of the adversarial classifier during training is not a reliable indicator of true invariance; external verification using a post-hoc attacker is essential.
• Tuning the adversary's capacity and weight, as well as using an ensemble of adversaries, improves performance but does not fully eliminate residual demographic information.
• No single method consistently achieves complete removal of demographic attributes across all datasets and attributes, indicating inherent limitations in adversarial training for this task.
• The study concludes that adversarial training should not be trusted as a standalone method for achieving demographic invariance in text representations.