[Paper Review] The Variational Fair Autoencoder

Christos Louizos, Kevin Swersky+3 more|UvA-DARE (University of Amsterdam)|Nov 3, 2015

Ethics and Social Impacts of AISocial Sciences179 citations

TL;DR

The Variational Fair Autoencoder (VFAE) learns disentangled, invariant representations by combining a variational autoencoder with a Maximum Mean Discrepancy (MMD) penalty to remove dependencies between sensitive factors (e.g., gender, lighting) and the latent representation. It achieves state-of-the-art performance in fair representation learning and domain adaptation, reducing sensitive attribute accuracy from 96% to 50% while improving classification accuracy from 78% to 85% on the Extended Yale B dataset.

ABSTRACT

We investigate the problem of learning representations that are invariant to certain nuisance or sensitive factors of variation in the data while retaining as much of the remaining information as possible. Our model is based on a variational autoencoding architecture with priors that encourage independence between sensitive and latent factors of variation. Any subsequent processing, such as classification, can then be performed on this purged latent representation. To remove any remaining dependencies we incorporate an additional penalty term based on the "Maximum Mean Discrepancy" (MMD) measure. We discuss how these architectures can be efficiently trained on data and show in experiments that this method is more effective than previous work in removing unwanted sources of variation while maintaining informative latent representations.

Motivation & Objective

• To learn disentangled latent representations that are invariant to known nuisance or sensitive factors of variation in the data.
• To address limitations in prior fair representation learning methods that rely on clustering or fail to penalize higher-order moment dependencies.
• To improve generalization in classification tasks by purging information about sensitive attributes while preserving task-relevant information.
• To extend variational autoencoders with a dual regularization strategy: factorized priors and MMD-based posterior alignment to enforce invariance.
• To demonstrate effectiveness in both fair representation learning and domain adaptation, outperforming existing methods in accuracy-invariance trade-offs.

Proposed method

• The model uses a deep variational autoencoder architecture with a factorized prior p(z)p(s) to encourage marginal independence between the latent code z and sensitive variable s.
• The encoder qϕ(z|x,s) is trained to produce a posterior that is conditionally independent of s by minimizing KL divergence between the posterior and prior.
• An MMD penalty is applied to minimize differences in the full posterior distributions qϕ(z|s=k) across all values k of the sensitive variable s, targeting all moments of the latent distribution.
• The objective function combines the standard VAE ELBO with an MMD regularization term to jointly optimize representation quality and invariance.
• The model is trained end-to-end using the SGVB (reparameterization) gradient estimator, enabling differentiable optimization of both the VAE and MMD components.
• The method supports both unsupervised and semi-supervised learning, with the latter enabling joint optimization of representation learning and downstream classification.

Experimental results

Research questions

• RQ1 Can a variational autoencoder be effectively extended to learn representations that are invariant to sensitive or nuisance factors of variation in the data?
• RQ2 How can higher-order statistical dependencies between sensitive attributes and latent representations be removed beyond simple mean-level alignment?
• RQ3 Does combining factorized priors with MMD regularization lead to better disentanglement and invariance than prior methods that use only one form of regularization?
• RQ4 To what extent can such a model improve fairness and generalization in downstream classification tasks?
• RQ5 How does the VFAE compare to adversarial and domain-regularized methods in learning invariant representations for domain adaptation?

Key findings

• On the Extended Yale B dataset, the VFAE reduced sensitive attribute (lighting) classification accuracy from 96% to approximately 50%, indicating strong removal of unwanted variation.
• The same model improved classification accuracy on the identity label from 78% to 85%, demonstrating improved task performance after removing nuisance factors.
• The VFAE achieved a better trade-off between invariance and accuracy than the NN + MMD baseline, which reported only 82% accuracy on the same task.
• t-SNE visualizations showed that after VFAE processing, data points clustered primarily by identity (y) rather than lighting (s), confirming successful disentanglement.
• The model outperformed prior approaches in fair representation learning by addressing higher-order moment leakage that clustering-based penalties fail to capture.
• The VFAE demonstrated competitive performance against adversarial domain adaptation methods, validating its effectiveness in learning invariant representations.