[Paper Review] Invariant Representations without Adversarial Training

Daniel Moyer, Shuyang Gao+3 more|arXiv (Cornell University)|May 24, 2018

Adversarial Robustness in Machine Learning24 references97 citations

TL;DR

The paper proposes learning c-invariant representations by directly minimizing an information-theoretic upper bound on I(z,c) within VAE and VIB frameworks, avoiding adversarial training while matching or exceeding state-of-the-art methods for fair representations and controllable generative transformations.

ABSTRACT

Representations of data that are invariant to changes in specified factors are useful for a wide range of problems: removing potential biases in prediction problems, controlling the effects of covariates, and disentangling meaningful factors of variation. Unfortunately, learning representations that exhibit invariance to arbitrary nuisance factors yet remain useful for other tasks is challenging. Existing approaches cast the trade-off between task performance and invariance in an adversarial way, using an iterative minimax optimization. We show that adversarial training is unnecessary and sometimes counter-productive; we instead cast invariant representation learning as a single information-theoretic objective that can be directly optimized. We demonstrate that this approach matches or exceeds performance of state-of-the-art adversarial approaches for learning fair representations and for generative modeling with controllable transformations.

Motivation & Objective

• Motivation to remove dependence on nuisance covariates c while preserving task-relevant information.
• Replace adversarial training with a tractable information-theoretic objective.
• Adapt VAE and Variational Information Bottleneck (VIB) frameworks to enforce invariance.
• Demonstrate practical benefits in fair classification and controllable image generation.

Proposed method

• Formulate invariant coding as minimizing L + λ I(z,c) where L is task-related loss.
• Derive a variational upper bound for I(z,c) leading to a practical loss involving KL(q(z|x) || p(z)) and a reconstruction term p(x|z,c).
• Approximate KL[q(z|x) || q(z)] to avoid computing the intractable marginal q(z) using pairwise KL(q(z|x) || q(z|x')).
• Provide three-branch models for supervised settings: encoder q(z|x), decoder p(x|z,c), and predictor p(y|z).
• Show how to obtain an adversarial interpretation from the alternative bound I(z,c) ≤ H(c) − H(c|z) and connecting it to inferring c from z.

Experimental results

Research questions

• RQ1 Does minimizing an upper bound on I(z,c) produce invariant latent codes without adversarial training?
• RQ2 How does c-invariant VAE/VIB perform on fair classification benchmarks compared to adversarial baselines?
• RQ3 Can the proposed objective enable controllable transformations in generative modeling (Fader-like capabilities) without access to c at test time?
• RQ4 What is the practical impact of replacing adversarial training with a variational bound on mutual information?

Key findings

• The proposed c-agnostic objective matches or exceeds adversarial methods on fair classification tasks.
• The method provides competitive adversarial error across different adversary strengths, without requiring c at test time.
• In MNIST experiments, the approach enables Fader-network-like manipulation by conditioning the decoder on z and a target c.
• t-SNE visualizations show invariant representations clustering less by c under the proposed method compared with baselines.
• The variational bound approach avoids training instability often observed with adversarial training.