[Paper Review] Controllable Invariance through Adversarial Feature Learning

Qizhe Xie, Zihang Dai+3 more|arXiv (Cornell University)|May 31, 2017

Adversarial Robustness in Machine Learning18 references60 citations

TL;DR

The paper presents a generic adversarial framework to learn representations invariant to a specified attribute s while preserving task-relevant information, via a three-player minimax game among an encoder, a discriminator, and a predictor. It provides theoretical equilibrium analysis and demonstrates improved generalization across fair classification, language-independent generation, and lighting-independent image classification.

ABSTRACT

Learning meaningful representations that maintain the content necessary for a particular task while filtering away detrimental variations is a problem of great interest in machine learning. In this paper, we tackle the problem of learning representations invariant to a specific factor or trait of data. The representation learning process is formulated as an adversarial minimax game. We analyze the optimal equilibrium of such a game and find that it amounts to maximizing the uncertainty of inferring the detrimental factor given the representation while maximizing the certainty of making task-specific predictions. On three benchmark tasks, namely fair and bias-free classification, language-independent generation, and lighting-independent image classification, we show that the proposed framework induces an invariant representation, and leads to better generalization evidenced by the improved performance.

Motivation & Objective

• Motivate and formalize learning representations invariant to a specified factor s while preserving predictive power for y.
• Develop a generic, extensible adversarial framework that can handle discrete, continuous, or structured s.
• Provide theoretical analysis of equilibrium where invariant and predictive objectives are balanced.
• Demonstrate applicability across fair classification, multilingual translation, and image classification under varying lighting.
• Show empirical improvements over baselines and existing invariant-feature methods.

Proposed method

• Formulate a minimax game with an encoder E, a discriminator D, and a predictor M, optimizing J(E,M,D)=E_{x,s,y}[ γ log q_D(s|h) − log q_M(y|h) ] where h = E(x,s).
• Discriminator D tries to predict s from h to maximize q_D(s|h); encoder E and predictor M minimize this quantity to remove s from h while preserving y information.
• Provide theoretical analysis showing that optimal D and M given E are q_D*(s|h)=p̃(s|h) and q_M*(y|h)=p̃(y|h), reducing the objective to a balance of conditional entropies: −γ H(p̃(s|h)) + H(p̃(y|h)).
• Describe parametric instantiations for sentence generation (multilingual MT with language as s), fair classification (s as nuisance factors like gender/age), and image classification (lighting as s).
• Discuss optimization approaches, including gradient-reversal (GAN-like) and alternating training, and use Adam optimization.

Experimental results

Research questions

• RQ1 Can we learn a representation h that preserves task-relevant information y while removing information about a nuisance attribute s?
• RQ2 How does the balance between invariance and predictive accuracy (controlled by γ) affect equilibrium and performance?
• RQ3 Do adversarially learned invariant representations improve generalization across domain-independent tasks (fair classification, multilingual generation, lighting-invariant recognition) compared to standard baselines?
• RQ4 Is p(y|x,s) preferable to p(y|x) for certain tasks, and how does this affect cross-domain performance?
• RQ5 What are the practical effects and limitations of implementing the framework with discrete, continuous, or structured s?

Key findings

• The framework yields invariant representations and improved generalization across three domains: fair classification, multilingual translation, and lighting-invariant image classification.
• In fair classification, the learned representations reduce leakage of s while maintaining or improving accuracy on y, and improve unbiasedness across minority groups.
• In multilingual MT, the model with a discriminator outperforms bilingual and standard multilingual baselines, achieving higher BLEU scores (e.g., 36.1 vs 35.5 and 35.2/35.5 baselines across fr-en and de-en).
• In the Extended Yale B lighting-invariant task, the model achieves 89% identity classification accuracy vs 85% for the best baseline, while reducing s-predictability from 0.96 to 0.57, indicating effective removal of lighting information.
• Theoretical analysis shows a win-win equilibrium when s is irrelevant to y, and a competing equilibrium when s informs y, with γ modulating the trade-off between invariance and predictive power.