[Paper Review] Exploring Disentangled Feature Representation Beyond Face Identification

Yu Liu, Fangyin Wei+4 more|arXiv (Cornell University)|Apr 10, 2018

Face recognition and analysis15 references17 citations

TL;DR

This paper proposes the Identity Distilling and Dispelling Autoencoder (D²AE), a minimal-supervision framework that learns disentangled face features by adversarially training two streams: one to distill identity-specific features for verification and another to dispel identity information, capturing complementary attributes and variances. The method achieves state-of-the-art 99.80% accuracy on LFW and enables unsupervised semantic face editing with strong generalization across identity and attribute tasks.

ABSTRACT

This paper proposes learning disentangled but complementary face features with minimal supervision by face identification. Specifically, we construct an identity Distilling and Dispelling Autoencoder (D2AE) framework that adversarially learns the identity-distilled features for identity verification and the identity-dispelled features to fool the verification system. Thanks to the design of two-stream cues, the learned disentangled features represent not only the identity or attribute but the complete input image. Comprehensive evaluations further demonstrate that the proposed features not only maintain state-of-the-art identity verification performance on LFW, but also acquire competitive discriminative power for face attribute recognition on CelebA and LFWA. Moreover, the proposed system is ready to semantically control the face generation/editing based on various identities and attributes in an unsupervised manner.

Motivation & Objective

• To learn disentangled, complementary face features with minimal supervision, specifically using only identity labels.
• To improve generalization and completeness of feature representations by simultaneously learning identity-distilled and identity-dispelled features.
• To enable zero-shot transfer to downstream tasks like face attribute recognition and unsupervised face editing without fine-tuning.
• To provide a unified, end-to-end framework that maintains high performance in identity verification while capturing intra-personal and attribute-level variations.
• To explore the interpretability and disentanglement quality of features through adversarial supervision and visualization.

Proposed method

• The D²AE framework uses a two-stream autoencoder architecture: one stream learns identity-distilled features via identity classification loss, and the other learns identity-dispelled features via adversarial training to fool the identity classifier.
• The identity-dispelled stream is trained using an adversarial loss (L_I^adv) that encourages the feature extractor to produce representations that are non-informative for identity prediction.
• A reconstruction loss (L_H) ensures that the autoencoder preserves the full input image information in the latent space, maintaining completeness.
• The model is trained end-to-end with only identity supervision, avoiding the need for additional annotations for attributes or other tasks.
• The disentangled features are formed by combining the identity-distilled and identity-dispelled streams, enabling both identity-specific and attribute-specific representations.
• The framework supports semantic face editing by manipulating the disentangled components without retraining or control units.

Experimental results

Research questions

• RQ1 Can disentangled face features be learned with only identity supervision, without explicit annotations for attributes or other factors?
• RQ2 Does adversarial training of identity-dispelled features improve the completeness and generalization of the learned representation?
• RQ3 To what extent do the disentangled features maintain state-of-the-art performance in identity verification while enabling zero-shot attribute recognition?
• RQ4 Can the disentangled features support unsupervised, semantic face editing such as identity-preserving attribute modification?
• RQ5 How do the identity-distilled and identity-dispelled components contribute differently to various downstream tasks like verification and attribute classification?

Key findings

• The D²AE model achieves 99.80% accuracy on the LFW face verification benchmark, outperforming most state-of-the-art methods with a single model.
• On the CelebA and LFWA attribute recognition benchmarks, D²AE achieves competitive performance, with 83.1% average accuracy on LFWA and 83.1% on CelebA, demonstrating strong disentanglement.
• The ablation study confirms that removing either the identity-adversarial loss (L_I^adv) or the reconstruction loss (L_H) significantly degrades performance, validating their critical role.
• The identity-dispelled features (D²AE-𝒫) outperform the identity-distilled features (D²AE-𝒯) on ID-irrelevant attributes like 'Smiling' and 'Wearing Lipstick', while the reverse holds for ID-relevant attributes like 'Male' and 'Black Hair'.
• The model enables effective semantic face editing: by manipulating the disentangled components, users can perform identity-preserving attribute transfer and interpolation without training control modules.
• The t-SNE visualization shows that the disentangled features are trivially separable by attributes, confirming their interpretability and disentanglement quality.