[Paper Review] Disentangling Content and Style via Unsupervised Geometry Distillation

Wayne Wu, Kaidi Cao+3 more|arXiv (Cornell University)|May 11, 2019

Generative Adversarial Networks and Image Synthesis32 references4 citations

TL;DR

This paper proposes an unsupervised disentanglement framework that separates image content and style using a two-branch autoencoder with geometry-based priors. By projecting latent content into a soft structured point tensor and enforcing spatial consistency via prior losses and KL regularization, the model achieves state-of-the-art disentanglement and visual analogy quality on real and synthetic datasets, generating high-fidelity $256\times256$ images with disentangled content and style representations without human annotations.

ABSTRACT

It is challenging to disentangle an object into two orthogonal spaces of content and style since each can influence the visual observation differently and unpredictably. It is rare for one to have access to a large number of data to help separate the influences. In this paper, we present a novel framework to learn this disentangled representation in a completely unsupervised manner. We address this problem in a two-branch Autoencoder framework. For the structural content branch, we project the latent factor into a soft structured point tensor and constrain it with losses derived from prior knowledge. This constraint encourages the branch to distill geometry information. Another branch learns the complementary style information. The two branches form an effective framework that can disentangle object's content-style representation without any human annotation. We evaluate our approach on four image datasets, on which we demonstrate the superior disentanglement and visual analogy quality both in synthesized and real-world data. We are able to generate photo-realistic images with 256*256 resolution that are clearly disentangled in content and style.

Motivation & Objective

• To learn disentangled content and style representations in a completely unsupervised manner, without human-annotated landmarks or attributes.
• To address the challenge of learning structural content from natural images without supervision, by leveraging geometric prior knowledge.
• To ensure disentanglement by enforcing content invariance in style representations and uniform, repeatable distribution of structural points across images.
• To achieve high-quality image generation and visual analogy with clear separation of content and style, validated on real-world and synthetic datasets.

Proposed method

• A two-branch autoencoder architecture: one branch learns structured content via a soft structured point tensor projection using a layer-wise softmax on the last layer.
• Prior knowledge is encoded through specialized losses that enforce high repeatability and uniform spatial distribution of structural points across images.
• The style branch learns complementary representations, with skip-connections and a KL divergence loss to encourage disentanglement and content-invariant style.
• The model uses a variational Bayesian framework with ELBO optimization, incorporating prior loss $L_{\text{prior}}$, reconstruction loss $L_{\text{recon}}$, and KL divergence loss $L_{\text{KL}}$.
• VGG feature loss is applied to improve perceptual quality and reduce blur in generated images.
• Latent space walking and content/style swapping are used to qualitatively validate disentanglement and manifold coverage.

Experimental results

Research questions

• RQ1 Can we learn disentangled content and style representations in a fully unsupervised setting without any human-annotated attributes or landmarks?
• RQ2 Can geometric prior knowledge—specifically, structured point distribution—effectively guide unsupervised disentanglement of content from style?
• RQ3 Does enforcing content-invariant style representation through KL regularization improve disentanglement and generation quality?
• RQ4 How does the proposed method compare to state-of-the-art unsupervised methods in terms of disentanglement, reconstruction, and visual analogy quality?

Key findings

• On the Cat face dataset, the method achieved a landmark detection error of 0.030%, significantly outperforming prior unsupervised methods (e.g., 0.038% for Jakab et al., 2018).
• The Inception Score on the Cat dataset reached 1.968, indicating high-quality and diverse image generation, with SSIM of 0.449, showing strong reconstruction fidelity.
• On CelebA, the method achieved a style consistency score of 3.886 × 10⁻⁶ and content consistency of 1.529 × 10⁻⁶, outperforming both baseline and SOTA unsupervised methods.
• Abllation studies showed that removing the VGG loss caused a 10% drop in Inception Score (from 1.968 to 1.796), indicating its critical role in reducing blur and improving perceptual quality.
• Removing the KL loss degraded the Inception Score to 1.720 and increased style consistency error to 6.556 × 10⁻⁵, demonstrating its key role in learning content-invariant style representations.
• Visualizations confirmed effective content-style disentanglement: content was preserved across columns (e.g., facial expression and pose), while style was transferred consistently across rows (e.g., hair color, lighting, makeup).