[Paper Review] Swapping Autoencoder for Deep Image Manipulation

Taesung Park, Jun-Yan Zhu+5 more|arXiv (Cornell University)|Jul 1, 2020

Generative Adversarial Networks and Image SynthesisComputer Science88 references145 citations

TL;DR

The Swapping Autoencoder learns two disentangled latent codes (structure and texture) to enable realistic image editing via texture/structure swapping and latent-space arithmetic, with faster test-time embedding than prior GAN-based methods.

ABSTRACT

Deep generative models have become increasingly effective at producing realistic images from randomly sampled seeds, but using such models for controllable manipulation of existing images remains challenging. We propose the Swapping Autoencoder, a deep model designed specifically for image manipulation, rather than random sampling. The key idea is to encode an image with two independent components and enforce that any swapped combination maps to a realistic image. In particular, we encourage the components to represent structure and texture, by enforcing one component to encode co-occurrent patch statistics across different parts of an image. As our method is trained with an encoder, finding the latent codes for a new input image becomes trivial, rather than cumbersome. As a result, it can be used to manipulate real input images in various ways, including texture swapping, local and global editing, and latent code vector arithmetic. Experiments on multiple datasets show that our model produces better results and is substantially more efficient compared to recent generative models.

Motivation & Objective

• Address the challenge of editing existing images using deep generative models without retraining for each task.
• Learn a disentangled latent space where one code captures structure and the other texture, enabling realistic swaps.
• Provide an encoder–decoder framework that supports fast embedding of real images and versatile editing (global/local, texture transfer, and latent arithmetic).

Proposed method

• Propose Swapping Autoencoder with an encoder E and generator G built on StyleGAN2 architecture.
• Split latent code z into a structure code zs (spatial tensor) and a texture code yt (global vector).
• Train with reconstruction loss Lrec to ensure accurate input reconstruction.
• Use GAN losses: LGAN,rec to ensure realistic reconstructions and LGAN,swap to ensure realism of swapped hybrids (zs from one image, yt from another).
• Introduce a patch co-occurrence discriminator Dpatch that enforces that patches of swapped outputs share texture statistics with patches from the source texture image.
• Optimize a total loss Ltotal = Lrec + 0.5 LGAN,rec + 0.5 LGAN,swap + LCooccurGAN.
• Edge cases: keep zs local to preserve structure; yt enforces global texture consistency; architecture follows a fully convolutional pipeline enabling high-resolution editing.

Experimental results

Research questions

• RQ1 Can an unsupervised autoencoder learn separable structure and texture codes that support realistic image editing via code swapping?
• RQ2 Does a co-occurrence patch discriminator improve the interpretability and editability of the texture code?
• RQ3 Can test-time embedding be fast and accurate enough for practical image manipulation applications?
• RQ4 How do latent-space manipulations (texture/structure swaps and vector arithmetic) perform across diverse datasets?
• RQ5 Is the resulting embedding space suitable for local and global editing, and multi-domain transformations?

Key findings

• The model achieves realistic image hybrids by swapping structure and texture codes while preserving texture and structure where appropriate.
• Test-time embedding is significantly faster (about 1000x) than optimization-based baselines and achieves competitive reconstruction quality (LPIPS) across datasets.
• Human perceptual studies show Swapping Autoencoder outperforms baselines in realism of swap-generated images across Church, FFHQ, and Waterfall datasets (AMT fooling rates: 31.3% for Church, 19.4% for FFHQ, 41.8% for Waterfall; average 31.0%).
• Latent-space operations yield smooth and controllable edits, including region-based edits and domain translations, via vector arithmetic in the texture and structure spaces.
• The method supports multi-domain, fine-grained editing and continuous transformations (e.g., adding snow) via mean-difference vectors between domains.
• Compared to Im2StyleGAN, StyleGAN2, STROTSS, and WCT 2, Swapping Autoencoder achieves higher perceptual realism in image hybrids and better style-content balance in user studies.