[Paper Review] Resolution Dependent GAN Interpolation for Controllable Image Synthesis Between Domains

Justin N. M. Pinkney, Doron Adler|arXiv (Cornell University)|Oct 11, 2020

Generative Adversarial Networks and Image SynthesisComputer Science7 references61 citations

TL;DR

The paper introduces resolution-dependent layer swapping between pre-trained StyleGAN models to interpolate between domains, enabling novel domain generation with controllable stylistic features.

ABSTRACT

GANs can generate photo-realistic images from the domain of their training data. However, those wanting to use them for creative purposes often want to generate imagery from a truly novel domain, a task which GANs are inherently unable to do. It is also desirable to have a level of control so that there is a degree of artistic direction rather than purely curation of random results. Here we present a method for interpolating between generative models of the StyleGAN architecture in a resolution dependent manner. This allows us to generate images from an entirely novel domain and do this with a degree of control over the nature of the output.

Motivation & Objective

• Motivate and enable generation from novel domains by interpolating between generative models while preserving controllable features tied to image resolution.
• Leverage the known role of different StyleGAN layers in capturing features at specific resolutions to achieve controlled cross-domain synthesis.
• Demonstrate a practical method for toonification and ukiyo-e style transfer through layer-based model interpolation.

Proposed method

• Define a resolution-based layer swapping scheme to interpolate between a base model and a transferred model.
• Train a transferred model via transfer learning on a new domain and fix the mapping network parameters to those of the base model.
• Construct interpolated weights p_interp by selecting per-layer contributions from base or transferred models depending on layer resolution r and a swap threshold r_swap.
• Use the interpolated model to generate images that combine structural characteristics from one domain with rendering from the other."
• Provide an encoding/embedding approach to map arbitrary faces into the base model's latent space for toonified outputs.

Experimental results

Research questions

• RQ1 Can resolution-dependent layer swapping between StyleGAN models produce images in a novel domain not present in either base or transferred datasets?
• RQ2 To what extent can we control the balance between structural features and rendering quality by choosing which resolution layers are swapped?
• RQ3 How does encoding an arbitrary face into the base model affect toonified outputs when using the interpolated model?
• RQ4 What are practical transfer-learning settings (datasets, training duration) that yield stable interpolation results within StyleGAN2 frameworks?

Key findings

• Layer swapping at different resolutions enables generation of images with mixed-domain characteristics, e.g., cartoon-like structure with photo-realistic rendering.
• Interpolated models can preserve broad identity features when using the same latent vector across base and interpolated models.
• High-resolution layers from the base model combined with low-resolution layers from a transferred model yield outputs with cartoon structural traits and realistic textures.
• The same latent input can produce Toonified outputs that retain identity while adopting new stylistic structures.