[Paper Review] Semantic Style Transfer and Turning Two-Bit Doodles into Fine Artworks

Alex J. Champandard|arXiv (Cornell University)|Mar 5, 2016

Generative Adversarial Networks and Image SynthesisComputer Science7 references205 citations

TL;DR

The paper augments CNN-based style transfer with semantic annotations to provide content-aware control, improving quality and enabling doodle-to-artwork transformations.

ABSTRACT

Convolutional neural networks (CNNs) have proven highly effective at image synthesis and style transfer. For most users, however, using them as tools can be a challenging task due to their unpredictable behavior that goes against common intuitions. This paper introduces a novel concept to augment such generative architectures with semantic annotations, either by manually authoring pixel labels or using existing solutions for semantic segmentation. The result is a content-aware generative algorithm that offers meaningful control over the outcome. Thus, we increase the quality of images generated by avoiding common glitches, make the results look significantly more plausible, and extend the functional range of these algorithms---whether for portraits or landscapes, etc. Applications include semantic style transfer and turning doodles with few colors into masterful paintings!

Motivation & Objective

• Address unpredictability and artifacts in neural style transfer by injecting semantic information.
• Enable content-aware style transfer and segmentation-aware manipulation for portraits, landscapes, and beyond.
• Integrate pixel labeling/semantic segmentation with patch-based style transfer to improve plausibility.
• Provide user-controllable mechanisms via manually authored or automated semantic maps.
• Demonstrate semantic style transfer and image synthesis by analogy (doodle-to-paintings).

Proposed method

• Augment the CNN by concatenating semantic map channels m^l with activations x^l, weighted by a parameter gamma.
• Adopt a patch-based style transfer with semantic-aware nearest-neighbor matching using normalized cross-correlation.
• Represent input semantic maps with M channels, down-sampled to match feature resolutions, and concatenate them to form s^l.
• Compute style loss E_s using nearest-neighbor patches across semantic-augmented features as in E_s(s,s_s) = sum_i ||Psi_i(s) - Psi_NN(i)(s_s)||^2.
• Use an augmented architecture compatible with existing patch-based implementations; allow semantic maps to be static during optimization.
• Optimize with an objective E = alpha E_c + beta E_s using L-BFGS; gamma controls semantic influence; beta controls style strength.

Experimental results

Research questions

• RQ1 Does incorporating semantic maps reduce artifacts and increase controllability in style transfer compared to semantically unaware methods?
• RQ2 How can pixel labeling/semantic segmentation be integrated into patch-based style transfer without overhauling existing algorithms?
• RQ3 How do style weight beta and semantic weight gamma affect content accuracy and style fidelity?
• RQ4 Can the approach handle portrait and other semantic-rich regions more faithfully (e.g., hair, skin, background) in style transfer?
• RQ5 What are the computational and memory implications of adding semantic channels to the generation process?

Key findings

• Semantic annotations reduce glitches and artifacts, yielding more plausible results, especially in portraits and semantically segmented regions.
• A semantic weighting parameter gamma provides controllable influence of the annotation; a default gamma around 50 balances semantic channels with activations.
• Portrait results show improved transfer of skin tones and background colors when semantic maps are used.
• The method maintains compatibility with existing patch-based style transfer algorithms and can interpolate between semantic and non-semantic operation by adjusting gamma and beta.
• Memory and computation increase with semantic channels (roughly 1% extra memory for RGB, about 5% extra computation; increases are larger with pixel-wise labels).
• Rendering times for the experiments are in the range of several minutes on a GTX 970 with 4 GB RAM (3–8 minutes depending on resolution).