[Paper Review] Instance Normalization: The Missing Ingredient for Fast Stylization

Dmitry Ulyanov, Andrea Vedaldi+1 more|arXiv (Cornell University)|Jul 27, 2016

Handwritten Text Recognition Techniques3 references3,103 citations

TL;DR

The paper shows that replacing batch normalization with instance normalization in fast stylization generators (and applying it at test time) yields significantly better quality images in real-time stylization.

ABSTRACT

It this paper we revisit the fast stylization method introduced in Ulyanov et. al. (2016). We show how a small change in the stylization architecture results in a significant qualitative improvement in the generated images. The change is limited to swapping batch normalization with instance normalization, and to apply the latter both at training and testing times. The resulting method can be used to train high-performance architectures for real-time image generation. The code will is made available on github at https://github.com/DmitryUlyanov/texture_nets. Full paper can be found at arXiv:1701.02096.

Motivation & Objective

• Motivate real-time style transfer by improving generator architecture quality.
• Investigate whether contrast normalization can be efficiently implemented within the architecture.
• Demonstrate that instance normalization leads to superior stylization results over batch normalization.

Proposed method

• Replace batch normalization layers with instance normalization layers in the stylization generator networks.
• Apply instance normalization at both training and testing times (no freezing or removal).
• Use the same generator architectures as prior fast stylization methods and retrain with instance normalization.
• Provide intuition that instance normalization removes instance-specific contrast information to simplify generation.
• Compare performance across two generator architectures to assess generality of the modification.

Experimental results

Research questions

• RQ1 Does instance normalization improve the quality of fast stylization compared to batch normalization?
• RQ2 Is applying instance normalization at test time important for achieving higher-quality stylizations?
• RQ3 Do different generator architectures both benefit from instance normalization in this task?

Key findings

• Replacing batch normalization with instance normalization significantly improves stylization quality across tested architectures.
• Both the Ulyanov et al. (2016) and Johnson et al. (2016) generated networks benefit similarly from instance normalization.
• Using instance normalization at train and test times yields better results than keeping batch normalization behavior.
• The more efficient residuals architecture (Johnson et al., 2016) remains favorable after the switch to instance normalization.