[Paper Review] Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesis

Dmitry Ulyanov, Andrea Vedaldi+1 more|arXiv (Cornell University)|Jan 9, 2017

Generative Adversarial Networks and Image Synthesis17 references4 citations

TL;DR

This paper improves feed-forward neural networks for image stylization and texture synthesis by introducing instance normalization to boost visual quality and a diversity-encouraging learning objective based on the Julesz ensemble to enhance output variety. The result is faster, higher-quality, and more diverse stylized images and textures, approaching the performance of slow optimization-based methods while maintaining real-time inference.

ABSTRACT

The recent work of Gatys et al., who characterized the style of an image by the statistics of convolutional neural network filters, ignited a renewed interest in the texture generation and image stylization problems. While their image generation technique uses a slow optimization process, recently several authors have proposed to learn generator neural networks that can produce similar outputs in one quick forward pass. While generator networks are promising, they are still inferior in visual quality and diversity compared to generation-by-optimization. In this work, we advance them in two significant ways. First, we introduce an instance normalization module to replace batch normalization with significant improvements to the quality of image stylization. Second, we improve diversity by introducing a new learning formulation that encourages generators to sample unbiasedly from the Julesz texture ensemble, which is the equivalence class of all images characterized by certain filter responses. Together, these two improvements take feed forward texture synthesis and image stylization much closer to the quality of generation-via-optimization, while retaining the speed advantage.

Motivation & Objective

• Address the visual quality gap between fast feed-forward stylization networks and slower optimization-based methods like Gatys et al.'s.
• Improve diversity in generated textures and stylized images, which is limited in existing feed-forward networks despite their speed.
• Develop a training formulation that encourages generators to sample uniformly from the Julesz ensemble, ensuring diverse outputs.
• Demonstrate that architectural changes and a new learning objective can significantly enhance both quality and diversity without sacrificing inference speed.

Proposed method

• Replace batch normalization with instance normalization in generator networks, especially for stylization, to better normalize content-dependent features and improve training stability and performance.
• Formulate a learning objective that minimizes the Kullback-Leibler divergence between the generated distribution and a quasi-uniform distribution over the Julesz ensemble, promoting diversity.
• Estimate the entropy of generated samples using a differentiable non-parametric estimator to enable end-to-end training of the diversity-inducing objective.
• Integrate the entropy term into the loss function as a negative entropy penalty, balancing reconstruction fidelity and diversity via a hyperparameter λ.
• Train generators using a combined loss: a standard style reconstruction loss (similar to Gatys et al.) minus λ times the estimated entropy of the generated samples.
• Use noise z as a random seed in the generator g(x₀, z), enabling diverse outputs for the same content image x₀ when z is varied.

Experimental results

Research questions

• RQ1 Can instance normalization significantly improve the visual quality of feed-forward stylization networks compared to batch normalization?
• RQ2 Does a learning objective that encourages uniform sampling from the Julesz ensemble lead to more diverse texture and stylized image generation?
• RQ3 Can the combination of instance normalization and entropy-based diversity regularization produce results that rival the quality of optimization-based methods while retaining real-time inference?
• RQ4 How does the diversity hyperparameter λ affect the trade-off between visual fidelity and output diversity in texture and stylization networks?
• RQ5 To what extent can feed-forward networks trained with these methods match the perceptual quality of slow, iterative optimization techniques?

Key findings

• Instance normalization significantly improves training convergence and visual quality in stylization networks, with StyleNet IN achieving lower training loss and better qualitative results than StyleNet BN.
• The use of instance normalization allows the generator to start optimization from a better initialization point, reducing the number of optimization steps needed to reach high-quality results.
• The entropy-based diversity regularization enables high-capacity generators like TextureNetV2 to produce diverse outputs without sacrificing visual quality, unlike standard training which leads to mode collapse.
• When λ is tuned appropriately, the diversity term effectively prevents mode collapse and produces varied, high-fidelity textures and stylized images, while avoiding artifacts that occur with overly high λ values.
• The proposed method achieves visual quality comparable to the slow optimization-based method of Gatys et al., as confirmed by qualitative comparisons where human observers preferred results from the new method over existing fast networks.
• The method maintains real-time inference speed, with generators producing stylized images and textures in a single forward pass, three orders of magnitude faster than iterative optimization.