[Paper Review] Perceptual Losses for Real-Time Style Transfer and Super-Resolution

Justin Johnson, Alexandre Alahi+1 more|arXiv (Cornell University)|Mar 27, 2016

Advanced Image Processing Techniques52 references1,031 citations

TL;DR

The paper trains feed-forward transformation networks using perceptual losses from a pretrained loss network (VGG-16) to achieve real-time style transfer and single-image super-resolution, delivering comparable style-transfer quality to Gatys et al. with 1000x speedup, and improving detail reproduction in super-resolution over per-pixel losses.

ABSTRACT

We consider image transformation problems, where an input image is transformed into an output image. Recent methods for such problems typically train feed-forward convolutional neural networks using a \emph{per-pixel} loss between the output and ground-truth images. Parallel work has shown that high-quality images can be generated by defining and optimizing \emph{perceptual} loss functions based on high-level features extracted from pretrained networks. We combine the benefits of both approaches, and propose the use of perceptual loss functions for training feed-forward networks for image transformation tasks. We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results.

Motivation & Objective

• Motivate image-to-image transformation tasks to benefit from perceptual, high-level feature losses rather than only per-pixel losses.
• Develop fast feed-forward transformation networks that can approximate optimization-based style transfer and improve super-resolution aesthetics.
• Demonstrate that perceptual losses enable semantic understanding transfer from a pretrained loss network to the transformer network.
• Show real-time performance for style transfer and qualitative improvements for super-resolution compared to per-pixel losses.

Proposed method

• Use a fixed loss network phi (pretrained on image classification, e.g., VGG-16) to define perceptual losses.
• Define and optimize feature reconstruction loss (content) and style reconstruction loss (texture) between the transformed output and targets.
• Train a feed-forward image transformation network f_W with residual blocks, downsampling/upsampling, and tanh output to produce images within [0,255].
• Train separate networks per style target (for style transfer) and per upsampling factor (for super-resolution).
• Incorporate a pixel or total variation loss as needed; rely primarily on perceptual losses for training.
• Evaluate in two tasks: style transfer (comparing to Gatys et al.) and single-image super-resolution (×4 and ×8).

Experimental results

Research questions

• RQ1 Can perceptual losses from a pretrained classifier enable a fast feed-forward network to approximate optimization-based style transfer results?
• RQ2 Does training with perceptual losses improve visual quality and perceptual realism in single-image super-resolution compared to per-pixel losses?
• RQ3 To what extent do higher-layer perceptual features preserve content versus texture for style transfer?
• RQ4 How does the proposed approach generalize to higher image resolutions beyond training size?

Key findings

• The style-transfer networks achieve results qualitatively similar to Gatys et al. but are three orders of magnitude faster, enabling real-time or video-rate processing (about 20 FPS for 512×512 images).
• For super-resolution, networks trained with a perceptual loss produce visually pleasing fine details and edges, better capturing structure than per-pixel loss methods, though PSNR/SSIM may be slightly reduced due to texture and high-frequency artifacts.
• The approach generalizes to larger images in a fully convolutional manner, allowing application to inputs larger than training size.
• Perceptual losses effectively transfer semantic knowledge from the loss network to the transformation network, enabling semantically aware image transformations.
• The method integrates with existing style-transfer objectives (Gatys et al.) while offering drastic speed improvements over optimization-based generation.