[Paper Review] Photographic Image Synthesis with Cascaded Refinement Networks

Qifeng Chen, Vladlen Koltun|arXiv (Cornell University)|Jul 28, 2017

Generative Adversarial Networks and Image Synthesis46 references80 citations

TL;DR

The paper introduces Cascaded Refinement Networks (CRN) to synthesize photorealistic images from semantic layouts with a single feedforward model trained by a perceptual regression loss, scalable to 2 megapixels and outperforming GAN-based baselines.

ABSTRACT

We present an approach to synthesizing photographic images conditioned on semantic layouts. Given a semantic label map, our approach produces an image with photographic appearance that conforms to the input layout. The approach thus functions as a rendering engine that takes a two-dimensional semantic specification of the scene and produces a corresponding photographic image. Unlike recent and contemporaneous work, our approach does not rely on adversarial training. We show that photographic images can be synthesized from semantic layouts by a single feedforward network with appropriate structure, trained end-to-end with a direct regression objective. The presented approach scales seamlessly to high resolutions; we demonstrate this by synthesizing photographic images at 2-megapixel resolution, the full resolution of our training data. Extensive perceptual experiments on datasets of outdoor and indoor scenes demonstrate that images synthesized by the presented approach are considerably more realistic than alternative approaches. The results are shown in the supplementary video at https://youtu.be/0fhUJT21-bs

Motivation & Objective

• Develop a model to generate photorealistic images conditioned on pixelwise semantic layouts.
• Avoid adversarial training and demonstrate a stable, end-to-end regression approach.
• Scale image synthesis to high resolutions (up to 2 MP) while preserving realism.
• Evaluate realism against strong baselines using human perceptual studies.
• Explore diversity in outputs for a given semantic layout.

Proposed method

• Propose a Cascaded Refinement Network (CRN): a sequence of refinement modules across increasing resolutions starting from a 4x8 layout.
• Each module M^i takes downsampled layout L and upsampled F^{i-1}, concatenates them, and outputs a new feature map at its resolution.
• Use 3x3 convolutions, layer normalization, and Leaky ReLU; final output is a 1x1 projection to 3 color channels.
• Train end-to-end with a perceptual (content) loss based on differences in VGG-19 activations between the synthesized image and the reference image: L_{I,L} = sum_l lambda_l || Phi_l(I) - Phi_l(g(L;theta)) ||_1.
• Employ a multi-scale loss by using features from multiple VGG layers (conv1_2, conv2_2, conv3_2, conv4_2, conv5_2).
• Optionally extend to generate diverse outputs by outputting k images and using a diversity-driven loss that selects the best content per semantic class.

Experimental results

Research questions

• RQ1 Can a single feedforward network reproduce photorealistic images from semantic layouts without adversarial training?
• RQ2 Does a cascaded, multi-resolution architecture enable high-fidelity synthesis at high resolutions?
• RQ3 How does CRN compare to GAN-based or other baselines in terms of perceived realism?
• RQ4 Can the approach produce diverse outputs for the same semantic layout?
• RQ5 What is the effect of perceptual (content) loss on synthesis quality and realism?

Key findings

• CRN images are rated more realistic than multiple baselines (including GAN-based Isola et al.) in perceptual experiments.
• The approach scales to high resolution (up to 2 megapixels) with end-to-end training.
• A perceptual loss over a pre-trained network (VGG-19) guides synthesis toward both fine details and global structure.
• Diversity loss can produce a collection of plausible outputs for a single semantic layout, distributing variability across semantic classes.
• Time-limited human studies show CRN rivals real images early in viewing, indicating high perceived realism.
• Across Cityscapes and NYU datasets, CRN consistently outperforms baselines in pairwise realism judgments.