[Paper Review] Recovering Realistic Texture in Image Super-resolution by Deep Spatial Feature Transform

Xintao Wang, Ke Yu+2 more|arXiv (Cornell University)|Apr 9, 2018

Advanced Image Processing Techniques38 references102 citations

TL;DR

Introduces Spatial Feature Transform (SFT) layers to condition a single SR network on semantic segmentation priors, enabling realistic, class-specific texture recovery in a single forward pass.

ABSTRACT

Despite that convolutional neural networks (CNN) have recently demonstrated high-quality reconstruction for single-image super-resolution (SR), recovering natural and realistic texture remains a challenging problem. In this paper, we show that it is possible to recover textures faithful to semantic classes. In particular, we only need to modulate features of a few intermediate layers in a single network conditioned on semantic segmentation probability maps. This is made possible through a novel Spatial Feature Transform (SFT) layer that generates affine transformation parameters for spatial-wise feature modulation. SFT layers can be trained end-to-end together with the SR network using the same loss function. During testing, it accepts an input image of arbitrary size and generates a high-resolution image with just a single forward pass conditioned on the categorical priors. Our final results show that an SR network equipped with SFT can generate more realistic and visually pleasing textures in comparison to state-of-the-art SRGAN and EnhanceNet.

Motivation & Objective

• Motivate and address the challenge of recovering realistic textures in SR by leveraging semantic class priors.
• Develop a parameter-efficient conditioning mechanism to modulate SR features spatially.
• Demonstrate that segmentation-guided texture generation improves perceptual quality over prior SRGAN/EnhanceNet baselines.
• Enable end-to-end training of SR network with conditional SFT layers using standard losses.

Proposed method

• Propose Spatial Feature Transform (SFT) layers that generate per-location affine parameters (gamma, beta) from segmentation probability maps.
• Insert SFT layers into a single SR network to perform spatially-aware feature modulation across intermediate layers.
• Use a condition network to produce shared conditioning maps, broadcast to all SFT layers for efficiency.
• Train the generator with perceptual and adversarial (GAN) losses, using a VGG-based perceptual loss and a discriminator conditioned on category labels.
• Utilize segmentation probability maps as priors; segmentation network processes LR upsampled inputs to yield P maps for conditioning.
• Demonstrate that the approach can produce HR outputs in a single forward pass conditioned on semantic priors.

Experimental results

Research questions

• RQ1 Can semantic segmentation priors guide SR to produce textures faithful to semantic classes?
• RQ2 Does a spatially conditioned feature transform improve texture realism compared to prior SRGAN/EnhanceNet approaches?
• RQ3 Is it feasible to inject spatial priors into SR networks efficiently through a shared conditioning mechanism?
• RQ4 How does SR conditioned on segmentation maps perform across different outdoor scene categories (e.g., sky, building, grass, water, plant, animal)?

Key findings

• SFT-GAN generates richer, more realistic textures across defined outdoor categories than SRGAN and EnhanceNet in user studies.
• The model yields perceptually superior results with clear boundaries between semantic regions due to spatial modulation.
• Modulation parameters (gamma, beta) correlate with segmentation probability maps and preserve spatial region boundaries.
• A single forward pass with SFT layers conditioned on segmentation maps suffices to produce high-quality HR textures.
• Qualitative results show better texture fidelity for animal fur, building bricks, and grass textures in comparison to baselines.