[Paper Review] Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network

Wenzhe Shi, José Caballero+6 more|arXiv (Cornell University)|Sep 16, 2016

Advanced Image Processing TechniquesComputer Science43 references210 citations

TL;DR

Introduces ESPCN, a CNN that performs SR in LR space with a sub-pixel convolution layer, enabling real-time 1080p video SR on a single K2 GPU and outperforming prior CNN methods in PSNR with much faster runtimes.

ABSTRACT

Recently, several models based on deep neural networks have achieved great success in terms of both reconstruction accuracy and computational performance for single image super-resolution. In these methods, the low resolution (LR) input image is upscaled to the high resolution (HR) space using a single filter, commonly bicubic interpolation, before reconstruction. This means that the super-resolution (SR) operation is performed in HR space. We demonstrate that this is sub-optimal and adds computational complexity. In this paper, we present the first convolutional neural network (CNN) capable of real-time SR of 1080p videos on a single K2 GPU. To achieve this, we propose a novel CNN architecture where the feature maps are extracted in the LR space. In addition, we introduce an efficient sub-pixel convolution layer which learns an array of upscaling filters to upscale the final LR feature maps into the HR output. By doing so, we effectively replace the handcrafted bicubic filter in the SR pipeline with more complex upscaling filters specifically trained for each feature map, whilst also reducing the computational complexity of the overall SR operation. We evaluate the proposed approach using images and videos from publicly available datasets and show that it performs significantly better (+0.15dB on Images and +0.39dB on Videos) and is an order of magnitude faster than previous CNN-based methods.

Motivation & Objective

• Motivate real-time single image and video super-resolution with reduced computational cost compared to upscaling to HR early in the network.
• Propose and validate an efficient sub-pixel convolution layer to reconstruct HR from LR feature maps.
• Demonstrate LR-space feature extraction and LR-to-HR upscaling at the end of the network to reduce complexity.
• Evaluate against existing SISR methods on standard datasets and show PSNR and speed improvements.

Proposed method

• Propose a CNN that processes LR images with L-1 layers to extract features, then uses a sub-pixel convolution (periodic shuffling) layer to generate HR output.
• Introduce a sub-pixel convolution layer that reshapes a feature map of size H×W×C·r² to rH×rW×C using a PS operator.
• Train with mean squared error on HR-LR pairs generated via Gaussian blur and downsampling with stride r.
• Show that last-layer upscaling with nL−1 learned filters per feature map is more efficient than pre-upscaling in HR space.
• Compare tanh vs relu activations and demonstrate superior performance of tanh in this SR setting.

Experimental results

Research questions

• RQ1 Can LR-space feature extraction combined with a learned upscaling (sub-pixel) layer achieve higher efficiency without sacrificing SR quality?
• RQ2 Does learning upscaling filters per feature map outperform fixed bicubic-like upscaling at the input?
• RQ3 What PSNR and speed gains does ESPCN achieve over existing CNN-based SR methods on standard image and video datasets?
• RQ4 Is real-time HD video SR feasible on a single GPU using the proposed architecture?

Key findings

• ESPCN outperforms prior CNN-based SR methods in PSNR on standard image benchmarks (average gains shown in the paper) and in video PSNR on HD datasets.
• The method achieves significantly faster runtimes, e.g., 4.7 ms per image for Set14 on a K2 GPU and 0.038 s per frame for 1080p video at scale 3 (0.029 s at scale 4).
• The approach yields +0.15 dB PSNR on images and +0.39 dB PSNR on videos compared to previous CNN approaches.
• The last-layer sub-pixel convolution enables HD video SR in real time on a single GPU, with an overall complexity reduction of about 2.5×r² compared to a SRCNN-like pipeline.
• Training with ImageNet data and using tanh as the final activation further improves SR performance compared to relu.
• The method is demonstrated to be nearly an order of magnitude faster than previous CNN-based SR methods while achieving state-of-the-art or competitive PSNR across multiple datasets.