[Paper Review] 3DSRnet: Video Super-resolution using 3D Convolutional Neural Networks

Soo Ye Kim, Jeongyeon Lim+2 more|arXiv (Cornell University)|Dec 21, 2018

Advanced Image Processing Techniques25 references43 citations

TL;DR

Proposes a 3D-CNN based video super-resolution framework (3DSRnet) that preserves temporal depth, uses residual learning with sub-pixel outputs, and includes a scene-change handling module to avoid performance drops without motion alignment. It achieves state-of-the-art PSNR gains on Vidset4.

ABSTRACT

In video super-resolution, the spatio-temporal coherence between, and among the frames must be exploited appropriately for accurate prediction of the high resolution frames. Although 2D convolutional neural networks (CNNs) are powerful in modelling images, 3D-CNNs are more suitable for spatio-temporal feature extraction as they can preserve temporal information. To this end, we propose an effective 3D-CNN for video super-resolution, called the 3DSRnet that does not require motion alignment as preprocessing. Our 3DSRnet maintains the temporal depth of spatio-temporal feature maps to maximally capture the temporally nonlinear characteristics between low and high resolution frames, and adopts residual learning in conjunction with the sub-pixel outputs. It outperforms the most state-of-the-art method with average 0.45 and 0.36 dB higher in PSNR for scales 3 and 4, respectively, in the Vidset4 benchmark. Our 3DSRnet first deals with the performance drop due to scene change, which is important in practice but has not been previously considered.

Motivation & Objective

• Exploit spatio-temporal coherence in video SR without motion alignment.
• Develop an end-to-end 3D-CNN that preserves temporal information across layers.
• Address scene changes in input frames via a dedicated scene-change subnet.
• Improve SR performance using residual learning and sub-pixel output structures.
• Demonstrate state-of-the-art PSNR gains on Vidset4 compared to prior methods.

Proposed method

• Use a 3D-CNN video SR subnet with five-frame input windows to produce a single high-resolution middle frame.
• Maintain temporal depth in feature maps across layers via extrapolation (padding) of GTFMs to preserve temporal information.
• Employ a multi-channel output to realize direct LR-to-HR mapping without bicubic upscaling in the network input path.
• Apply residual learning by predicting the residual between HR and bicubic-upscaled LR frames.
• Incorporate a lightweight scene-change detection and frame replacement (SF) subnet to locate scene boundaries and replace frames from other scenes with temporally closest frames from the same scene.
• Train separate datasets for the video SR subnet (smallSet and largeSet) and for the SF subnet; optimize with mean squared error loss and Adam optimization.

Experimental results

Research questions

• RQ1 Can a 3D-CNN without explicit motion alignment effectively exploit spatio-temporal information for video SR?
• RQ2 Does preserving temporal depth in 3D feature maps improve SR performance over traditional 2D-CNN video SR approaches?
• RQ3 How does a dedicated scene-change detection and frame replacement component affect SR quality in videos with scene transitions?
• RQ4 How does 3DSRnet compare to state-of-the-art image and video SR methods on Vidset4 in terms of PSNR and SSIM?

Key findings

• 3DSRnet outperforms state-of-the-art methods on Vidset4 by average 0.45 dB in PSNR for scale 3 and 0.36 dB for scale 4.
• The model maintains temporal depth through multiple 3D convolution layers and uses extrapolation to preserve temporal information until the final 2D HR output.
• The SF subnet achieves high scene-change detection accuracy (around 99.9%) and improves PSNR by about 0.25–0.46 dB on frames affected by scene changes.
• On Vidset4, 3DSRnet achieves PSNR gains across scale factors 2, 3, and 4 compared to contemporaneous methods, as indicated by Table 5 results.
• Inference time on NVIDIA TITAN X is 166 ms for scale 2 and 788 ms for scale 4 to upscale a 960x540 frame from five input frames.
• The network benefits from a multi-channel output and residual learning, achieving higher perceptual quality and preventing excessive computation for 3D convolutions.