[Paper Review] Space-Time-Aware Multi-Resolution Video Enhancement

Muhammad Haris, Greg Shakhnarovich+1 more|arXiv (Cornell University)|Mar 30, 2020

Advanced Image Processing Techniques57 references4 citations

TL;DR

This paper proposes STARnet, a deep learning framework for joint space-time super-resolution (ST-SR) that simultaneously enhances video resolution in both spatial and temporal dimensions. By leveraging mutual dependencies between space and time through multi-resolution feature learning and direct skip connections from low- to high-resolution features, STARnet achieves state-of-the-art performance on ST-SR, S-SR, and T-SR tasks across multiple benchmarks, outperforming sequential or independent methods by significant margins.

ABSTRACT

We consider the problem of space-time super-resolution (ST-SR): increasing spatial resolution of video frames and simultaneously interpolating frames to increase the frame rate. Modern approaches handle these axes one at a time. In contrast, our proposed model called STARnet super-resolves jointly in space and time. This allows us to leverage mutually informative relationships between time and space: higher resolution can provide more detailed information about motion, and higher frame-rate can provide better pixel alignment. The components of our model that generate latent low- and high-resolution representations during ST-SR can be used to finetune a specialized mechanism for just spatial or just temporal super-resolution. Experimental results demonstrate that STARnet improves the performances of space-time, spatial, and temporal video super-resolution by substantial margins on publicly available datasets.

Motivation & Objective

• To address the limitation of existing super-resolution methods that treat spatial and temporal upsampling independently.
• To develop a unified deep learning framework that jointly optimizes spatial and temporal super-resolution for improved performance.
• To exploit the complementary relationship between high-resolution spatial details and high-frame-rate motion information to enhance video reconstruction.
• To enable transfer learning by finetuning the same model for standalone spatial or temporal super-resolution tasks.
• To demonstrate that joint learning in multiple resolutions improves accuracy in both large and subtle motion estimation.

Proposed method

• STARnet employs a multi-resolution encoder-decoder architecture that learns features at both low- and high-resolution space-time representations.
• The network uses direct skip connections (purple arrows in Fig. 1) from low-resolution inputs to high-resolution outputs, enabling end-to-end joint optimization of ST-SR.
• Spatial and temporal features are jointly optimized via lateral connections across multiple resolutions, enhancing motion estimation and reconstruction fidelity.
• The model uses optical flow and residual learning to improve alignment and feature refinement during up-sampling.
• A shared backbone allows finetuning for standalone S-SR or T-SR, where the model inherits ST-SR-aware representations.
• The framework supports both $4\times$ spatial and $2\times$ temporal up-sampling, with ablation studies validating the contribution of multi-resolution learning.

Experimental results

Research questions

• RQ1 Can joint learning of spatial and temporal super-resolution improve video reconstruction beyond sequential or independent processing?
• RQ2 How does multi-resolution feature learning enhance the modeling of both large and subtle motions in video?
• RQ3 To what extent does direct skip connection from low- to high-resolution features improve ST-SR performance compared to cascaded approaches?
• RQ4 Can a single ST-SR model be effectively finetuned for standalone S-SR or T-SR with improved performance?
• RQ5 Does leveraging high-resolution spatial details improve temporal interpolation, and vice versa?

Key findings

• STARnet achieves a PSNR of 39.13 and SSIM of 0.991 on ST-SR for Vimeo90K, outperforming the best combination of RBPN and DAIN by 0.38dB.
• STAR-S (finetuned for S-SR) improves PSNR by 1.19dB over DBPN, 0.87dB over DBPN-MI, and 0.55dB over RBPN on Vimeo90K.
• STAR-T$_{\text{HR}}$ achieves state-of-the-art performance on T-SR at original resolution, with sharper textures and reduced blur on motion regions compared to ToFlow and DAIN.
• STAR-T$_{\text{LR}}$, finetuned on low-resolution inputs, outperforms both STAR-T$_{\text{HR}}$ and SOTA methods on subtle motion interpolation, achieving 39.30 PSNR and 0.991 SSIM on S-LR.
• Visual results show STARnet produces fewer artifacts and blur, especially in motion regions, compared to ToFlow and DAIN, as confirmed by red arrows in Figure 7.
• Joint learning in multiple resolutions enables better estimation of both large motions (on S-LR) and subtle motions (on S-HR), resolving limitations of single-resolution approaches.