[Paper Review] Perceptual Video Super Resolution with Enhanced Temporal Consistency

Eduardo Pérez-Pellitero, Mehdi S. M. Sajjadi+2 more|arXiv (Cornell University)|Jul 20, 2018

Advanced Image Processing Techniques37 references4 citations

TL;DR

This paper proposes a recurrent generative adversarial network with a video discriminator and novel temporal consistency losses for perceptual video super-resolution, achieving state-of-the-art results in perceptual quality and reduced flickering by leveraging recurrent feature propagation and multi-frame warping.

ABSTRACT

With the advent of perceptual loss functions, new possibilities in super-resolution have emerged, and we currently have models that successfully generate near-photorealistic high-resolution images from their low-resolution observations. Up to now, however, such approaches have been exclusively limited to single image super-resolution. The application of perceptual loss functions on video processing still entails several challenges, mostly related to the lack of temporal consistency of the generated images, i.e., flickering artifacts. In this work, we present a novel adversarial recurrent network for video upscaling that is able to produce realistic textures in a temporally consistent way. The proposed architecture naturally leverages information from previous frames due to its recurrent architecture, i.e. the input to the generator is composed of the low-resolution image and, additionally, the warped output of the network at the previous step. Together with a video discriminator, we also propose additional loss functions to further reinforce temporal consistency in the generated sequences. The experimental validation of our algorithm shows the effectiveness of our approach which obtains images with high perceptual quality and improved temporal consistency.

Motivation & Objective

• Address the lack of temporal consistency in perceptual video super-resolution, where perceptual losses cause flickering artifacts.
• Transfer the success of perceptual loss functions from single-image to video super-resolution, which remains underexplored.
• Improve video quality by ensuring consistent high-frequency details across frames while maintaining photorealistic textures.
• Overcome limitations of sliding-window approaches that ignore prior super-resolved frames, leading to temporal inconsistency.
• Develop a training framework that explicitly enforces temporal coherency through dedicated loss functions and recurrent architecture.

Proposed method

• Employ a recurrent generator that takes both the current low-resolution frame and the warped previous high-resolution output as input, enabling memory of prior results.
• Introduce a video discriminator that evaluates spatiotemporal coherence across consecutive frames, promoting realistic motion and consistency.
• Apply multi-image warping to align neighboring frames with the current frame before processing, improving feature alignment and reducing motion artifacts.
• Design two novel loss terms: a static loss ($\mathcal{L}_{T_d}$) to minimize flickering in motionless regions and a variance distance loss ($\mathcal{L}_{T_s}$) to reduce pixel statistic variation over time.
• Optimize the model using a composite loss combining perceptual loss ($\mathcal{L}_c$), adversarial loss ($\mathcal{L}_A$), and the two temporal consistency losses.
• Use a pre-trained feature extractor (e.g., VGG or AlexNet) for perceptual loss computation, with an additional linear calibration layer to improve alignment.

Experimental results

Research questions

• RQ1 Can perceptual loss functions used in single-image super-resolution be successfully extended to video super-resolution without introducing temporal flickering?
• RQ2 How does incorporating recurrent memory of previous frames improve temporal consistency in video super-resolution?
• RQ3 To what extent do dedicated temporal consistency losses ($\mathcal{L}_{T_d}$ and $\mathcal{L}_{T_s}$) reduce flickering and improve perceptual quality in generated video sequences?
• RQ4 How does the proposed method compare to existing GAN-based and MSE-optimized VSR models in terms of perceptual quality and temporal stability?
• RQ5 Does the combination of a video discriminator and recurrent architecture outperform non-recurrent, sliding-window approaches in maintaining long-term temporal coherence?

Key findings

• The proposed method achieves state-of-the-art LPIPS and NIQE scores on the vid4 and seq12 datasets, indicating superior perceptual quality and reduced distortion.
• Among perceptual methods, the model trained with $\mathcal{L}_c$ achieves the best LPIPS scores on both vid4 and seq12, outperforming SRGAN and Enhancenet.
• Even without direct optimization on temporal losses, the recurrent architecture with video discriminator significantly improves temporal consistency, outperforming non-recurrent models like SRGAN.
• The ablation study confirms that $\mathcal{L}_{T_d}$ has a stronger impact on temporal consistency than $\mathcal{L}_{T_s}$, and combining both with $\mathcal{L}_A$ yields the best overall performance.
• The model with $\mathcal{L}_c$ loss achieves the highest temporal consistency across all metrics, including warping error PSNR and tLPIPS, demonstrating effectiveness of the full loss formulation.
• Qualitative analysis shows that while SRGAN and Enhancenet generate fine textures, they often introduce over-sharpening and implausible patterns, which are mitigated by the proposed method.