[Paper Review] NTIRE 2021 Challenge on Quality Enhancement of Compressed Video: Methods and Results

Ren Yang, Radu Timofte+70 more|arXiv (Cornell University)|Apr 21, 2021

Advanced Image Processing Techniques66 references4 citations

TL;DR

This paper presents the NTIRE 2021 challenge on compressed video quality enhancement, introducing the Large-scale Diverse Video (LDV) dataset and evaluating 482 submissions across three tracks: fidelity enhancement at fixed QP (Tracks 1 and 3) and perceptual quality enhancement at fixed bit-rate (Track 2). State-of-the-art results were achieved using spatiotemporal networks with gated fusion, deformable convolutions, and multi-scale attention mechanisms, with top teams improving PSNR by up to 38.31 dB and MS-SSIM above 0.956.

ABSTRACT

This paper reviews the first NTIRE challenge on quality enhancement of compressed video, with a focus on the proposed methods and results. In this challenge, the new Large-scale Diverse Video (LDV) dataset is employed. The challenge has three tracks. Tracks 1 and 2 aim at enhancing the videos compressed by HEVC at a fixed QP, while Track 3 is designed for enhancing the videos compressed by x265 at a fixed bit-rate. Besides, the quality enhancement of Tracks 1 and 3 targets at improving the fidelity (PSNR), and Track 2 targets at enhancing the perceptual quality. The three tracks totally attract 482 registrations. In the test phase, 12 teams, 8 teams and 11 teams submitted the final results of Tracks 1, 2 and 3, respectively. The proposed methods and solutions gauge the state-of-the-art of video quality enhancement. The homepage of the challenge: https://github.com/RenYang-home/NTIRE21_VEnh

Motivation & Objective

• To establish a benchmark for video quality enhancement under diverse compression conditions using a large-scale, diverse video dataset.
• To evaluate methods for improving both fidelity (PSNR) and perceptual quality in compressed videos.
• To compare state-of-the-art approaches across fixed QP and fixed bit-rate compression settings.
• To promote the use of the newly introduced Large-scale Diverse Video (LDV) dataset for video restoration research.
• To advance the field by enabling standardized evaluation on a unified, diverse, and challenging test set.

Proposed method

• Proposed a three-track challenge framework: Tracks 1 and 3 focus on fidelity enhancement at fixed QP and fixed bit-rate, respectively, while Track 2 targets perceptual quality improvement.
• Employed the Large-scale Diverse Video (LDV) dataset, comprising 240 videos across 10 scene categories with variations in motion, frame rate, lighting, and camera stability.
• Used spatiotemporal convolutional networks with gated fusion mechanisms to exploit both spatial and temporal correlations in video sequences.
• Integrated deformable convolutions and multi-scale feature extraction to better handle motion and compression artifacts.
• Applied progressive alignment and data augmentation techniques to improve training stability and generalization.
• Utilized 3D-convolutional and U-Net-based architectures for multi-frame residual learning and feature refinement in video enhancement.

Experimental results

Research questions

• RQ1 How effective are spatiotemporal deep learning models in restoring compressed video with high fidelity and perceptual quality?
• RQ2 To what extent can multi-scale and attention-based architectures improve artifact suppression in compressed videos?
• RQ3 How does the diversity of content, motion, and frame rate in the LDV dataset affect model generalization and performance?
• RQ4 What is the relative performance of different network designs (e.g., gated fusion, deformable convolutions, 3D-convolutions) in video quality enhancement?
• RQ5 Can unified models generalize across fixed QP and fixed bit-rate compression scenarios?

Key findings

• The top-performing method in Track 1 (fidelity at fixed QP) achieved a PSNR of 38.31 dB, significantly outperforming baseline methods.
• In Track 2 (perceptual quality), the best method achieved an MS-SSIM of 0.9562, indicating high perceptual fidelity to the original video.
• Track 3 (fixed bit-rate) saw strong performance with PSNR values exceeding 35 dB for several top teams, demonstrating effectiveness in bit-rate constrained restoration.
• Methods using gated fusion and deformable temporal modeling consistently outperformed standard video super-resolution baselines.
• The LDV dataset proved effective in evaluating model generalization, with top models achieving high performance across diverse video categories.
• The challenge revealed that perceptual quality enhancement (Track 2) required different architectural choices than fidelity-based enhancement, with attention mechanisms playing a key role.