[Paper Review] Residual Feature Distillation Network for Lightweight Image Super-Resolution

Jie Liu, Jie Tang+1 more|arXiv (Cornell University)|Sep 24, 2020

Advanced Image Processing TechniquesComputer Science30 references32 citations

TL;DR

The paper introduces RFDN, a lightweight SISR model built with feature distillation connections and shallow residual blocks, achieving state-of-the-art trade-offs between performance and model size, and an enhanced version winning AIM 2020.

ABSTRACT

Recent advances in single image super-resolution (SISR) explored the power of convolutional neural network (CNN) to achieve a better performance. Despite the great success of CNN-based methods, it is not easy to apply these methods to edge devices due to the requirement of heavy computation. To solve this problem, various fast and lightweight CNN models have been proposed. The information distillation network is one of the state-of-the-art methods, which adopts the channel splitting operation to extract distilled features. However, it is not clear enough how this operation helps in the design of efficient SISR models. In this paper, we propose the feature distillation connection (FDC) that is functionally equivalent to the channel splitting operation while being more lightweight and flexible. Thanks to FDC, we can rethink the information multi-distillation network (IMDN) and propose a lightweight and accurate SISR model called residual feature distillation network (RFDN). RFDN uses multiple feature distillation connections to learn more discriminative feature representations. We also propose a shallow residual block (SRB) as the main building block of RFDN so that the network can benefit most from residual learning while still being lightweight enough. Extensive experimental results show that the proposed RFDN achieve a better trade-off against the state-of-the-art methods in terms of performance and model complexity. Moreover, we propose an enhanced RFDN (E-RFDN) and won the first place in the AIM 2020 efficient super-resolution challenge. Code will be available at https://github.com/njulj/RFDN.

Motivation & Objective

• Motivate fast and accurate single image super-resolution for edge devices.
• Revisit information distillation to design lightweight building blocks.
• Propose feature distillation connections (FDC) as a lightweight alternative to channel splitting.
• Introduce shallow residual blocks (SRB) to enhance residual learning without extra parameters.
• Demonstrate RFDN and enhanced RFDN (E-RFDN) achieve strong SR performance with few parameters.

Proposed method

• Propose feature distillation connections (FDC) equivalent to channel splitting but more lightweight and flexible.
• Rethink IMDB by decoupling the distillation and refinement operations into DL (distilled features) and RL (refinement) layers.
• Replace 3x3 channel-reducing convolutions with 1x1 convolutions for the distillation paths while keeping a 3x3 kernel for spatial refinement (RFDB).
• Introduce a shallow residual block (SRB) with a single 3x3 convolution and an identity skip to leverage residual learning without extra parameters.
• Stack multiple RFDBs with a final fusion (1x1 conv followed by 3x3 conv) and a reconstruction block using sub-pixel upsampling.
• Train with L1 loss on DIV2K, evaluating on Set5, Set14, BSD100, Urban100, Manga109 across ×2, ×3, ×4 scales.

Experimental results

Research questions

• RQ1 Can a lightweight distillation-based design outperform existing lightweight SR models with fewer parameters?
• RQ2 Does replacing channel splitting with 1x1-based distillation paths (FDC) improve flexibility and efficiency?
• RQ3 Do shallow residual blocks (SRB) enhance SR performance without adding parameters?
• RQ4 How does the proposed RFDB-based architecture compare to IMDN and other state-of-the-art lightweight SR models in PSNR/SSIM and complexity?
• RQ5 Can an enhanced version (E-RFDN) win efficiency-focused SR challenges?

Key findings

• RFDN achieves a favorable PSNR/SSIM balance with significantly fewer parameters than many lightweight SR models.
• In ablations, SRB provides measurable gains without extra parameters, and FDC yields substantial improvement over baselines.
• RFDB with FDC and SRB outperforms the baseline IMDB-based designs in both performance and parameter efficiency.
• RFDN/M variants reach competitive or superior results to IMDN while using fewer parameters (about 534K–550K).
• Enhanced RFDN (E-RFDN) won first place in the AIM 2020 efficient super-resolution challenge, demonstrating strong efficiency and performance gains.