[Paper Review] SwiftSRGAN -- Rethinking Super-Resolution for Efficient and Real-time Inference

Koushik Sivarama Krishnan, Karthik Sivarama Krishnan|arXiv (Cornell University)|Nov 28, 2021

Advanced Image Processing Techniques23 references15 citations

TL;DR

This paper proposes SwiftSRGAN, a lightweight, real-time super-resolution model that leverages depth-wise separable convolutions and a MobileNet-based perceptual loss to achieve state-of-the-art inference speed—74× faster than SRGANs—while maintaining competitive PSNR and SSIM scores, enabling real-time deployment on low-end devices.

ABSTRACT

In recent years, there have been several advancements in the task of image super-resolution using the state of the art Deep Learning-based architectures. Many super-resolution-based techniques previously published, require high-end and top-of-the-line Graphics Processing Unit (GPUs) to perform image super-resolution. With the increasing advancements in Deep Learning approaches, neural networks have become more and more compute hungry. We took a step back and, focused on creating a real-time efficient solution. We present an architecture that is faster and smaller in terms of its memory footprint. The proposed architecture uses Depth-wise Separable Convolutions to extract features and, it performs on-par with other super-resolution GANs (Generative Adversarial Networks) while maintaining real-time inference and a low memory footprint. A real-time super-resolution enables streaming high resolution media content even under poor bandwidth conditions. While maintaining an efficient trade-off between the accuracy and latency, we are able to produce a comparable performance model which is one-eighth (1/8) the size of super-resolution GANs and computes 74 times faster than super-resolution GANs.

Motivation & Objective

• To develop a real-time, efficient super-resolution model suitable for mobile and embedded devices with limited computational resources.
• To reduce model size and inference latency without sacrificing perceptual quality or reconstruction accuracy.
• To enable high-quality image upscaling in bandwidth-constrained environments such as streaming and edge computing.
• To demonstrate that efficient architectures can achieve performance comparable to larger, compute-heavy GAN-based super-resolution models.

Proposed method

• Employs depth-wise separable convolutions to drastically reduce parameters and FLOPs compared to standard convolutions.
• Uses a lightweight MobileNetV2 backbone for feature extraction instead of heavier VGG networks to reduce computational cost.
• Integrates a perceptual loss based on feature maps from MobileNetV2 to guide high-fidelity image generation.
• Combines adversarial loss with content loss to enhance realism and detail preservation in super-resolved outputs.
• Applies a multi-scale loss strategy using feature maps from multiple layers of the MobileNetV2 network.
• Trains using AdamW optimizer with mixed-precision training and a ReduceLROnPlateau scheduler for improved convergence.

Experimental results

Research questions

• RQ1 Can depth-wise separable convolutions significantly reduce model size and inference latency in super-resolution without degrading image quality?
• RQ2 How does replacing VGG with MobileNet in the perceptual loss affect training speed and performance?
• RQ3 Can a lightweight GAN-based architecture achieve competitive PSNR and SSIM scores while enabling real-time inference on low-end hardware?
• RQ4 What is the trade-off between model efficiency and super-resolution quality in real-world streaming and mobile applications?

Key findings

• SwiftSRGAN achieves an inference time of 5.605 ms per frame for 270p-to-1080p upscaling, which is 74× faster than SRGAN (812 ms) and 100× faster than ESRGAN.
• The model is 1/8 the size of standard super-resolution GANs, significantly reducing memory footprint and enabling deployment on low-end devices.
• On the Set5 benchmark, SwiftSRGAN achieves PSNR of 25.13 and SSIM of 0.794, demonstrating competitive performance against larger models.
• Visual results show that SwiftSRGAN preserves fine details, lighting, reflections, and color accuracy comparable to high-resolution ground truth.
• The use of MobileNet-based perceptual loss reduces training time and model size while maintaining perceptual quality.
• The model enables real-time video upscaling at 60 FPS on low-end hardware, making it viable for cloud gaming, surveillance, and mobile AR/VR applications.