[Paper Review] Achieving on-Mobile Real-Time Super-Resolution with Neural Architecture and Pruning Search

Zheng Zhan, Yifan Gong+10 more|arXiv (Cornell University)|Aug 18, 2021

Advanced Image Processing Techniques63 references4 citations

TL;DR

This paper proposes a joint neural architecture and pruning search framework to automatically generate lightweight, sparse super-resolution models that achieve real-time inference (under 50ms per frame) on mobile devices like the Samsung Galaxy S20, while maintaining competitive PSNR and SSIM scores. By leveraging a supernet with weight sharing and decoupling the search into three stages—supernet construction, compiler-aware architecture and pruning search, and pruning ratio optimization using Bayesian optimization—the method enables efficient, end-to-end search for optimal SR blocks and layer-wise pruning configurations, resulting in the first real-time 720p super-resolution on mobile platforms.

ABSTRACT

Though recent years have witnessed remarkable progress in single image super-resolution (SISR) tasks with the prosperous development of deep neural networks (DNNs), the deep learning methods are confronted with the computation and memory consumption issues in practice, especially for resource-limited platforms such as mobile devices. To overcome the challenge and facilitate the real-time deployment of SISR tasks on mobile, we combine neural architecture search with pruning search and propose an automatic search framework that derives sparse super-resolution (SR) models with high image quality while satisfying the real-time inference requirement. To decrease the search cost, we leverage the weight sharing strategy by introducing a supernet and decouple the search problem into three stages, including supernet construction, compiler-aware architecture and pruning search, and compiler-aware pruning ratio search. With the proposed framework, we are the first to achieve real-time SR inference (with only tens of milliseconds per frame) for implementing 720p resolution with competitive image quality (in terms of PSNR and SSIM) on mobile platforms (Samsung Galaxy S20).

Motivation & Objective

• To address the challenge of deploying real-time, high-quality single image super-resolution (SISR) on resource-limited mobile devices.
• To overcome the high computation and memory costs of deep learning-based SISR models in mobile environments.
• To jointly optimize neural architecture and pruning configurations for each layer to balance inference speed and image quality.
• To reduce the search cost and training overhead in model search by using a shared supernet and decoupled search stages.
• To enable real-time performance (≥20 FPS) for 720p video super-resolution on mobile platforms with competitive perceptual and metric quality.

Proposed method

• The method introduces a supernet with weight sharing to enable efficient search across diverse SR block and pruning configurations without retraining for each combination.
• It decouples the search into three stages: supernet construction, compiler-aware architecture and pruning search, and compiler-aware pruning ratio search.
• Bayesian optimization (BO) is used to accelerate the search for optimal SR blocks and pruning schemes, reducing the number of required evaluations.
• The framework performs compiler-aware search to ensure that the final pruned models are compatible with hardware acceleration and achieve low-latency inference.
• The search process is guided by target latency constraints (e.g., 50ms for ×2 and ×3, 40ms for ×4) to ensure real-time performance on mobile devices.
• The final models are evaluated on mobile hardware (Samsung Galaxy S20) using optimized inference frameworks like MNN and PyTorch Mobile.

Figure 1: (a) Unstructured pruning; (b) coarse-grained structured pruning (channel); (c) fine-grained structured pruning (block-based); and (d) fine-grained structured sparsity (pattern-based).

Experimental results

Research questions

• RQ1 Can a joint search framework for neural architecture and pruning configuration achieve real-time super-resolution on mobile devices?
• RQ2 How can supernet-based weight sharing and decoupled search reduce the training and search cost in SISR model search?
• RQ3 What is the impact of layer-wise, adaptive pruning schemes compared to fixed pruning across the entire model?
• RQ4 Can Bayesian optimization effectively accelerate the search for optimal SR blocks and pruning ratios under strict latency constraints?
• RQ5 What is the achievable trade-off between inference speed, model size, and image quality (PSNR/SSIM) for real-time mobile SISR?

Key findings

• The proposed method achieves real-time super-resolution on the Samsung Galaxy S20, with inference latency below 50ms per frame for ×2 and ×3 upscaling, and below 40ms for ×4 upscaling.
• For ×2 upscaling with a target latency of 50ms, the model achieves 31.93 PSNR and 0.8906 SSIM on Set5, outperforming CARN-M and FALSR-C in both accuracy and MACs.
• The real-time model (40ms target) achieves 30.74 PSNR and 0.8671 SSIM on Set5, demonstrating competitive image quality despite extreme model compression (12 MACs, 0.7M parameters).
• On the ×4 upscaling task, the model with 120ms latency achieves higher PSNR and SSIM than SRCNN, FSRCNN, and FEQE-P, while using fewer MACs.
• With compiler optimizations, the method achieves over 20 FPS for ×2 and ×3, and over 25 FPS for ×4, outperforming MNN and PyTorch Mobile.
• Abllation studies confirm that both architecture search and pruning search are essential: architecture search improves quality, while pruning search enables speed, and their combination yields optimal results.

Figure 2: Inference acceleration rate vs. pruning ratio of different pruning schemes. Results are measured on a Samsung Galaxy S20 smartphone, and a typical 3 $\times$ 3 CONV layer in WDSR block with 24/48 input/output channels and 320 $\times$ 180 feature size is used.