[Paper Review] SplitSR: An End-to-End Approach to Super-Resolution on Mobile Devices

Xin Liu, Yuang Li+5 more|arXiv (Cornell University)|Jan 20, 2021

Advanced Image Processing Techniques44 references4 citations

TL;DR

SplitSR proposes a hybrid deep learning architecture with a novel lightweight residual block (SplitSRBlock) that enables state-of-the-art on-device super-resolution on mobile CPUs. By leveraging channel-splitting and a hybrid design of standard and lightweight convolutions, it achieves up to 5× faster inference and higher accuracy than prior methods, enabling the first real-time, on-device deep learning super-resolution app (ZoomSR).

ABSTRACT

Super-resolution (SR) is a coveted image processing technique for mobile apps ranging from the basic camera apps to mobile health. Existing SR algorithms rely on deep learning models with significant memory requirements, so they have yet to be deployed on mobile devices and instead operate in the cloud to achieve feasible inference time. This shortcoming prevents existing SR methods from being used in applications that require near real-time latency. In this work, we demonstrate state-of-the-art latency and accuracy for on-device super-resolution using a novel hybrid architecture called SplitSR and a novel lightweight residual block called SplitSRBlock. The SplitSRBlock supports channel-splitting, allowing the residual blocks to retain spatial information while reducing the computation in the channel dimension. SplitSR has a hybrid design consisting of standard convolutional blocks and lightweight residual blocks, allowing people to tune SplitSR for their computational budget. We evaluate our system on a low-end ARM CPU, demonstrating both higher accuracy and up to 5 times faster inference than previous approaches. We then deploy our model onto a smartphone in an app called ZoomSR to demonstrate the first-ever instance of on-device, deep learning-based SR. We conducted a user study with 15 participants to have them assess the perceived quality of images that were post-processed by SplitSR. Relative to bilinear interpolation -- the existing standard for on-device SR -- participants showed a statistically significant preference when looking at both images (Z=-9.270, p<0.01) and text (Z=-6.486, p<0.01).

Motivation & Objective

• Address the challenge of deploying high-accuracy deep learning super-resolution (SR) models on resource-constrained mobile devices due to high memory and computational demands.
• Overcome the latency and privacy limitations of cloud-based SR by enabling end-to-end on-device inference on low-end mobile hardware.
• Develop a mobile-optimized SR system that maintains high image quality while operating within strict computational and memory budgets.
• Demonstrate the feasibility of real-time, on-device SR through a production-ready smartphone application (ZoomSR) for practical use cases.
• Enable privacy-preserving, low-latency SR for mobile health applications in low-resource settings where cloud connectivity is unreliable or unavailable.

Proposed method

• Propose a hybrid CNN architecture, SplitSR, combining standard convolutional blocks with lightweight SplitSRBlocks to balance accuracy and efficiency.
• Introduce the SplitSRBlock, a novel residual block that performs channel-splitting within 2D convolutions to preserve spatial information while reducing channel-wise computation.
• Use a modern deep learning compiler (TVM) to perform low-level operator optimizations, including kernel fusion and memory access optimization, for efficient inference on ARM CPUs.
• Design a model deployment pipeline that allows tuning the hybrid architecture for different computational budgets by adjusting hyperparameters such as channel split ratio and block placement.
• Implement and evaluate the system on a low-end ARM CPU (Google Pixel 4) and deploy it in a real-world app called ZoomSR for end-to-end on-device SR.
• Optimize inference latency through model scheduling and gesture prediction, anticipating zoom levels during pinch-to-zoom interactions to select the most appropriate model.

Experimental results

Research questions

• RQ1 Can a hybrid CNN architecture combining standard and lightweight residual blocks achieve high super-resolution accuracy while remaining efficient enough for on-device deployment on mobile CPUs?
• RQ2 How does channel-splitting in the SplitSRBlock improve the trade-off between computational efficiency and feature preservation in super-resolution?
• RQ3 To what extent can on-device super-resolution reduce latency and eliminate reliance on cloud processing compared to existing cloud-based SR solutions?
• RQ4 Can real-time, on-device deep learning super-resolution be successfully implemented in a production-grade mobile application with measurable user preference over traditional methods?
• RQ5 How can user interaction patterns (e.g., pinch-to-zoom gestures) be leveraged to optimize model selection and further reduce inference latency?

Key findings

• SplitSR achieves up to 5× faster inference than previous state-of-the-art methods on a low-end ARM CPU while maintaining higher image reconstruction accuracy.
• The user study showed statistically significant preference for SplitSR over bilinear interpolation in both image quality (Z = -9.270, p < 0.01) and text readability (Z = -6.486, p < 0.01).
• The ZoomSR app demonstrates the first-ever deployment of end-to-end, on-device deep learning-based super-resolution on a smartphone, enabling real-time processing without cloud dependency.
• The SplitSRBlock’s channel-splitting mechanism effectively reduces computation in the channel dimension while preserving spatial features, contributing to both speed and accuracy gains.
• The hybrid architecture allows flexible tuning for different computational budgets by adjusting the proportion of standard and lightweight blocks.
• Model performance is further enhanced through low-level optimizations via the TVM compiler, which enables efficient kernel scheduling and memory access patterns on mobile CPUs.