[Paper Review] LAPAR: Linearly-Assembled Pixel-Adaptive Regression Network for Single Image Super-Resolution and Beyond

Wenbo Li, Kun Zhou+4 more|arXiv (Cornell University)|May 21, 2021

Advanced Image Processing TechniquesComputer Science50 references107 citations

TL;DR

LAPAR introduces a lightweight, pixel-wise adaptive regression approach that assembles filters from a predefined dictionary to perform SISR and can extend to denoising and JPEG deblocking.

ABSTRACT

Single image super-resolution (SISR) deals with a fundamental problem of upsampling a low-resolution (LR) image to its high-resolution (HR) version. Last few years have witnessed impressive progress propelled by deep learning methods. However, one critical challenge faced by existing methods is to strike a sweet spot of deep model complexity and resulting SISR quality. This paper addresses this pain point by proposing a linearly-assembled pixel-adaptive regression network (LAPAR), which casts the direct LR to HR mapping learning into a linear coefficient regression task over a dictionary of multiple predefined filter bases. Such a parametric representation renders our model highly lightweight and easy to optimize while achieving state-of-the-art results on SISR benchmarks. Moreover, based on the same idea, LAPAR is extended to tackle other restoration tasks, e.g., image denoising and JPEG image deblocking, and again, yields strong performance. The code is available at https://github.com/dvlab-research/Simple-SR.

Motivation & Objective

• Motivate high-quality SISR with low model complexity for practical efficiency.
• Propose a linearly-assembled, pixel-adaptive regression framework that regresses coefficients over a dictionary of filter bases.
• Demonstrate that a predefined dictionary of Gaussian and DoG filters can yield competitive or superior SR results with few parameters.
• Show that the same framework can be extended to denoising and JPEG deblocking with strong performance.

Proposed method

• Represent the LR-to-HR mapping as a linear coefficient regression over a dictionary of predefined base filters.
• Learn spatially variant assembly coefficients for each pixel via a lightweight network (LAPARnet).
• Assemble pixel-adaptive filters as F_i = Phi_i D, where D is a fixed dictionary of L filters (k x k).
• Predict high-resolution pixels by y_hat_i = Phi_i D B_i^T, with B_i the local patch around the i-th pixel from a bicubic upsampled image.
• Use a Charbonnier loss for optimization to stabilize training.
• Dictionary design uses 72 Gaussian and DoG filters to capture structure and edges; filters are normalized to sum to 1.

Experimental results

Research questions

• RQ1 Can a linearly-assembled, pixel-adaptive regression framework achieve state-of-the-art performance with substantially fewer parameters than traditional SR models?
• RQ2 Does a predefined dictionary of simple filters (Gaussian and DoG) suffice to capture diverse image structures across scales?
• RQ3 Can the LAPAR framework be extended to other restoration tasks such as denoising and JPEG deblocking without losing performance?

Key findings

• LAPAR-A achieves state-of-the-art results among lightweight SR methods on multiple benchmarks with fewer parameters and Multiply-Adds.
• A 72-filter Gaussian/DoG dictionary outperforms smaller dictionaries and random or RAISR-based filters in PSNR on Set5/B100 scales.
• LAPAR maintains competitive SR performance across x2, x3, and x4 scales while offering significantly lower model size (e.g., LAPAR-A ~0.66M params for x4).
• LAPAR variants provide fast inference (e.g., 22.2–37.3 ms on an RTX 2080Ti for 1280x720 x4).
• The approach extends effectively to image denoising and JPEG deblocking, preserving color and structure and outperforming some baselines in PSNR/SSIM on standard sets.