[Paper Review] SamplingAug: On the Importance of Patch Sampling Augmentation for Single Image Super-Resolution

Shizun Wang, Ming Lu+5 more|arXiv (Cornell University)|Nov 30, 2021

Advanced Image Processing Techniques22 references4 citations

TL;DR

This paper proposes SamplingAug, a data augmentation method for single image super-resolution (SISR) that improves model performance by sampling the most informative low-resolution (LR)-high-resolution (HR) patch pairs. Instead of uniform sampling, it uses a heuristic metric based on PSNR between linear super-resolution and HR patches, accelerated via integral images, achieving ~100x speedup. The method consistently boosts PSNR across EDSR, RCAN, RDN, SRCNN, and ESPCN models at ×2, ×3, and ×4 scaling factors.

ABSTRACT

With the development of Deep Neural Networks (DNNs), plenty of methods based on DNNs have been proposed for Single Image Super-Resolution (SISR). However, existing methods mostly train the DNNs on uniformly sampled LR-HR patch pairs, which makes them fail to fully exploit informative patches within the image. In this paper, we present a simple yet effective data augmentation method. We first devise a heuristic metric to evaluate the informative importance of each patch pair. In order to reduce the computational cost for all patch pairs, we further propose to optimize the calculation of our metric by integral image, achieving about two orders of magnitude speedup. The training patch pairs are sampled according to their informative importance with our method. Extensive experiments show our sampling augmentation can consistently improve the convergence and boost the performance of various SISR architectures, including EDSR, RCAN, RDN, SRCNN and ESPCN across different scaling factors (x2, x3, x4). Code is available at https://github.com/littlepure2333/SamplingAug

Motivation & Objective

• To address the limitation of uniform patch sampling in SISR, which fails to prioritize informative patches.
• To develop a computationally efficient method for identifying and sampling the most informative LR-HR patch pairs.
• To improve convergence and performance of SISR models without modifying network architecture.
• To investigate whether non-overlapping sampling strategies offer advantages over greedy, overlapping sampling.

Proposed method

• Proposes a heuristic metric measuring informative importance as the PSNR between linearly upsampled LR patches and their corresponding HR patches.
• Accelerates metric computation using integral images, reducing computational cost by approximately two orders of magnitude.
• Employs three sampling strategies: greedy sampling of top-p% patches, Non-Maximum Suppression (NMS), and Throwing-Dart (TD) sampling for non-overlapping patches.
• Applies the sampling strategy to select training patches, replacing uniform sampling in standard SISR training pipelines.
• Uses the same training protocol across multiple SISR architectures (EDSR, RCAN, RDN, SRCNN, ESPCN) to evaluate generalization.
• Validates the method on standard benchmarks (DIV2K, Set5, Set14) across multiple scaling factors (×2, ×3, ×4).

Experimental results

Research questions

• RQ1 Does prioritizing informative patch pairs through a data-driven metric improve SISR performance compared to uniform sampling?
• RQ2 Can a fast, lightweight metric for patch importance be computed efficiently at scale using integral images?
• RQ3 Do non-overlapping sampling strategies (NMS, TD) provide performance gains over greedy, overlapping sampling of top patches?
• RQ4 How does the proposed method compare to existing data augmentation techniques like CutBlur and PatchNet in SISR?
• RQ5 Is the performance gain consistent across different SISR architectures and scaling factors?

Key findings

• The proposed SamplingAug method achieves consistent PSNR improvements across multiple SISR architectures, with gains of +0.12 dB on EDSR at ×2 scaling on the DIV2K dataset.
• Greedy sampling of the top-p% most informative patches outperforms NMS and TD sampling, indicating that overlapping patches with high informativeness are more beneficial.
• The method achieves ~100x speedup in metric computation using integral images, enabling efficient evaluation of all patch pairs.
• SamplingAug outperforms CutBlur, especially without pretraining, where CutBlur underperforms the baseline, while SamplingAug consistently improves performance.
• The method surpasses PatchNet and OHEM in stability and performance, with OHEM showing inconsistent convergence and lower gains.
• The ablation study confirms that the proposed metric is more effective than edge-based metrics (Sobel, Canny, Laplacian), achieving the highest PSNR gain of +0.12 dB.