[Paper Review] Learning to Downsample for Segmentation of Ultra-High Resolution Images

Jin Chen, Ryutaro Tanno+3 more|arXiv (Cornell University)|Sep 22, 2021

Advanced Image Processing Techniques31 references21 citations

TL;DR

This paper proposes a learnable downsampling module that adaptively samples high-resolution images based on segmentation difficulty, optimizing sampling density through end-to-end training with an edge-aware regularization loss. The method achieves superior accuracy-efficiency trade-offs on ultra-high-resolution medical, aerial, and street-view images compared to uniform and state-of-the-art edge-based downsampling techniques.

ABSTRACT

Many computer vision systems require low-cost segmentation algorithms based on deep learning, either because of the enormous size of input images or limited computational budget. Common solutions uniformly downsample the input images to meet memory constraints, assuming all pixels are equally informative. In this work, we demonstrate that this assumption can harm the segmentation performance because the segmentation difficulty varies spatially. We combat this problem by introducing a learnable downsampling module, which can be optimised together with the given segmentation model in an end-to-end fashion. We formulate the problem of training such downsampling module as optimisation of sampling density distributions over the input images given their low-resolution views. To defend against degenerate solutions (e.g. over-sampling trivial regions like the backgrounds), we propose a regularisation term that encourages the sampling locations to concentrate around the object boundaries. We find the downsampling module learns to sample more densely at difficult locations, thereby improving the segmentation performance. Our experiments on benchmarks of high-resolution street view, aerial and medical images demonstrate substantial improvements in terms of efficiency-and-accuracy trade-off compared to both uniform downsampling and two recent advanced downsampling techniques.

Motivation & Objective

• To address the limitation of uniform downsampling in ultra-high-resolution image segmentation, which assumes all pixels are equally informative and often under-samples salient regions.
• To overcome the suboptimal performance of existing edge-based downsampling methods that rely on fixed, hand-designed objectives not directly optimized for segmentation accuracy.
• To develop an end-to-end trainable downsampling module that dynamically allocates sampling budget to regions of high segmentation difficulty.
• To regularize the learning process to prevent degenerate solutions such as over-sampling trivial regions like backgrounds.
• To demonstrate consistent improvements in segmentation accuracy and efficiency across diverse domains, including medical, aerial, and urban street-view imagery.

Proposed method

• A learnable deformation module $ D_{\theta} $ is introduced that takes a low-resolution image as input and predicts a sampling density map $ \mathbf{d} $, which guides non-uniform sampling on the original high-resolution image.
• The deformed sampler $ G_d $ uses the predicted density map to generate a downsampled image $ \hat{\mathbf{X}} $ by selecting pixels adaptively, rather than uniformly.
• The sampling process is differentiable, enabling end-to-end optimization of both the downsampling module $ D_{\theta} $ and the segmentation network $ S_{\phi} $ jointly.
• An edge-loss regularization term is introduced to encourage sampling concentration around object boundaries, improving robustness and preventing over-sampling of background regions.
• The method jointly optimizes the segmentation loss and the edge-loss, allowing the model to learn data-dependent sampling strategies that maximize segmentation performance.

Experimental results

Research questions

• RQ1 Can a learnable downsampling module improve segmentation accuracy on ultra-high-resolution images compared to uniform downsampling, which treats all pixels equally?
• RQ2 Does end-to-end optimization of downsampling and segmentation lead to better efficiency-accuracy trade-offs than fixed, hand-designed sampling strategies?
• RQ3 How does the inclusion of an edge-loss regularization term affect the robustness and generalization of the downsampling module?
• RQ4 To what extent does the learned sampling strategy adapt to domain-specific characteristics, such as object boundary clarity or semantic cue distribution?
• RQ5 Can the method generalize across diverse high-resolution datasets, including medical histology, aerial, and urban street-view images?

Key findings

• The proposed method achieves up to 4 percentage points higher mIoU than the second-best baseline across all downsampling scales on the Cityscapes dataset, demonstrating significant accuracy gains.
• The method reduces computational cost by up to 47% compared to the baseline while maintaining or improving segmentation performance.
• On the DeepGlobe and PCa-Histo datasets, the method shows superior performance when labels are based on higher-level semantic knowledge rather than precise boundaries.
• The learned sampling strategy is data-dependent: it samples densely near boundaries in Cityscapes but may sample away from boundaries in DeepGlobe and PCa-Histo, indicating adaptability to semantic complexity.
• The edge-loss regularization effectively prevents over-sampling of trivial regions and improves generalization, especially in cases where object boundaries are not the primary cues for segmentation.