[Paper Review] Checkerboard artifact free sub-pixel convolution: A note on sub-pixel convolution, resize convolution and convolution resize

Andrew P. Aitken, Christian Ledig+4 more|arXiv (Cornell University)|Jul 10, 2017

Image Processing Techniques and Applications16 references141 citations

TL;DR

The note proposes a convolution NN resize initialization for sub-pixel convolution that is free of checkerboard artifacts immediately after initialization and offers greater modeling power than resize convolution at the same computational cost.

ABSTRACT

The most prominent problem associated with the deconvolution layer is the presence of checkerboard artifacts in output images and dense labels. To combat this problem, smoothness constraints, post processing and different architecture designs have been proposed. Odena et al. highlight three sources of checkerboard artifacts: deconvolution overlap, random initialization and loss functions. In this note, we proposed an initialization method for sub-pixel convolution known as convolution NN resize. Compared to sub-pixel convolution initialized with schemes designed for standard convolution kernels, it is free from checkerboard artifacts immediately after initialization. Compared to resize convolution, at the same computational complexity, it has more modelling power and converges to solutions with smaller test errors.

Motivation & Objective

• Motivate the problem of checkerboard artifacts in deconvolution-based upsampling.
• Introduce an initialization method called convolution NN resize for sub-pixel convolution.
• Compare the proposed method to standard sub-pixel convolution initializations and to resize convolution in terms of artifacts and modeling power.

Proposed method

• Define convolution NN resize as an initialization for sub-pixel convolution.
• Argue that the initialization eliminates checkerboard artifacts immediately after initialization.
• Compare computational complexity and modeling power against resize convolution.
• Assess convergence behavior and test errors relative to alternative schemes.

Experimental results

Research questions

• RQ1 Can convolution NN resize initialization remove checkerboard artifacts without additional post-processing?
• RQ2 How does convolution NN resize compare to standard sub-pixel convolution initializations in terms of test error at equal computational cost?
• RQ3 What are the trade-offs between modeling power and artifacts for the proposed method versus resize convolution?

Key findings

• Convolution NN resize initialization yields artifact-free outputs immediately after initialization.
• At the same computational complexity, convolution NN resize has greater modeling power than resize convolution.
• The proposed method converges to solutions with smaller test errors compared to resize convolution under similar cost.