[Paper Review] Is the deconvolution layer the same as a convolutional layer?

Wenzhe Shi, José Caballero+5 more|arXiv (Cornell University)|Sep 22, 2016

Advanced Image Processing Techniques4 references106 citations

TL;DR

The note clarifies relationships among deconvolution (transposed and sub-pixel) layers and introduces efficient LR-space convolutions, arguing LR-space convolutions can outperform high-resolution upsampling under a fixed budget.

ABSTRACT

In this note, we want to focus on aspects related to two questions most people asked us at CVPR about the network we presented. Firstly, What is the relationship between our proposed layer and the deconvolution layer? And secondly, why are convolutions in low-resolution (LR) space a better choice? These are key questions we tried to answer in the paper, but we were not able to go into as much depth and clarity as we would have liked in the space allowance. To better answer these questions in this note, we first discuss the relationships between the deconvolution layer in the forms of the transposed convolution layer, the sub-pixel convolutional layer and our efficient sub-pixel convolutional layer. We will refer to our efficient sub-pixel convolutional layer as a convolutional layer in LR space to distinguish it from the common sub-pixel convolutional layer. We will then show that for a fixed computational budget and complexity, a network with convolutions exclusively in LR space has more representation power at the same speed than a network that first upsamples the input in high resolution space.

Motivation & Objective

• Explain how deconvolution layers relate to standard convolutions.
• Distinguish between transposed, sub-pixel, and efficient LR-space convolutions.
• Show why convolutions in low-resolution space can be more effective under a fixed computational budget.

Proposed method

• Discuss relationships between deconvolution forms (transposed convolution, sub-pixel convolution, efficient sub-pixel convolution).
• Introduce their efficient LR-space convolution as a distinct type from the common sub-pixel layer.
• Provide theoretical and conceptual comparison under fixed computational budget and complexity.

Experimental results

Research questions

• RQ1 What is the relationship between the proposed layer and the deconvolution layer?
• RQ2 Why are convolutions in low-resolution space a better choice under a fixed budget?
• RQ3 How do transposed, sub-pixel, and efficient sub-pixel convolutions differ in representation power and speed?

Key findings

• The note clarifies relationships among deconvolution variants and the authors’ efficient LR-space convolution.
• It distinguishes the efficient sub-pixel convolutional layer from the standard sub-pixel layer.
• For a fixed computational budget, networks with LR-space convolutions have greater representation power at the same speed than networks that first upsample to high resolution.
• The discussion provides insights to better answer practical CVPR questions about layer choices.