[Paper Review] A guide to convolution arithmetic for deep learning

Vincent Dumoulin, Francesco Visin|arXiv (Cornell University)|Mar 23, 2016

Generative Adversarial Networks and Image Synthesis5 references1,223 citations

TL;DR

A detailed guide explaining the relationships between input size, kernel size, padding, strides, and output size for convolutional, pooling, and transposed convolutional layers, including dilated convolutions and practical intuition for their arithmetic.

ABSTRACT

We introduce a guide to help deep learning practitioners understand and manipulate convolutional neural network architectures. The guide clarifies the relationship between various properties (input shape, kernel shape, zero padding, strides and output shape) of convolutional, pooling and transposed convolutional layers, as well as the relationship between convolutional and transposed convolutional layers. Relationships are derived for various cases, and are illustrated in order to make them intuitive.

Motivation & Objective

• Explain how input shape, kernel shape, padding, and strides determine output shapes in convolutional, pooling, and transposed convolutional layers.
• Provide intuitive and framework-agnostic relationships for 1-D/2-D/N-D convolutions and pooling.
• Clarify the connections between convolution and transposed convolution operations and their practical implications.
• Offer guidance on dilated (atrous) convolutions and their impact on receptive field and output size.

Proposed method

• Present a set of formal relationships linking i, k, s, p to output o for convolutions (Relationships 1–6).
• Explain half (same) padding and full padding with corresponding output-size formulas.
• Describe pooling arithmetic using the same framework as convolutions (Relationship 7).
• Derive transposed convolution arithmetic by viewing transposed conv as the gradient/transpose of a forward convolution (Sections 4.1–4.6).
• Introduce dilated convolutions and derive their output-size relationship (Relationship 15).
• Discuss the matrix-operator view of convolution to illuminate forward and backward passes (Section 4.1).

Experimental results

Research questions

• RQ1 What are the precise relationships between input size, kernel size, padding, and stride that determine convolution output size across axes?
• RQ2 How do padding modes (half/same and full) affect output shapes in convolution and transposed convolution?
• RQ3 What is the exact relationship between convolution and its transposed variant in terms of output shape and connectivity?
• RQ4 How do dilated convolutions modify the effective kernel size and output size?
• RQ5 How can a matrix interpretation clarify forward and backward passes for convolutions?

Key findings

• Convolution output size along each axis depends on input size, kernel size, stride, and padding, and these axes are independent.
• Half (same) padding preserves input size for odd kernel sizes with unit stride, while full padding increases output size by kernel-1.
• Non-unit strides modify the output size via a floor-based relation, and padding can be incorporated by replacing i with i+2p in the formula.
• Transposed convolution can be understood as the gradient (or transpose) of a forward convolution, with specific relationships mapping padding and stride between the two.
• Dilated convolutions increase the receptive field without enlarging the kernel, by an effective kernel size hat{k}=k+(k-1)(d-1), and adjust the output size accordingly.
• A convolution can be represented as a sparse matrix operation, enabling straightforward derivation of backward passes via transposition of the convolution matrix.