[Paper Review] An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solution

Rosanne Liu, Joel Lehman+5 more|arXiv (Cornell University)|Jul 9, 2018

Neural Networks and ApplicationsComputer Science645 citations

TL;DR

The paper shows CNNs struggle with coordinate transforms between Cartesian coordinates and pixel space, and introduces CoordConv, which appends coordinate channels to inputs to enable learned translation-dependent representations, improving speed and parameter efficiency.

ABSTRACT

Few ideas have enjoyed as large an impact on deep learning as convolution. For any problem involving pixels or spatial representations, common intuition holds that convolutional neural networks may be appropriate. In this paper we show a striking counterexample to this intuition via the seemingly trivial coordinate transform problem, which simply requires learning a mapping between coordinates in (x,y) Cartesian space and one-hot pixel space. Although convolutional networks would seem appropriate for this task, we show that they fail spectacularly. We demonstrate and carefully analyze the failure first on a toy problem, at which point a simple fix becomes obvious. We call this solution CoordConv, which works by giving convolution access to its own input coordinates through the use of extra coordinate channels. Without sacrificing the computational and parametric efficiency of ordinary convolution, CoordConv allows networks to learn either complete translation invariance or varying degrees of translation dependence, as required by the end task. CoordConv solves the coordinate transform problem with perfect generalization and 150 times faster with 10--100 times fewer parameters than convolution. This stark contrast raises the question: to what extent has this inability of convolution persisted insidiously inside other tasks, subtly hampering performance from within? A complete answer to this question will require further investigation, but we show preliminary evidence that swapping convolution for CoordConv can improve models on a diverse set of tasks. Using CoordConv in a GAN produced less mode collapse as the transform between high-level spatial latents and pixels becomes easier to learn. A Faster R-CNN detection model trained on MNIST showed 24% better IOU when using CoordConv, and in the RL domain agents playing Atari games benefit significantly from the use of CoordConv layers.

Motivation & Objective

• Demonstrate a surprising difficulty of standard CNNs at learning Cartesian-to-pixel coordinate transforms.
• Introduce CoordConv as a drop-in layer to provide access to coordinate information.
• Show CoordConv enables learning translation-aware representations with fewer parameters and faster training.
• Evaluate CoordConv across toy tasks and real-world models to assess generality and impact.

Proposed method

• Define Not-so-Clevr toy dataset with 9x9 squares on a 64x64 canvas and three fields per example (center coordinates, center pixel one-hot, and rendered image).
• Propose CoordConv layer by adding hard-coded coordinate channels to inputs before a standard convolution, effectively giving filters access to Cartesian coordinates.
• Compare standard convolutional networks versus CoordConv on supervised coordinate classification, regression, and rendering tasks with uniform and quadrant train/test splits.
• Demonstrate that CoordConv preserves efficiency with small parameter overhead and translate-invariant behavior controllable by learning.
• Apply CoordConv as a drop-in replacement in broader models to assess impact on image classification, object detection, generative modeling, and reinforcement learning.

Experimental results

Research questions

• RQ1 Can CNNs learn the mapping from Cartesian coordinates to pixel-space representations efficiently with standard convolution?
• RQ2 Does introducing explicit coordinate information via CoordConv improve learning of coordinate transforms and generalization?
• RQ3 Do CoordConv layers provide benefits beyond toy tasks in real-world models (detectors, GANs/VAEs, RL)?

Key findings

• Coordinate transform tasks are hard for standard CNNs even when supervised, with quadrant splits showing essentially no generalization.
• CoordConv achieves perfect train and test accuracy on coordinate tasks with far fewer parameters and much faster training (seconds vs hours).
• Replacing convolution with CoordConv improves performance in diverse settings, including MNIST-like object detection (24% IOU improvement with Faster R-CNN) and reduced mode collapse in GANs/VAEs.
• On ImageNet classification, CoordConv provides negligible improvement, indicating limited benefit for translation-invariant classification tasks.
• In Atari RL tasks, CoordConv improves performance on many games, though not universally across all games.