[Paper Review] What Do We Understand About Convolutional Networks?

Isma Hadji, Richard P. Wildes|arXiv (Cornell University)|Mar 23, 2018

Cell Image Analysis Techniques131 references101 citations

TL;DR

A survey of convolutional networks detailing architectures, building blocks, training methods, and transfer learning, with discussion of biology-inspired design and open challenges.

ABSTRACT

This document will review the most prominent proposals using multilayer convolutional architectures. Importantly, the various components of a typical convolutional network will be discussed through a review of different approaches that base their design decisions on biological findings and/or sound theoretical bases. In addition, the different attempts at understanding ConvNets via visualizations and empirical studies will be reviewed. The ultimate goal is to shed light on the role of each layer of processing involved in a ConvNet architecture, distill what we currently understand about ConvNets and highlight critical open problems.

Motivation & Objective

• Motivate the need to understand ConvNets beyond their performance on benchmarks.
• Review prominent multilayer architectures and their design decisions.
• Examine the building blocks of ConvNets from biological and theoretical perspectives.
• Summarize visualization and empirical studies used to understand ConvNets.
• Highlight current trends and critical open problems in ConvNet understanding.

Proposed method

• Review of multilayer network architectures and their historical development.
• Discussion of neural network building blocks (RBMs, auto-encoders, RNNs, ConvNets, GANs).
• Description of training methods including gradient descent, backpropagation, and unsupervised pretraining.
• Analysis of transfer learning and its practical guidelines.
• Synthesis of architectural evolutions in 2D ConvNets (AlexNet to DenseNet) and their core innovations.

Experimental results

Research questions

• RQ1 What aspects of learned convolutional kernels and architecture choices are actually responsible for ConvNet performance?
• RQ2 Why do certain architectural decisions (e.g., depth, pooling, nonlinearity) lead to better results?
• RQ3 How do various training and pretraining strategies affect data efficiency and generalization?
• RQ4 What is the role and impact of transfer learning across datasets and tasks?
• RQ5 What are the open problems and limitations in understanding ConvNets?

Key findings

• ConvNets combine local connections and weight sharing with pooling to build hierarchical representations and invariances.
• Deeper networks (e.g., VGG, GoogLeNet, ResNet, DenseNet) achieve better performance, enabled by innovations like smaller filters, inception modules, and skip connections.
• ReLU activations, dropout, and data augmentation significantly aid training efficiency and generalization.
• Transfer learning is effective due to hierarchical representations, with higher-layer fine-tuning often yielding best results; similarity between tasks affects transfer.
• GANs provide unsupervised learning capabilities and have spurred advances in image synthesis and related tasks.
• Training strategies evolved from unsupervised pretraining (e.g., RBMs) to supervised backpropagation, with ongoing interest in hybrid and self-supervised methods.