[Paper Review] CondConv: Conditionally Parameterized Convolutions for Efficient Inference

Brandon Yang, Gabriel Bender+2 more|arXiv (Cornell University)|Apr 9, 2019

Advanced Neural Network Applications45 references283 citations

TL;DR

This paper introduces Conditionally Parameterized Convolutions (CondConv), which generate per-example convolutional kernels as a weighted combination of expert kernels to increase capacity without proportional inference cost, improving accuracy for ImageNet and COCO across multiple architectures.

ABSTRACT

Convolutional layers are one of the basic building blocks of modern deep neural networks. One fundamental assumption is that convolutional kernels should be shared for all examples in a dataset. We propose conditionally parameterized convolutions (CondConv), which learn specialized convolutional kernels for each example. Replacing normal convolutions with CondConv enables us to increase the size and capacity of a network, while maintaining efficient inference. We demonstrate that scaling networks with CondConv improves the performance and inference cost trade-off of several existing convolutional neural network architectures on both classification and detection tasks. On ImageNet classification, our CondConv approach applied to EfficientNet-B0 achieves state-of-the-art performance of 78.3% accuracy with only 413M multiply-adds. Code and checkpoints for the CondConv Tensorflow layer and CondConv-EfficientNet models are available at: https://github.com/tensorflow/tpu/tree/master/models/official/efficientnet/condconv.

Motivation & Objective

• Challenge the assumption of shared convolutional kernels across all examples.
• Increase model capacity and performance without a large rise in inference cost.
• Demonstrate CondConv as a drop-in replacement across CNN architectures.
• Show improved ImageNet classification and COCO detection with CondConv-enabled models.

Proposed method

• Parameterize convolutional kernels as a linear combination of n expert kernels: Output(x) = σ((α1W1 + ... + αnWn) * x).
• Compute per-example routing weights αi = ri(x) via global average pooling, a fully connected layer, and a sigmoid activation.
• Share routing weights across layers within a block to regularize and stabilize training.
• Train using either the per-example kernel computation or the equivalent linear mixture of experts formulation for efficiency.
• Apply CondConv to multiple architectures (MobileNetV1/V2, ResNet-50, MnasNet, EfficientNet) and evaluate on ImageNet and COCO.
• Regularize with dropout, AutoAugment, Mixup, and Shake-Shake-inspired expert dropout as needed.

Experimental results

Research questions

• RQ1 Does CondConv improve accuracy with only modest increases in inference cost across diverse CNN architectures?
• RQ2 How does increasing the number of experts per CondConv layer affect performance and efficiency?
• RQ3 Where in the network should CondConv be applied to maximize accuracy-cost trade-offs?
• RQ4 What is the nature of the learned routing weights and their interpretability across classes?
• RQ5 How does CondConv fare on both image classification (ImageNet) and object detection (COCO) tasks?

Key findings

• CondConv consistently improves top-1 accuracy across MobileNetV1, MobileNetV2, MnasNet-A1, ResNet-50, and EfficientNet-B0 with less than 10% inference cost increase.
• On ImageNet, CondConv with 8 experts yields 78.3% top-1 with 413M multiply-adds for EfficientNet-B0, and 79.5% with CondConv-EfficientNet-B0-depth at 614M MADDs.
• CondConv-augmented models achieve higher COCO minival mAP than baselines at comparable or lower MADDs (e.g., CondConv-MobileNetV1(0.75x) and SSD300 show higher mAP with similar cost).
• Routing weights become more class-specific at deeper layers, and final-layer experts show a bi-modal distribution, indicating specialization of experts.
• Applying CondConv across all layers yields better performance; placing CondConv in very early layers provides diminishing returns.