[Paper Review] Aggregated Residual Transformations for Deep Neural Networks

Saining Xie, Ross Girshick+3 more|arXiv (Cornell University)|Nov 16, 2016

Adversarial Robustness in Machine Learning40 references385 citations

TL;DR

The paper introduces ResNeXt, a modular multi-branch architecture that increases cardinality (the number of parallel transformations) to improve accuracy without increasing depth or width, and demonstrates its effectiveness on ImageNet, ImageNet-5K, CIFAR, and COCO compared to ResNet and Inception variants.

ABSTRACT

We present a simple, highly modularized network architecture for image classification. Our network is constructed by repeating a building block that aggregates a set of transformations with the same topology. Our simple design results in a homogeneous, multi-branch architecture that has only a few hyper-parameters to set. This strategy exposes a new dimension, which we call "cardinality" (the size of the set of transformations), as an essential factor in addition to the dimensions of depth and width. On the ImageNet-1K dataset, we empirically show that even under the restricted condition of maintaining complexity, increasing cardinality is able to improve classification accuracy. Moreover, increasing cardinality is more effective than going deeper or wider when we increase the capacity. Our models, named ResNeXt, are the foundations of our entry to the ILSVRC 2016 classification task in which we secured 2nd place. We further investigate ResNeXt on an ImageNet-5K set and the COCO detection set, also showing better results than its ResNet counterpart. The code and models are publicly available online.

Motivation & Objective

• Motivate architecture design by exploring how increasing cardinality affects representational power in CNNs.
• Propose a modular, homogeneous multi-branch block that aggregates transformed embeddings.
• Show that increasing cardinality can outperform deeper or wider networks at similar complexity.
• Demonstrate the effectiveness of ResNeXt on ImageNet, ImageNet-5K, CIFAR, and COCO datasets.

Proposed method

• Adopt a modular residual block where multiple transformations of the same topology are aggregated by summation.
• Define cardinality C as the number of parallel transformations in a block and keep width/depth fixed while varying C.
• Use bottleneck-shaped transformations for each path to control parameter count and FLOPs.
• Reformulate the block equivalently via concatenation or grouped convolutions to illustrate different implementations.
• Preserve overall computational complexity while varying cardinality to compare performance.
• Train using SGD with multi-GPU setup and standard data augmentation; implement with grouped convolutions to realize the ResNeXt block.

Experimental results

Research questions

• RQ1 Does increasing cardinality (the number of parallel transformations) improve accuracy when depth and width are held roughly constant?
• RQ2 How does ResNeXt compare to ResNet and Inception-style architectures at similar computational budgets on ImageNet?
• RQ3 Is the gain from higher cardinality consistent across different scales of data (ImageNet, ImageNet-5K) and tasks (classification, detection, CIFAR)?
• RQ4 What is the role of residual connections versus aggregated transformations in optimization and representational power?

Key findings

• Increasing cardinality yields lower validation error than baseline ResNets at similar complexity.
• For ResNeXt-50, increasing C to 32 with 4d bottleneck reduces top-1 error from 23.9% (ResNet-50) to 22.2%.
• ResNeXt-101 with 32×4d achieves 21.2% top-1 error, outperforming ResNet-101 (21.2% vs 21.2%? Note: source shows 21.2% for ResNeXt-101 and 21.2% for ResNet-101; interpret carefully: the table indicates ResNeXt-101 32×4d 21.2%, ResNet-101 1×64d 22.0% – citation in text).
• Increasing cardinality with preserved complexity outperforms going deeper or wider at similar FLOPs; e.g., 2× FLOPs with ResNeXt-101 (32×4d) improves over ResNet-101 by up to ~1.3% top-1.
• On ImageNet-5K, ResNeXt-50 reduces 5K-way top-1 error by 3.2% over ResNet-50 and ResNeXt-101 reduces by 2.3% over ResNet-101.
• ResNeXt improves COCO detection AP and AP@IoU=0.5 over ResNet counterparts without increasing complexity.