[Paper Review] Batch-Shaped Channel Gated Networks.

Babak Ehteshami Bejnordi, Tijmen Blankevoort+1 more|arXiv (Cornell University)|Jul 15, 2019

Advanced Neural Network ApplicationsComputer Science27 references9 citations

TL;DR

This paper proposes Batch-Shaped Channel Gated Networks, a method that dynamically gates individual convolutional channels based on input features using a novel residual block architecture. By applying batch-shaping to align feature posteriors with a prior, the model reduces computational cost per sample to levels comparable to smaller networks while achieving higher accuracy—74.60% top-1 accuracy on ImageNet with a ResNet50-like architecture, surpassing ResNet18’s 69.76%.

ABSTRACT

We present a method that trains large capacity neural networks with significantly improved accuracy and lower dynamic computational cost. We achieve this by gating the deep-learning architecture on a fine-grained-level. Individual convolutional maps are turned on/off conditionally on features in the network. To achieve this, we introduce a new residual block architecture that gates convolutional channels in a fine-grained manner. We also introduce a generally applicable tool $batch$-$shaping$ that matches the marginal aggregate posteriors of features in a neural network to a pre-specified prior distribution. We use this novel technique to force gates to be more conditional on the data. We present results on CIFAR-10 and ImageNet datasets for image classification, and Cityscapes for semantic segmentation. Our results show that our method can slim down large architectures conditionally, such that the average computational cost on the data is on par with a smaller architecture, but with higher accuracy. In particular, on ImageNet, our ResNet50 and ResNet34 gated networks obtain 74.60% and 72.55% top-1 accuracy compared to the 69.76% accuracy of the baseline ResNet18 model, for similar complexity. We also show that the resulting networks automatically learn to use more features for difficult examples and fewer features for simple examples.

Motivation & Objective

• To reduce the dynamic computational cost of large neural networks without sacrificing accuracy.
• To enable fine-grained, conditional gating of convolutional channels based on input-dependent features.
• To develop a general-purpose tool, batch-shaping, that matches feature posterior distributions to a pre-specified prior to improve gate conditioning.
• To achieve higher accuracy than smaller baseline models while maintaining similar computational complexity.
• To enable models to automatically allocate more features to difficult examples and fewer to easy ones.

Proposed method

• Introduces a new residual block architecture that enables fine-grained, conditional gating of individual convolutional channels.
• Employs a novel batch-shaping technique to match the marginal aggregate posteriors of network features to a pre-specified prior distribution.
• Uses the batch-shaping mechanism to enhance the conditional dependence of gates on input features, improving gate sensitivity.
• Applies the gating mechanism during inference, allowing the network to dynamically activate only relevant channels per input.
• Trains the network with standard backpropagation while incorporating the batch-shaping regularization to guide gate learning.
• Leverages the conditional gating to reduce average computational cost per sample, matching that of smaller models.

Experimental results

Research questions

• RQ1 Can fine-grained, input-conditioned channel gating improve accuracy while reducing average computational cost in large networks?
• RQ2 How effective is batch-shaping in improving the conditional dependence of gates on input features?
• RQ3 Can gated networks automatically allocate more features to difficult examples and fewer to easy ones?
• RQ4 To what extent can gated large architectures outperform smaller baseline models in accuracy with similar complexity?
• RQ5 Does the proposed method generalize across different tasks and datasets, such as image classification and semantic segmentation?

Key findings

• On ImageNet, the gated ResNet50 achieves 74.60% top-1 accuracy, significantly outperforming the baseline ResNet18's 69.76%, despite similar computational complexity.
• The gated ResNet34 achieves 72.55% top-1 accuracy, demonstrating that the method improves accuracy even for smaller architectures.
• The average computational cost per sample is reduced to levels comparable to smaller networks due to conditional gating.
• The model learns to use more channels for difficult examples and fewer for easy ones, demonstrating adaptive computation.
• The method generalizes across tasks, showing strong performance on both CIFAR-10 for image classification and Cityscapes for semantic segmentation.
• Batch-shaping effectively improves gate conditioning by aligning feature posteriors with a prior, leading to more meaningful and data-dependent gating decisions.