[Paper Review] GaterNet: Dynamic Filter Selection in Convolutional Neural Network via a Dedicated Global Gating Network.

Zhourong Chen, Yang Li+2 more|arXiv (Cornell University)|Nov 27, 2018

Advanced Neural Network ApplicationsComputer Science24 references21 citations

TL;DR

GaterNet proposes a dynamic filter selection mechanism in CNNs using a dedicated global gater network that generates binary gates to activate specific filters in the backbone network based on input-dependent conditions. This approach improves model generalization and achieves state-of-the-art performance on CIFAR-10 and consistent gains on ImageNet.

ABSTRACT

The concept of conditional computation for deep nets has been proposed previously to improve model performance by selectively using only parts of the model conditioned on the sample it is processing. In this paper, we investigate input-dependent dynamic filter selection in deep convolutional neural networks (CNNs). The problem is interesting because the idea of forcing different parts of the model to learn from different types of samples may help us acquire better filters in CNNs, improve the model generalization performance and potentially increase the interpretability of model behavior. We propose a novel yet simple framework called GaterNet, which involves a backbone and a gater network. The backbone network is a regular CNN that performs the major computation needed for making a prediction, while a global gater network is introduced to generate binary gates for selectively activating filters in the backbone network based on each input. Extensive experiments on CIFAR and ImageNet datasets show that our models consistently outperform the original models with a large margin. On CIFAR-10, our model also improves upon state-of-the-art results.

Motivation & Objective

• To improve CNN generalization through input-dependent dynamic filter selection.
• To enhance model interpretability by enabling different filters to specialize in processing different input types.
• To develop a simple yet effective framework for conditional computation in deep CNNs.
• To outperform standard CNNs and existing dynamic filter methods on benchmark datasets.

Proposed method

• A global gater network is introduced to generate binary gates for filter activation based on input features.
• The gater network processes global input statistics to produce gate values that determine which filters in the backbone are activated.
• Filter selection is dynamic and input-specific, allowing different filters to respond to different input patterns.
• The backbone CNN performs standard convolutional operations, but only activated filters contribute to the output based on gater decisions.
• The entire system is trained end-to-end with a standard classification loss.
• Binary gating enables computational efficiency and model sparsity, improving inference efficiency.

Experimental results

Research questions

• RQ1 Can input-dependent dynamic filter selection improve CNN generalization and performance?
• RQ2 How does a global gater network compare to other dynamic routing mechanisms in filter selection?
• RQ3 Can dynamic filter selection enhance model interpretability by enabling filter specialization?
• RQ4 What performance gains can be achieved on standard benchmarks like CIFAR-10 and ImageNet?
• RQ5 Does the proposed method achieve state-of-the-art results compared to existing dynamic CNN approaches?

Key findings

• GaterNet achieves consistent performance gains over standard CNNs on both CIFAR-10 and ImageNet datasets.
• On CIFAR-10, the model improves upon state-of-the-art results, demonstrating strong generalization.
• The dynamic filter selection mechanism enhances model interpretability by enabling specialized filter behavior per input type.
• The use of binary gates enables computational efficiency and model sparsity without sacrificing accuracy.
• The global gater network effectively learns to route inputs to appropriate filters based on input characteristics.
• The framework is simple yet effective, showing strong empirical results across multiple benchmarks.