[Paper Review] Swish: a Self-Gated Activation Function

Prajit Ramachandran, Barret Zoph+1 more|arXiv (Cornell University)|Oct 16, 2017

Domain Adaptation and Few-Shot Learning19 references326 citations

TL;DR

This paper introduces Swish, a self-gated activation function defined as $f(x) = x \cdot \text{sigmoid}(x)$, which outperforms ReLU in deeper networks. Experiments show it improves ImageNet top-1 accuracy by 0.9% on Mobile NASNet-A and 0.6% on Inception-ResNet-v2, with minimal architectural changes required for deployment.

ABSTRACT

The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose a new activation function, named Swish, which is simply $f(x) = x \cdot ext{sigmoid}(x)$. Our experiments show that Swish tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9% for Mobile NASNet-A and 0.6% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

Motivation & Objective

• To address the limitations of ReLU and existing ReLU alternatives in deep learning models.
• To identify an activation function that consistently improves performance across deeper architectures.
• To develop a simple, effective activation function that maintains compatibility with existing neural network frameworks.
• To evaluate the impact of the proposed activation on challenging benchmarks like ImageNet.

Proposed method

• Propose Swish as a new activation function defined by $f(x) = x \cdot \text{sigmoid}(x)$.
• Leverage the smooth, non-monotonic nature of the sigmoid gate to enable self-gating behavior.
• Use gradient-based optimization to train models with Swish in place of ReLU.
• Evaluate performance on standard benchmarks including ImageNet, Mobile NASNet-A, and Inception-ResNet-v2.
• Ensure backward compatibility by maintaining the same input-output dimensionality as ReLU.

Experimental results

Research questions

• RQ1 Does a self-gated activation function like Swish improve performance on deeper models compared to ReLU?
• RQ2 Can Swish achieve consistent gains across diverse and challenging deep learning architectures?
• RQ3 How does Swish compare to other ReLU variants in terms of accuracy and training dynamics?
• RQ4 Is Swish easy to integrate into existing neural network frameworks without architectural changes?

Key findings

• Swish improves top-1 ImageNet classification accuracy by 0.9% when replacing ReLU in Mobile NASNet-A.
• Swish achieves a 0.6% accuracy gain over ReLU on Inception-ResNet-v2 for ImageNet top-1 classification.
• The performance gains are consistent across deeper models, indicating Swish's effectiveness in deep architectures.
• Swish maintains simplicity and compatibility, enabling straightforward replacement of ReLU units in any neural network.