[Paper Review] Searching for Activation Functions

Prajit Ramachandran, Barret Zoph+1 more|arXiv (Cornell University)|Oct 16, 2017

Domain Adaptation and Few-Shot Learning39 references750 citations

TL;DR

The paper uses automated search to discover scalar activation functions, introduces Swish (f(x)=x·sigmoid(βx)), and shows Swish often outperforms ReLU across deep models and tasks.

ABSTRACT

The choice of activation functions in deep networks has a significant effect on the training dynamics and task performance. Currently, the most successful and widely-used activation function is the Rectified Linear Unit (ReLU). Although various hand-designed alternatives to ReLU have been proposed, none have managed to replace it due to inconsistent gains. In this work, we propose to leverage automatic search techniques to discover new activation functions. Using a combination of exhaustive and reinforcement learning-based search, we discover multiple novel activation functions. We verify the effectiveness of the searches by conducting an empirical evaluation with the best discovered activation function. Our experiments show that the best discovered activation function, $f(x) = x \cdot ext{sigmoid}(βx)$, which we name Swish, tends to work better than ReLU on deeper models across a number of challenging datasets. For example, simply replacing ReLUs with Swish units improves top-1 classification accuracy on ImageNet by 0.9\% for Mobile NASNet-A and 0.6\% for Inception-ResNet-v2. The simplicity of Swish and its similarity to ReLU make it easy for practitioners to replace ReLUs with Swish units in any neural network.

Motivation & Objective

• Motivate the impact of activation functions on training dynamics and task performance.
• Propose a search-based approach to discover novel scalar activation functions.
• Identify and validate top activation functions found by exhaustive and reinforcement learning-based search.
• Demonstrate Swish's empirical benefits across multiple architectures and datasets.

Proposed method

• Design a search space built from unary and binary functions to compose activation functions via repeated core units.
• Use exhaustive search for small spaces and an RNN controller with reinforcement learning for large spaces to propose candidate functions.
• Train a child network (e.g., ResNet-20 on CIFAR-10) to evaluate each candidate by validation accuracy.
• Employ distributed training to parallelize training of candidate activation functions and update the search policy based on rewards.
• Define Swish as f(x)=x·sigmoid(βx) with β either fixed or trainable, and analyze its properties and derivatives.

Experimental results

Research questions

• RQ1 Can automated search discover activation functions that outperform hand-designed ones like ReLU?
• RQ2 What are the characteristics of high-performing activation functions found by the search?
• RQ3 Does Swish generalize across multiple model families and tasks beyond the search setting?
• RQ4 How does Swish compare to ReLU on large-scale datasets like ImageNet and NLP translation tasks?

Key findings

• Swish (f(x)=x·sigmoid(βx)) often matches or outperforms ReLU on deep networks across datasets and architectures.
• Swish with β fixed at 1 (Swish-1) or trainable β frequently outperforms ReLU on CIFAR-10/100, ImageNet mobile models, and several ImageNet architectures.
• Top activation functions tend to be simple (1–2 core units) and often use the raw preactivation x as input to the final binary function.
• Swish is smooth, non-monotonic, unbounded above, and its gradient properties differ from ReLU, with favorable optimization behavior observed in practice.
• On ImageNet, Swish improves top-1 accuracy by 0.9% on Mobile NASNet-A and 0.6% on Inception-ResNet-v2 when replacing ReLU.
• Swish-1 and Swish consistently match or outperform baselines across multiple model families and tasks, including machine translation with Transformers.