[Paper Review] A Comprehensive Survey and Performance Analysis of Activation Functions in Deep Learning.

Shiv Ram Dubey, Satish Kumar Singh+1 more|arXiv (Cornell University)|Sep 29, 2021

Advanced Neural Network Applications153 references17 citations

TL;DR

This paper presents a comprehensive survey and empirical performance analysis of 18 state-of-the-art activation functions (AFs) across diverse deep learning architectures and datasets. It evaluates AFs based on key properties like output range, smoothness, and monotonicity, and provides insights into their behavior and suitability for different tasks, with code publicly released for reproducibility.

ABSTRACT

Neural networks have shown tremendous growth in recent years to solve numerous problems. Various types of neural networks have been introduced to deal with different types of problems. However, the main goal of any neural network is to transform the non-linearly separable input data into more linearly separable abstract features using a hierarchy of layers. These layers are combinations of linear and nonlinear functions. The most popular and common non-linearity layers are activation functions (AFs), such as Logistic Sigmoid, Tanh, ReLU, ELU, Swish and Mish. In this paper, a comprehensive overview and survey is presented for AFs in neural networks for deep learning. Different classes of AFs such as Logistic Sigmoid and Tanh based, ReLU based, ELU based, and Learning based are covered. Several characteristics of AFs such as output range, monotonicity, and smoothness are also pointed out. A performance comparison is also performed among 18 state-of-the-art AFs with different networks on different types of data. The insights of AFs are presented to benefit the researchers for doing further research and practitioners to select among different choices. The code used for experimental comparison is released at: \url{this https URL}.

Motivation & Objective

• To provide a systematic overview of major classes of activation functions used in deep neural networks.
• To analyze key characteristics of activation functions, including output range, monotonicity, and smoothness.
• To empirically compare the performance of 18 state-of-the-art activation functions across multiple network architectures and datasets.
• To guide researchers and practitioners in selecting optimal activation functions based on problem-specific requirements.
• To release code for reproducibility and further benchmarking of activation function performance.

Proposed method

• Categorization of activation functions into four main classes: Logistic Sigmoid/Tanh-based, ReLU-based, ELU-based, and Learning-based AFs.
• Systematic analysis of activation function properties such as output range, monotonicity, and smoothness to understand their impact on network behavior.
• Empirical evaluation of 18 SOTA activation functions on various deep learning models and datasets to assess generalization and accuracy.
• Performance comparison across different network types (e.g., CNNs, MLPs) and data types (e.g., image, tabular) to assess robustness and adaptability.
• Use of standardized training protocols and metrics to ensure fair and reproducible comparisons.
• Open-sourcing the experimental code to enable community reuse and extension.

Experimental results

Research questions

• RQ1 How do different classes of activation functions (e.g., ReLU-based, Swish, Mish) compare in terms of performance across diverse deep learning tasks?
• RQ2 Which activation function properties—such as smoothness, output range, and monotonicity—most significantly influence model accuracy and convergence?
• RQ3 How do activation functions perform across different network architectures and data types (e.g., image vs. tabular data)?
• RQ4 What are the relative strengths and weaknesses of learning-based activation functions compared to hand-designed ones?
• RQ5 Which activation function offers the best trade-off between performance, training stability, and computational efficiency?

Key findings

• ReLU-based activation functions, particularly ReLU and its variants, consistently achieve strong performance across most benchmarks due to their computational efficiency and sparsity-inducing properties.
• Swish and Mish showed superior performance on certain vision and NLP tasks, particularly where smoothness and non-monotonicity were beneficial.
• ELU-based functions demonstrated improved convergence speed and accuracy on deeper networks, especially in convolutional architectures.
• Learning-based activation functions, while flexible, did not universally outperform hand-designed ones and often required more hyperparameter tuning.
• The choice of activation function significantly impacts model performance, with no single AF being optimal across all architectures and datasets.
• Smoothness and output range were found to be critical factors influencing training stability and final accuracy, especially in deeper networks.