[Paper Review] Activation Functions: Comparison of trends in Practice and Research for Deep Learning

Chigozie Nwankpa, Winifred Ijomah+2 more|arXiv (Cornell University)|Nov 8, 2018

Advanced Neural Network Applications42 references1,168 citations

TL;DR

This paper surveys activation functions (AFs) used in deep learning, compiling existing AFs and contrasting practical deployment trends with literature results to help function choice in applications.

ABSTRACT

Deep neural networks have been successfully used in diverse emerging domains to solve real world complex problems with may more deep learning(DL) architectures, being developed to date. To achieve these state-of-the-art performances, the DL architectures use activation functions (AFs), to perform diverse computations between the hidden layers and the output layers of any given DL architecture. This paper presents a survey on the existing AFs used in deep learning applications and highlights the recent trends in the use of the activation functions for deep learning applications. The novelty of this paper is that it compiles majority of the AFs used in DL and outlines the current trends in the applications and usage of these functions in practical deep learning deployments against the state-of-the-art research results. This compilation will aid in making effective decisions in the choice of the most suitable and appropriate activation function for any given application, ready for deployment. This paper is timely because most research papers on AF highlights similar works and results while this paper will be the first, to compile the trends in AF applications in practice against the research results from literature, found in deep learning research to date.

Motivation & Objective

• Summarize activation functions used in deep learning and their historical evolution.
• Highlight trends in practical deployments of AFs across diverse deep learning applications.
• Provide a compiled reference of AF types and their reported advantages and limitations for deployment decisions.

Proposed method

• Organizes activation functions into categories and variants, detailing definitions and formulas across AF families.
• Reviews literature results and notes practical deployment trends for AFs in deep architectures.
• Compares performance attributes and training behaviors of AFs to guide selection for specific tasks.

Experimental results

Research questions

• RQ1 What activation functions are commonly used in deep learning in practice and in research?
• RQ2 What are the observed trends or differences between practice and research in AF usage across domains?
• RQ3 What guidance can be given for selecting activation functions for particular applications based on compiled evidence?

Key findings

• The paper compiles a comprehensive list of activation functions and variants including Sigmoid, Tanh, ReLU, Leaky ReLU, PReLU, RReLU, SReLU, Softplus, ELU, PELU, SELU, Maxout, Swish, ELiSH, and HardELiSH.
• It discusses the advantages and drawbacks of each AF, such as training speed, gradient behavior, and zero-centering effects, to inform deployment decisions.
• The work emphasizes the gap between practical AF usage and reported research results, aiming to align choices with deployment contexts.
• It highlights how newer AFs like Swish and ELiSH compare to traditional functions like ReLU in terms of information flow and training dynamics.
• The paper positions AF selection as a key factor for performance and generalization across diverse tasks like object recognition, speech, NLP, and more.