[Paper Review] Activation Functions in Artificial Neural Networks: A Systematic Overview

Johannes Lederer|arXiv (Cornell University)|Jan 25, 2021

Neural Networks and ApplicationsComputer Science32 references44 citations

TL;DR

This paper provides a comprehensive, analytic survey of common activation functions, their mathematical properties (including first and second derivatives), and the practical implications for expressivity and optimization in neural networks.

ABSTRACT

Activation functions shape the outputs of artificial neurons and, therefore, are integral parts of neural networks in general and deep learning in particular. Some activation functions, such as logistic and relu, have been used for many decades. But with deep learning becoming a mainstream research topic, new activation functions have mushroomed, leading to confusion in both theory and practice. This paper provides an analytic yet up-to-date overview of popular activation functions and their properties, which makes it a timely resource for anyone who studies or applies neural networks.

Motivation & Objective

• Motivate the need for a systematic, objective overview of activation functions given the proliferation of options.
• Summarize the mathematical properties of common activation functions (including derivatives and curvature).
• Discuss practical implications for network expressivity, optimization costs, and theoretical analyses.
• Provide guidance for theorists and practitioners on how activation choices affect learning dynamics.

Proposed method

• Define activation functions and their derivatives (first and second) and directional derivatives.
• Classify and analyze sigmoid-type activations (logistic, arctan, tanh, softsign) and their properties.
• Discuss implications of activation derivatives on optimization algorithms (e.g., SGD) and computational costs.
• Explain how activation choices influence network expressivity (e.g., linear vs. nonlinear nets) and theoretical considerations.

Experimental results

Research questions

• RQ1 What are the common activation functions and their key mathematical properties?
• RQ2 How do activation functions and their derivatives affect the expressivity and optimization of neural networks?
• RQ3 What practical implications arise from different activation choices for training dynamics and theory?
• RQ4 How does linear activation influence the ability of networks to approximate functions?
• RQ5 What guidance can be drawn for practitioners selecting activation functions in deep learning?

Key findings

• Activation functions shape network outputs and influence both expressivity and optimization complexity.
• First and second derivatives of activation functions impact gradient-based optimization and theoretical analyses.
• Linear-activation networks are necessarily linear and cannot approximate nonlinear target functions, highlighting the role of nonlinearity in expressivity.
• Sigmoid-type activations (logistic, arctan, tanh, softsign) provide smooth, bounded alternatives to hard thresholds and differ mainly in output range.
• Logistic activation serves as a smooth approximation to the binary step, with all derivatives being smooth and well-defined.
• The work emphasizes objective, mathematical evaluation of activation functions over heuristic reasons, with proofs and appendices providing rigorous support.