[论文解读] Three Decades of Activations: A Comprehensive Survey of 400 Activation Functions for Neural Networks

Vladimír Kunc, Jǐŕı Kléma|arXiv (Cornell University)|Feb 14, 2024

Neural Networks and Applications被引用 14

一句话总结

一份全面的综述，编目并讨论在三十年内用于神经网络的400个激活函数。

ABSTRACT

Neural networks have proven to be a highly effective tool for solving complex problems in many areas of life. Recently, their importance and practical usability have further been reinforced with the advent of deep learning. One of the important conditions for the success of neural networks is the choice of an appropriate activation function introducing non-linearity into the model. Many types of these functions have been proposed in the literature in the past, but there is no single comprehensive source containing their exhaustive overview. The absence of this overview, even in our experience, leads to redundancy and the unintentional rediscovery of already existing activation functions. To bridge this gap, our paper presents an extensive survey involving 400 activation functions, which is several times larger in scale than previous surveys. Our comprehensive compilation also references these surveys; however, its main goal is to provide the most comprehensive overview and systematization of previously published activation functions with links to their original sources. The secondary aim is to update the current understanding of this family of functions.

研究动机与目标

• 激发编目和分析神经网络中庞大激活函数景观的必要性。
• 提供结构化的分类法和比较概述，帮助研究人员为特定任务选择激活函数。
• 突出三十年来激活函数研究的历史发展、趋势线和存在的空白。

提出的方法

• 编纂一个建立在大量文献基础上的400个激活函数目录。
• 按数学性质、行为和实际考虑对函数进行分类。
• 概述每个函数的核心特征、优点、局限性及典型用例。

实验结果

研究问题

• RQ1 定义神经网络中使用的激活函数的广义类别和属性是什么？
• RQ2 在过去三十年中，激活函数如何演变，采用和性能上出现了哪些模式？
• RQ3 可以为从业者在选择适用于不同网络结构和任务的激活函数时提供哪些指南？
• RQ4 在激活函数研究中还存在哪些空白或尚待解答的问题？

主要发现

• 本论文编纂并整理了400个激活函数，为研究人员提供了一个整合的参考。
• 它提供了一个分类法和比较视角，以便在各种神经网络场景中选择函数。
• 该综述强调影响激活函数选择的历史趋势和实际考虑因素。
• 它指出文献中的潜在空白以及在激活函数设计方面未来探索的机会。