[Paper Review] <<The>> quest for the golden activation function

Mina Basirat, Peter M. Roth|arXiv (Cornell University)|Aug 2, 2018

Machine Learning and Data Classification29 references27 citations

TL;DR

This paper proposes a neuro-evolutionary framework using genetic algorithms to automatically learn task-specific activation functions, introducing two new functions—ELiSH and HardELiSH—showcasing improved performance across CIFAR-10, CIFAR-100, and Tiny ImageNet benchmarks, with distinct optimal activation functions emerging for different datasets and architectures.

ABSTRACT

Deep Neural Networks have been shown to be beneficial for a variety of tasks, in particular allowing for end-to-end learning and reducing the requirement for manual design decisions. However, still many parameters have to be chosen in advance, also raising the need to optimize them. One important, but often ignored system parameter is the selection of a proper activation function. Thus, in this paper we target to demonstrate the importance of activation functions in general and show that for different tasks different activation functions might be meaningful. To avoid the manual design or selection of activation functions, we build on the idea of genetic algorithms to learn the best activation function for a given task. In addition, we introduce two new activation functions, ELiSH and HardELiSH, which can easily be incorporated in our framework. In this way, we demonstrate for three different image classification benchmarks that different activation functions are learned, also showing improved results compared to typically used baselines.

Motivation & Objective

• Address the lack of theoretical foundation and manual tuning in activation function selection for deep neural networks.
• Demonstrate that different image classification tasks require different optimal activation functions.
• Develop an automated, efficient, and scalable method to evolve activation functions tailored to specific tasks.
• Introduce two novel activation functions—ELiSH and HardELiSH—based on theoretical insights and empirical performance.
• Evaluate the framework across diverse benchmarks to validate generalization and performance gains.

Proposed method

• Employ a genetic algorithm framework to evolve piecewise activation functions layer-by-layer, starting from simple initial functions.
• Represent activation functions as symbolic expressions using a grammar-based representation to enable evolutionary operations like mutation and crossover.
• Use a fitness function based on validation accuracy to guide the evolutionary search toward better-performing activation functions.
• Integrate the newly proposed ELiSH and HardELiSH functions into the evolutionary framework as candidate solutions.
• Fix network architecture and optimize only the activation functions, enabling task-specific adaptation without architectural changes.
• Apply the method to ResNet-56 and VGG-16 architectures on CIFAR-10, CIFAR-100, and Tiny ImageNet datasets.

Experimental results

Research questions

• RQ1 Can a genetic algorithm effectively evolve activation functions that outperform standard baselines on image classification tasks?
• RQ2 Do different datasets require fundamentally different activation functions, even with the same network architecture?
• RQ3 How do the newly proposed ELiSH and HardELiSH functions compare to existing activation functions in terms of accuracy and stability?
• RQ4 What structural and functional characteristics define high-performing activation functions across varying dataset complexities?
• RQ5 Can the evolved activation functions generalize across different network depths and architectures?

Key findings

• For CIFAR-10, the top-performing activation function was ELiSH with 93.84% test accuracy, outperforming ReLU (92.85%) and Swish (93.42%).
• On CIFAR-100, the best-performing function was a hybrid of HardELiSH for negative inputs and max(SeLU, SeLU + Linear) for positive inputs, achieving 74.65% accuracy with ResNet-56.
• For Tiny ImageNet, ELiSH achieved 57.34% accuracy on ResNet-56, surpassing ReLU (57.27%) and SeLU (50.09%), with HardELiSH + min(ELU, Swish) achieving 57.53%.
• The evolved activation functions exhibited distinct behaviors: contraction mappings were optimal for CIFAR-10, while expansion mappings were preferred for CIFAR-100 and Tiny ImageNet.
• The top-performing functions often combined multiple operations (e.g., max, min, composition), indicating complex, non-linear behavior tailored to data distribution.
• The results confirm that activation function choice is not universal—different tasks and datasets require different functional forms, which can be automatically discovered via evolution.