[Paper Review] Data-aware customization of activation functions reduces neural network error

Fuchang Gao, Boyu Zhang|arXiv (Cornell University)|Jan 16, 2023

Machine Learning in Materials Science5 citations

TL;DR

This paper proposes data-aware customization of activation functions to reduce neural network error, introducing the 'seagull' activation function log(1 + x²) for partially exchangeable target functions. By leveraging linear algebra and the Diaconis-Shahshahani Approximation Theorem, the method ensures that even activation functions preserve symmetry in predictions, leading to up to an order-of-magnitude error reduction in regression tasks, especially when applied at the first layer processing exchangeable variables.

ABSTRACT

Activation functions play critical roles in neural networks, yet current off-the-shelf neural networks pay little attention to the specific choice of activation functions used. Here we show that data-aware customization of activation functions can result in striking reductions in neural network error. We first give a simple linear algebraic explanation of the role of activation functions in neural networks; then, through connection with the Diaconis-Shahshahani Approximation Theorem, we propose a set of criteria for good activation functions. As a case study, we consider regression tasks with a partially exchangeable target function, \emph{i.e.} $f(u,v,w)=f(v,u,w)$ for $u,v\in \mathbb{R}^d$ and $w\in \mathbb{R}^k$, and prove that for such a target function, using an even activation function in at least one of the layers guarantees that the prediction preserves partial exchangeability for best performance. Since even activation functions are seldom used in practice, we designed the ``seagull'' even activation function $\log(1+x^2)$ according to our criteria. Empirical testing on over two dozen 9-25 dimensional examples with different local smoothness, curvature, and degree of exchangeability revealed that a simple substitution with the ``seagull'' activation function in an already-refined neural network can lead to an order-of-magnitude reduction in error. This improvement was most pronounced when the activation function substitution was applied to the layer in which the exchangeable variables are connected for the first time. While the improvement is greatest for low-dimensional data, experiments on the CIFAR10 image classification dataset showed that use of ``seagull'' can reduce error even for high-dimensional cases. These results collectively highlight the potential of customizing activation functions as a general approach to improve neural network performance.

Motivation & Objective

• To address the poor generalization of standard neural networks by integrating domain-specific structural knowledge into activation function design.
• To improve model performance by customizing activation functions based on underlying data structure, particularly symmetry properties like partial exchangeability.
• To demonstrate that even activation functions can preserve functional symmetries critical for accurate prediction in structured data.
• To develop a practical, plug-and-play activation function—'seagull'—that enhances performance without architectural changes.
• To validate the approach across diverse regression problems and high-dimensional datasets like CIFAR10.

Proposed method

• Using linear algebra to frame neural network approximation as a rank-enhancement problem via activation functions.
• Deriving theoretical criteria for effective activation functions using the Diaconis-Shahshahani Approximation Theorem.
• Proving that even activation functions preserve partial exchangeability in target functions f(u,v,w) = f(v,u,w).
• Designing the 'seagull' activation function g(x) = log(1 + x²) to satisfy theoretical criteria and ensure even symmetry.
• Applying the seagull function as a replacement in existing neural networks without retraining or modifying loss functions.
• Evaluating performance across 26+ regression tasks with varying dimensionality, smoothness, curvature, and exchangeability.

Experimental results

Research questions

• RQ1 Can data-aware customization of activation functions significantly reduce neural network error in regression tasks?
• RQ2 Does using an even activation function preserve partial exchangeability in target functions, leading to improved performance?
• RQ3 How does the 'seagull' activation function log(1 + x²) compare to standard activations in terms of error reduction?
• RQ4 Is the performance gain most pronounced when the seagull function is applied at the first layer processing exchangeable variables?
• RQ5 Can this approach generalize to high-dimensional data, such as image classification on CIFAR10?

Key findings

• Substituting standard activation functions with the 'seagull' function log(1 + x²) led to an order-of-magnitude reduction in error across multiple regression tasks.
• The error reduction was most significant when the seagull function was applied to the layer where exchangeable variables were first connected.
• The improvement was consistent across different original activation functions, loss functions, optimizers, and levels of data noise.
• Even in high-dimensional settings like CIFAR10, the seagull function reduced error, though the gains were smaller than in low-dimensional cases.
• Theoretical analysis confirmed that even activation functions preserve partial exchangeability, explaining the performance gains.
• Empirical results validated that the seagull function outperforms standard ReLU and other common activations under the same training conditions.