[Paper Review] A Modified Activation Function with Improved Run-Times For Neural Networks

V.I.E Anireh, Emmanuel N. Osegi|arXiv (Cornell University)|Jul 6, 2016

Neural Networks and Applications16 references3 citations

TL;DR

This paper proposes a modified hyperbolic tangent activation function that replaces the irrational constant e with an integer approximation derived from a quadratic Real Number Formula (RNF) algorithm, combined with adaptive normalization to mitigate vanishing gradients. The method achieves faster training run-times and improved learning accuracy on both synthetic and real-world datasets, demonstrating speed-ups without sacrificing performance.

ABSTRACT

In this paper we present a modified version of the Hyperbolic Tangent Activation Function as a learning unit generator for neural networks. The function uses an integer calibration constant as an approximation to the Euler number, e, based on a quadratic Real Number Formula (RNF) algorithm and an adaptive normalization constraint on the input activations to avoid the vanishing gradient. We demonstrate the effectiveness of the proposed modification using a hypothetical and real world dataset and show that lower run-times can be achieved by learning algorithms using this function leading to improved speed-ups and learning accuracies during training.

Motivation & Objective

• To address the computational overhead of using the transcendental constant e in traditional activation functions like tanh.
• To reduce training run-time in neural networks without compromising learning accuracy.
• To mitigate the vanishing gradient problem during backpropagation through input normalization and function modification.
• To develop a computationally efficient activation function suitable for real-time and resource-constrained applications.
• To validate the proposed function on both hypothetical and real-world datasets for performance and speed evaluation.

Proposed method

• Replaces the Euler's number e in the standard hyperbolic tangent function with an integer-calibrated approximation derived from a quadratic Real Number Formula (RNF) algorithm.
• Applies an adaptive normalization constraint to input activations to stabilize gradients and prevent saturation.
• Designs a modified activation function that maintains the S-shaped curve of tanh but with faster computation due to integer arithmetic.
• Employs a calibration constant derived from RNF to approximate e with high precision using only integer operations.
• Integrates the modified function into feedforward neural network architectures for training and evaluation.
• Uses a hybrid approach combining numerical approximation and dynamic input scaling to enhance training stability and speed.

Experimental results

Research questions

• RQ1 Can replacing the irrational constant e in the tanh function with an integer approximation reduce computational cost in neural network training?
• RQ2 Does the proposed activation function maintain or improve learning accuracy compared to standard tanh and ReLU variants?
• RQ3 To what extent does adaptive normalization mitigate the vanishing gradient problem in deep networks?
• RQ4 Can the modified activation function achieve measurable run-time improvements in both synthetic and real-world datasets?
• RQ5 How does the RNF-based approximation of e compare to standard floating-point implementations in terms of precision and speed?

Key findings

• The modified activation function achieved lower training run-times compared to standard tanh and ReLU on both hypothetical and real-world datasets.
• The use of an integer-calibrated approximation of e reduced computational complexity, enabling faster inference and training cycles.
• Adaptive normalization effectively mitigated the vanishing gradient problem, especially in deeper network architectures.
• The proposed function demonstrated improved learning accuracy on benchmark datasets, outperforming standard tanh in some cases.
• The RNF algorithm provided a precise integer approximation of e that maintained functional similarity to the original tanh while enabling faster arithmetic.
• Empirical results showed measurable speed-ups in training, with the method being particularly effective in resource-constrained environments.