[Paper Review] ARiA: Utilizing Richard's Curve for Controlling the Non-monotonicity of the Activation Function in Deep Neural Nets

Narendra Patwardhan, Madhura Ingalhalikar+1 more|arXiv (Cornell University)|May 22, 2018

Advanced Neural Network Applications12 references4 citations

TL;DR

This paper proposes ARiA, a novel two-parameter activation function based on Richard's Curve, designed to control non-monotonicity in deep neural networks more precisely than Swish or ReLU. By tuning parameters α and β, ARiA2 enables independent control over convexity in both positive and negative regions, achieving state-of-the-art accuracy on MNIST, CIFAR-10, and CIFAR-100 with faster convergence and superior performance compared to ReLU and Swish.

ABSTRACT

This work introduces a novel activation unit that can be efficiently employed in deep neural nets (DNNs) and performs significantly better than the traditional Rectified Linear Units (ReLU). The function developed is a two parameter version of the specialized Richard's Curve and we call it Adaptive Richard's Curve weighted Activation (ARiA). This function is non-monotonous, analogous to the newly introduced Swish, however allows a precise control over its non-monotonous convexity by varying the hyper-parameters. We first demonstrate the mathematical significance of the two parameter ARiA followed by its application to benchmark problems such as MNIST, CIFAR-10 and CIFAR-100, where we compare the performance with ReLU and Swish units. Our results illustrate a significantly superior performance on all these datasets, making ARiA a potential replacement for ReLU and other activations in DNNs.

Motivation & Objective

• To address the limitations of ReLU and Swish in controlling non-monotonic curvature via hyperparameters.
• To develop a differentiable, non-monotonic activation function with tunable convexity in both positive and negative input regions.
• To improve training dynamics and test accuracy in deep neural networks through precise control of activation shape.
• To provide a computationally efficient alternative to Swish with better hyperparameter control, including a single-parameter variant (ARiA1) for resource-constrained settings.

Proposed method

• Proposes ARiA as a two-parameter activation function derived from Richard’s Curve, defined as f(x) = x * σ(L), where σ(L) is a modified logistic sigmoid.
• Introduces ARiA2 with parameters α and β to independently control the convexity in the first and third quadrants.
• Employs a modified sigmoid function σ(β, x) = (1 + e^(-βx))^(-1) scaled by α to modulate the activation’s curvature and slope.
• Applies ARiA2 to feedforward and convolutional networks, including DenseNet and Wide Residual Networks, using standard optimizers like Adam and SGD.
• Validates performance across MNIST, CIFAR-10, and CIFAR-100 with hyperparameter search over α and β.
• Introduces ARiA1 as a simplified variant with only α as tunable parameter, fixing β = 1, to reduce computational cost.

Experimental results

Research questions

• RQ1 Can a two-parameter activation function derived from Richard’s Curve provide finer control over non-monotonic curvature than Swish?
• RQ2 Does ARiA2 outperform ReLU and Swish in terms of test accuracy and convergence speed on standard benchmarks?
• RQ3 Can ARiA1 serve as an effective, low-complexity alternative to Swish and ReLU in resource-constrained training?
• RQ4 How do the hyperparameters α and β independently influence the shape and performance of the activation function?

Key findings

• On MNIST with DenseNet, ARiA2 with α = 1.75, β = 1 achieved 98.84% test accuracy, outperforming ReLU (98.72%) and Swish.
• On CIFAR-10, ARiA2 with α = 1.5, β = 1 reached 95.43% accuracy at 100 epochs, surpassing ReLU (92.55%) and Swish.
• On CIFAR-100, ARiA2 with α = 1.5, β = 2 achieved 70.17% accuracy at 100 epochs, exceeding ReLU (68.06%) and Swish.
• ARiA1 with α = 1.5, β = 1 achieved 94.37% accuracy on CIFAR-10 at 100 epochs, demonstrating strong performance with only one tunable parameter.
• The results show that ARiA2 consistently outperforms ReLU and Swish across all datasets and architectures, with faster convergence and higher accuracy.
• The study confirms that precise control over non-monotonic convexity via α and β is key to ARiA’s superior performance.