[Paper Review] TanhSoft -- a family of activation functions combining Tanh and Softplus

Koushik Biswas, Sandeep Kumar+2 more|arXiv (Cornell University)|Sep 8, 2020

Advanced Neural Network Applications26 references4 citations

TL;DR

This paper introduces TanhSoft, a novel family of activation functions combining tanh and softplus components via four tunable hyper-parameters: $\tanh(\alpha x + \beta e^{\gamma x})\ln(\delta + e^x)$. Through hyper-parameter tuning, specific TanhSoft variants outperform ReLU, Swish, and other standard activation functions, achieving up to 2.57% higher top-1 accuracy on CIFAR-100 with Inception-v3 and 1.24% improvement with DenseNet-169.

ABSTRACT

Deep learning at its core, contains functions that are composition of a linear transformation with a non-linear function known as activation function. In past few years, there is an increasing interest in construction of novel activation functions resulting in better learning. In this work, we propose a family of novel activation functions, namely TanhSoft, with four undetermined hyper-parameters of the form tanh(αx+βe^{γx})ln(δ+e^x) and tune these hyper-parameters to obtain activation functions which are shown to outperform several well known activation functions. For instance, replacing ReLU with xtanh(0.6e^x)improves top-1 classification accuracy on CIFAR-10 by 0.46% for DenseNet-169 and 0.7% for Inception-v3 while with tanh(0.87x)ln(1 +e^x) top-1 classification accuracy on CIFAR-100 improves by 1.24% for DenseNet-169 and 2.57% for SimpleNet model.

Motivation & Objective

• To develop a systematic, hyper-parameterized family of activation functions that can be tuned to improve deep learning performance beyond existing standard functions.
• To address limitations of ReLU and other popular activation functions, such as non-zero mean, dying ReLU, and vanishing gradients, by introducing a more flexible non-linearity.
• To explore whether combining tanh and softplus components with tunable parameters can yield superior generalization and convergence in deep neural networks.
• To empirically validate the proposed family on benchmark datasets (CIFAR-10, CIFAR-100) across diverse architectures (DenseNet, Inception, MobileNet, WideResNet).

Proposed method

• Propose a novel activation function family defined as $ f(x; \alpha, \beta, \gamma, \delta) = \tanh(\alpha x + \beta e^{\gamma x}) \ln(\delta + e^x) $, combining tanh and softplus components with four tunable hyper-parameters.
• Use a systematic hyper-parameter search to identify optimal configurations of $\alpha$, $\beta$, $\gamma$, and $\delta$ that maximize performance on standard vision benchmarks.
• Train multiple deep learning models (DenseNet-121, DenseNet-169, Inception-v3, SimpleNet, MobileNet, WideResNet) on CIFAR-10 and CIFAR-100 using the Adam optimizer with standard training protocols.
• Compare the performance of the best-tuned TanhSoft variants against ReLU, Leaky ReLU, ELU, Swish, and Softplus across top-1 and top-3 accuracy metrics.
• Evaluate the impact of individual hyper-parameters (e.g., $\beta$ and $\gamma$) on model accuracy through ablation on the SimpleNet architecture.
• Visualize training and test accuracy/loss curves to assess generalization and convergence behavior on WideResNet-28-10 for CIFAR-100.

Experimental results

Research questions

• RQ1 Can a hyper-parameterized combination of tanh and softplus functions yield activation functions that outperform established ones like ReLU and Swish?
• RQ2 What specific values of the four hyper-parameters ($\alpha$, $\beta$, $\gamma$, $\delta$) lead to the best performance on image classification benchmarks?
• RQ3 Does the proposed TanhSoft family improve convergence speed and final accuracy across diverse deep network architectures?
• RQ4 How does the performance of TanhSoft compare to Swish and ReLU across multiple datasets (CIFAR-10, CIFAR-100) and model types?
• RQ5 Can the hyper-parameterized design enable a more systematic and effective search for improved activation functions compared to ad hoc modifications?

Key findings

• TanhSoft-1 ($\alpha = 0.87$) improved top-1 accuracy on CIFAR-100 by 1.24% with DenseNet-169 and 2.57% with SimpleNet compared to ReLU.
• TanhSoft-2 ($\beta = 0.6$, $\gamma = 1$) achieved 0.46% higher top-1 accuracy than ReLU on CIFAR-10 with DenseNet-169 and 0.7% higher with Inception-v3.
• On the SimpleNet model, TanhSoft-2 with $\beta = 0.6$ achieved 92.23% top-1 accuracy on CIFAR-10, outperforming ReLU (91.01%) and Swish (91.59%).
• For CIFAR-100, TanhSoft-2 achieved 69.28% top-1 accuracy on Inception-v3, surpassing ReLU (69.09%) and Swish (67.61%).
• The best TanhSoft variant ($\beta = 0.6$, $\gamma = 1$) achieved 92.23% top-1 accuracy on CIFAR-10 with SimpleNet, indicating strong generalization across architectures.
• Training curves for WideResNet-28-10 on CIFAR-100 showed that TanhSoft-2 achieved faster convergence and lower test loss than ReLU and Swish, suggesting improved optimization dynamics.