[Paper Review] Mish: A Self Regularized Non-Monotonic Activation Function

Diganta Misra|arXiv (Cornell University)|Aug 23, 2019

Advanced Neural Network Applications15 references18 citations

TL;DR

This paper introduces Mish, a self-regularized, non-monotonic activation function defined as f(x) = x·tanh(softplus(x)), which improves training dynamics and model performance in deep neural networks. Experimental results show Mish outperforms ReLU and Leaky ReLU by up to 1% in Top-1 accuracy on ImageNet-1k and improves AP50 by 2.1% on MS-COCO object detection, even with identical hyperparameters and architecture.

ABSTRACT

We propose $ extit{Mish}$, a novel self-regularized non-monotonic activation function which can be mathematically defined as: $f(x)=x anh(softplus(x))$. As activation functions play a crucial role in the performance and training dynamics in neural networks, we validated experimentally on several well-known benchmarks against the best combinations of architectures and activation functions. We also observe that data augmentation techniques have a favorable effect on benchmarks like ImageNet-1k and MS-COCO across multiple architectures. For example, Mish outperformed Leaky ReLU on YOLOv4 with a CSP-DarkNet-53 backbone on average precision ($AP_{50}^{val}$) by 2.1$\%$ in MS-COCO object detection and ReLU on ResNet-50 on ImageNet-1k in Top-1 accuracy by $\approx$1$\%$ while keeping all other network parameters and hyperparameters constant. Furthermore, we explore the mathematical formulation of Mish in relation with the Swish family of functions and propose an intuitive understanding on how the first derivative behavior may be acting as a regularizer helping the optimization of deep neural networks. Code is publicly available at https://github.com/digantamisra98/Mish.

Motivation & Objective

• To develop an activation function that enhances training dynamics and generalization in deep neural networks.
• To address limitations of monotonic activations like ReLU and Leaky ReLU, which may hinder optimization and generalization.
• To explore the mathematical and optimization benefits of non-monotonic, smooth activation functions in deep learning.
• To evaluate the impact of data augmentation in conjunction with Mish on major computer vision benchmarks.
• To provide theoretical and empirical insights into how Mish's derivative behavior acts as a regularizer.

Proposed method

• Proposes Mish as f(x) = x·tanh(softplus(x)), combining the properties of ReLU and Swish with improved smoothness.
• Uses softplus(x) = log(1 + exp(x)) as a smooth approximation to ReLU, enabling differentiability at zero.
• Employs the hyperbolic tangent to introduce non-monotonicity, allowing for more complex feature representations.
• Analyzes the first derivative of Mish to explain its self-regularizing effect during optimization.
• Validates performance across multiple architectures (e.g., ResNet-50, CSP-DarkNet-53) on ImageNet-1k and MS-COCO.
• Applies data augmentation techniques consistently across experiments to assess their synergistic effect with Mish.

Experimental results

Research questions

• RQ1 How does Mish compare to ReLU and Leaky ReLU in terms of Top-1 accuracy on ImageNet-1k?
• RQ2 To what extent does Mish improve average precision (AP50) in object detection on MS-COCO?
• RQ3 What is the role of Mish’s non-monotonic and smooth nature in optimizing deep neural networks?
• RQ4 How does Mish interact with data augmentation techniques in improving model generalization?
• RQ5 Can the derivative behavior of Mish be interpreted as a form of implicit regularization?

Key findings

• Mish achieved a 1% improvement in Top-1 accuracy over ReLU on ResNet-50 when trained on ImageNet-1k with identical hyperparameters.
• On MS-COCO with YOLOv4 and CSP-DarkNet-53, Mish improved AP50 by 2.1% compared to Leaky ReLU.
• The non-monotonic and smooth nature of Mish contributes to better optimization dynamics and generalization.
• Data augmentation techniques further enhanced performance gains when combined with Mish on ImageNet-1k and MS-COCO.
• The first derivative of Mish exhibits behavior that may act as a regularizer, stabilizing training and improving convergence.
• Mish outperformed established activation functions across multiple benchmarks without architectural or hyperparameter changes.